I have read through this thread: binary-pattern-matching-when-input-is-a-stream on a related subject, but i’m not able to make any headway.
I get this error trying to load and parse a large binary directly:
iex> JQL.save "--1566364097905", "STMT.ENTRY"
eheap_alloc: Cannot allocate 18446744071692551144 bytes of memory (of type "heap").
Crash dump is being written to: erl_crash.dump...
I’m doing an ETL (extract - transform - load) for a reporting project, moving data from a proprietary platform into SQL Server for easier reporting.
The parser has worked fine until I hit this size issue.
Please how / can we use streams to handle these kind of situations?
(particularly when initial or existing parser had not been built with streaming in mind)
JQL Parser: https://gist.github.com/CharlesOkwuagwu/4c6c89d96db7876bc0d27fecd518340e (updated)
Sample large file (~850mb unzipped): https://paperlesssolutionsltd.com.ng/java/--1566364097905.7z
Thanks.
