I have read through this thread: binary-pattern-matching-when-input-is-a-stream on a related subject, but i’m not able to make any headway. I get this error trying to load and parse a large binary directly: iex> JQL.save "--1566364097905", "STMT.ENTRY" eheap_alloc: Cannot allocate 1844674407169255…

Found a really simple way to solve this. Just read the data-header, then parse row by row: def read(src, cnt \\ nil) do {:ok, fid} = :file.open("_cache/#{src}", [:raw, :read, :binary]) {:h, start, n} = _p(fid, :h) cnt = cnt || n Logger.debug("Reading: #{cnt} of #{n} rows") …

No surprise there as I assume that you do not have 163 PB of RAM available for your machine. It seems like you want to load a lot of data at once. Maybe try to split it into reasonable chunks?

But the error message you posted, it claims that it wants to allocate another 18446744071692551144 byte, which is 18014398507512256 kiB, 17592186042492 MiB, 17179869182 GiB, 16777215 TiB, 16383 PiB, so ~16 EiB… As you can see @hauleth even missed by a factor of ~100… There might be plenty of reaso…

Very strange. Im at a loss. I’ve uploaded the code and exact file I’m trying to process

Does the problem persist if you remove this line? https://gist.github.com/CharlesOkwuagwu/4c6c89d96db7876bc0d27fecd518340e#file-jql-ex-L102 You are not even using the result of term_to_binary… And unless compressed, it will always require more memory than the input.

That line was just for testing something else. removing it still brings allocation errors, but smaller: iex> JQL.save "--1566364097905", "STMT.ENTRY" eheap_alloc: Cannot allocate 1898305688 bytes of memory (of type "heap"). Crash dump is being written to: erl_crash.dump... def save(src, name) do…

Another problem might be, that you are appending to your accumulator all the time… This requires rebuilding the list all the time, eg: defp _r(<<"LI", 1::32, ";", b::bits>>, val), do: _r(b, val ++ [nil]) I’m not sure how to refactor though. As I currently will not try to understand the full parser…

@NobbZ i just uploaded a 3k sample that contains 5 rows of data. IF i can optimize the way i handle this, possibly the much larger file may equally be processed without memory issues

[image] NobbZ: Another problem might be, that you are appending to your accumulator all the time… This requires rebuilding the list all the time, eg: defp _r(<<"LI", 1::32, ";", b::bits>>, val), do: _r(b, val ++ [nil]) I see your point. The mistake might be in trying to build out / parse th…

@CharlesO At a minimum if you are going to parse it all in memory, you need to never append to a list in a loop. This is an O(n^2) operation. You should prepend to the list, and then reverse at the end. Or, use a body recursive function that builds the list front to back via recursion.

Elixir Programming Language Forum

Parse / Pattern match large binary data

Questions & Help Questions

CharlesO August 21, 2019, 11:46am 3

Im just loading 853mb …

I have 16 GB ram