Currently I have phoenix app which also schedules quantum jobs ( GitHub - quantum-elixir/quantum-core: ⌚ Cron-like job scheduler for Elixir · GitHub ) with overlap: false mode Jobs are importing & exporting relatively large json files. Now my problem is [Screenshot from 2020-03-06 16-29-16] Leaki…

Have you tried calling :erlang.garbage_collect() at the end of whatever function you’re having Quantum run?

@NobbZ jackpot! as the matter of fact I was using Jaxon for parsing since files were pretty big before. but just tested with Jason strings: :copy and issue does not occur anymore. I suspect this was misuse of Jaxon.stream on my side but I am completely satisfied with Jason results @benwilson512 ye…

From a quick glance it seems as if jaxon wasn’t copying and also did not provide an option to copy binaries. You might still be able to use jaxon when you :binary.copy/1 strings it gives you manually.

Basically I have let’s say 100MB file What I was doing was like: path |> File.stream!() |> Jaxon.Stream.query([:root, :all]) |> Stream.chunk_every(@chunk_size) |> Enum.map(... process chunk) now when process did finish there was like XXXMB leftover pe…

That is probably because of Enum.map, depending on what it does exactly, it will force the full stream. Also if you keep references to binaries in that Enum.map instead of using :binary.copy/1 as suggested, you might keep all the lines in memory until the sub references get collected. But overall,…

I suspected enum map but it does only this: Insert chunk into postgres (let’s say ~1000 items with subresources) returns struct %ChunkRepory{success: 700, failed: 300} map is aggregated later to sum of those maps,

Elixir Programming Language Forum

Memory leak with quantum and processing binary files

Questions & Help Questions

NobbZ March 6, 2020, 4:02pm 2

Have you tried strings: :copy (assuming you use Jason)?