Some general remarks for guidance:
-
Keep in mind the Erlang VM makes specific trade-offs in relation to high-performance, such as process preemption. It is better to have a predictable system that goes slightly less fast than a fast ones that is unpredictable (or crashes)
-
Streams have lower memory usage at the cost of higher CPU usage. If your goal is to go as fast as you can, not using streams may be better (such as
File.read! |> String.split("\n", trim: true)) -
You should see benefits by adding Task.async_stream and similar so you can leverage multi-core
-
I would assume that most of the time is taken by JSON parsing so remember Jason is a pure Elixir package. I assume that the json parsing in Python is most likely done in C. So you may have better results by using something like
jiffy(and a more apples to apples comparison)
