Hello, Why is this Elixir code: defmodule Foo do def run(file_name) do File.open! file_name, [:read], fn f -> IO.stream(f, :line) |> Enum.each(&process_line/1) end end defp process_line(line) do String.rstrip(line) |> String.split(",") end end [ file_name | _ ] = Syst…

Friend sent me this: http://blog.alainodea.com/en/article/398/blazing-fast-concurrent-text-i-o-in-erlang

Getting fast I/O stream processing in Elixir does require some non-obvious tweaks. Due to the way I/O works on the BEAM (there’s a special process that does I/O and passes results to your process as a message), you want the messages to be as long as possible to avoid overheads. Doing it line by li…

Hmm, using separate process to read separate sections of the file is one trick I have not tried. However, everything requires benchmarks on the hardware you’re actually going to run on. Latency is everything when it comes to schemes for concurrent I/O. My guess it that the size of the file would ha…

[image] bbense: Hmm, using separate process to read separate sections of the file is one trick I have not tried. Well, a part of that trick is manually partitioning the file by hand ahead of time with head -n | tail -n… :frowning: Not sure how to do that automatically without (slowly) scanni…

Correct me if I’m wrong but if you aren’t using mix then your protocols aren’t consolidated, so there’s a massive overhead to using the Enumerable protocol which is all of your Stream / Enum stuff.

The problem with most of these benchmarks is that what they actually do with the data is relatively trivial. Since most other languages are more or less using exactly the same I/O flow these benchmarks are reasonable tests of the relative speed of the text processing in each language. In the case o…

This topic got me interested both as an exercise with basic parts of Elixir, but also because I love writing shell scripts and I’m sure I’m not the only one considering Elixir as a nice language not just for servers, but tooling too. Pattern matching, easy packaging with escript, self-supervisioned …

[image] karolsluszniak: So I’ve also spent some time researching and optimizing and as result I wrote the following blog pos There are many interesting points in that post. However, I disagree that the task is mostly I/O bound. In fact, it seems to me the task is mostly CPU bound, spending mo…

Thanks for the feedback, that’s exactly what I was going after when I wrote that article :slight_smile: I’m not that good with binary pattern matching yet so it’s nice to see it put into good use. You’re right that the read version spends lots of time running string and regex operations in the filt…

[image] karolsluszniak: You could also reduce the rem(num, 2) == 0 || rem(num, 5) == 0 to a check if last char is one of 0 2 4 5 6 8 :slight_smile: Ah, this is a great trick! I just tried it, and it reduces the running time to 0.8 sec: defp filter_line(<<c::utf8, ?,::utf8, _::binary>>) whe…

If you want to limit the memory usage, then yes. To make it work faster, you can use a read-ahead buffer. Similarly, you could use delayed write on the writing side. Wow, the :delayed_write indeed is a game changer here! I’ve somehow missed it in the docs. I’ve already tried :read_ahead and it d…