Hello, Why is this Elixir code: defmodule Foo do def run(file_name) do File.open! file_name, [:read], fn f -> IO.stream(f, :line) |> Enum.each(&process_line/1) end end defp process_line(line) do String.rstrip(line) |> String.split(",") end end [ file_name | _ ] = Syst…

Friend sent me this: http://blog.alainodea.com/en/article/398/blazing-fast-concurrent-text-i-o-in-erlang

Running an elixirscript means to compile and then call into the compiled module. I do not think, that for that short script the compiletime will make any significant difference.

[image] NobbZ: I do not think, that for that short script the compiletime will make any significant difference. Yep you are right

on 4mb csv file php is about 10x faster

[image] andre1sk: on 4mb csv file php is about 10x faster Oye, so that’s it, huh. I’m writing a distributed computation framework (basically an Apache Storm rip off) in both Elixir and Ruby. I’m at the point now where I can benchmark and analyze post-computation statistics. As expected, El…

Measuring with time is a bit misleading, since you’re also measuring the time to start the VM. Probably won’t make big difference in this case, but to get a more precise number, use :timer.tc . Also, since chomp and split in Ruby are implemented in C, I’d expect some constant factor difference in Ru…

Getting fast I/O stream processing in Elixir does require some non-obvious tweaks. Due to the way I/O works on the BEAM (there’s a special process that does I/O and passes results to your process as a message), you want the messages to be as long as possible to avoid overheads. Doing it line by li…

Hmm, using separate process to read separate sections of the file is one trick I have not tried. However, everything requires benchmarks on the hardware you’re actually going to run on. Latency is everything when it comes to schemes for concurrent I/O. My guess it that the size of the file would ha…

[image] bbense: Hmm, using separate process to read separate sections of the file is one trick I have not tried. Well, a part of that trick is manually partitioning the file by hand ahead of time with head -n | tail -n… :frowning: Not sure how to do that automatically without (slowly) scanni…

Correct me if I’m wrong but if you aren’t using mix then your protocols aren’t consolidated, so there’s a massive overhead to using the Enumerable protocol which is all of your Stream / Enum stuff.

Elixir Programming Language Forum

Help with performance (file io)

Questions & Help Questions

andre1sk June 7, 2016, 2:59pm 16

if you use IO.binstream instead of File.stream it speeds things up x2