Hello, Why is this Elixir code: defmodule Foo do def run(file_name) do File.open! file_name, [:read], fn f -> IO.stream(f, :line) |> Enum.each(&process_line/1) end end defp process_line(line) do String.rstrip(line) |> String.split(",") end end [ file_name | _ ] = Syst…

Friend sent me this: http://blog.alainodea.com/en/article/398/blazing-fast-concurrent-text-i-o-in-erlang

Thanks! It’s twice as fast now: $ time elixir test.exs ../data_gen/posts.csv real 0m12.135s user 0m11.734s sys 0m0.807s (using this code) def run(file_name) do File.stream!(file_name, [:read]) |> Enum.each(&process_line/1) end Still twice as slow as Ruby though… :confounded: An…

With String.rstrip(line) |> String.split(","), you are ripping through the line multiple times. Try using the trim: option instead: defp process_line(line) do line |> String.split(",", trim: true) end

That change made no significant difference… :sad face:

Wild/dumb guess but maybe because it has to spin up a VM to run?

I assume that accounts for milliseconds, not many seconds… :blush:

I’m not sure how fast BEAM boots up, but have you tried profiling using fprof ?

What version of Ruby is that? Not all of them will use UTF-8 by default. You could also load the whole file into memory first, rather than stream. If I’m not mistaken that’s what the Ruby version does.

build executable with escript? [image] Writing a command line application in Elixir Writing a command line application in Elixir

Running an elixirscript means to compile and then call into the compiled module. I do not think, that for that short script the compiletime will make any significant difference.

Elixir Programming Language Forum

Help with performance (file io)

Questions & Help Questions

uranther June 7, 2016, 2:34am 2

Try File.stream!/3 instead of File.open! and IO.stream