Emulate File.stream! for a string variable

NobbZ · May 9, 2017, 12:19pm

Issue is already known to the elixir team

github.com/elixir-lang/elixir

Performance problem reading from StringIO

opened 01:17PM - 26 Apr 17 UTC

closed 02:42PM - 16 May 17 UTC

balexand

App:Elixir (stdlib)

### Environment Mac OS 10.12.4 on an older Macbook Pro ``` > elixir --ver…sion Erlang/OTP 19 [erts-8.3] [source] [64-bit] [smp:2:2] [async-threads:10] [hipe] [kernel-poll:false] [dtrace] Elixir 1.4.2 ``` ### Current behavior Reading from a `StringIO` using `IO.read(pid, :line)` is extremely slow. In the benchmark below with 30,000 lines it takes 2,500 times longer to read from a `StringIO` than to read from a file. ### Expected behavior I would expect reading from a `StringIO` to have similar performance to reading from a file. Furthermore, I would expect to read N lines in O(N) time. ### Benchmark ```elixir defmodule Read do def with_timing(label, pid) do {micro_sec, :ok} = :timer.tc(fn -> until_eof(pid) end) IO.puts "#{label}: #{round(micro_sec / 1_000)}ms" end def until_eof(pid) do case IO.read(pid, :line) do :eof -> :ok line when is_binary(line) -> until_eof(pid) end end end num_lines = 30_000 IO.puts "Number of lines: #{num_lines}" lots_of_lines = 0..num_lines |> Enum.map(fn _ -> "abcdefghijklmnopqrstuvwxyz" end) |> Enum.join("\n") {:ok, pid} = StringIO.open(lots_of_lines) Read.with_timing("StringIO", pid) # For comparison, read the same lines from file path = "/tmp/lots_of_lines.txt" File.write!(path, lots_of_lines, [:utf8]) file_pid = File.open!(path, [:read, :utf8]) Read.with_timing("File", file_pid) ``` ### Results ``` ➜ elixir perf.exs Number of lines: 10000 StringIO: 56336ms File: 94ms ➜ elixir perf.exs Number of lines: 20000 StringIO: 291517ms File: 172ms ➜ elixir perf.exs Number of lines: 30000 StringIO: 616679ms File: 247ms ```

also there is an PR addressing this issue

github.com/elixir-lang/elixir

StringIO read_line performance and binread bug fix (#6056)

master ← balexand:string_io

opened 11:13AM - 01 May 17 UTC

balexand

+50 -36

This pull request currently consists of 3 separate commits: 1. bf28be6243392a…5c40367d329fa7ed233e257a2b Adds test coverage to the existing implementation. 2. 2a475a6a8d1e80b1c8790915b0d08a3eb4357429 Fixes #6037, the performance/scalability problem. 3. c7e821394841c3eda7743f37933a60d501675125 Fixes a bug. `IO.binread` should return raw bytes instead of decoding the input as if it were Latin1. See [this comment](https://github.com/elixir-lang/elixir/issues/6037#issuecomment-298233279). I can split the last item into a separate pull-request if desired. Using [this benchmark](https://gist.github.com/balexand/629e37f97865ce404f002199c3b406cd), each `IO.read(pid, :line)` call takes: ``` Before: 2557µs when N=5,000 O(N) After: 14µs O(1) ``` `StringIO.get_until` suffers from the same performance problem but I've avoided touching it (and `collect_line`) because it has almost no test coverage. When I delete the entire function the only test that fails is: ``` 1) test with get until (ExUnit.CaptureIOTest) test/ex_unit/capture_io_test.exs:196 Assertion with == failed code: capture_io(fn -> :io.scan_erl_form('>') end) == ">" left: "" right: ">" stacktrace: test/ex_unit/capture_io_test.exs:197: (test) ``` I'd be happy to work on `get_until` in a separate pull-request. But first, does `StringIO` even need to support `get_until` or can we delete it? Also, is there any documentation about `get_until` other than http://erlang.org/doc/apps/stdlib/io_protocol.html?