I am building a json REST API with Phoenix that (among other services) has to convert static CSV files from a repository and “serve” them as response in json format; that will be used downstream by other services in our project. Some of these csv files are quite big, ranging for a several hundred kb…

I used the 10000_sales.csv file from NimbleCsv , I duplicated the content until the total size was 20M (19.2 Mio). It has 15 columns. Now with this code using chunks it takes 2.8 seconds with wget to download the JSON data (size is 50M), although in the browser it loads forever. defmodule XsvWeb.Pa…

Maybe try nimble_csv . It has NimbleCSV.RFC4180 build in, which should parse your file. "path/to/file" |> File.read!() |> NimbleCSV.RFC4180.parse_string() seems also faster than streaming the file (at least for ~20MB sized files). The only downside is, that parse_string/1 returns a list of lists …

I see two possible solutions. First, as @moogle19 suggested, you can use NumbleCSV’s parse_stream and send a chunked response using Plug.Conn.send_chunked/2 , never loading the whole file into memory. Second, you can, again, use parse_stream to stream the contents into a temporary file and use Plug.C…

Can you measure which code is slow ? Is it the parsing, the JSON encoding, etc. Also you could: use Stream.map instead of Enum.map to avoid keeping too much data in memory use Jason.Fragment on each in Stream.map to encode to JSON line by line use Jason.encode_to_iodata instead of Jason.encode to…

Can you measure which code is slow ? Is it the parsing, the JSON encoding, etc. Good idea, to find out which is the “culprit” and isolate the problem. But I don’t know how to do that.

When input data is expected to be large, always use streaming and never keep all of it in memory.

[image] quda: But I don’t know how to do that. You can use Benchee… [image] benchee Versatile (micro) benchmarking that is extensible. Get statistics such as: average, iterations per second, standard deviation and the median.

I tested it with a ~130MB CSV file and the default benchee config: Name ips average deviation median 99th % nimble 0.121 8.24 s ±0.00% 8.24 s 8.24 s csv 0.0173 57.91 s ±0.00% 57.91 s 57.91 s C…

You can wrap each part of the code with :timer.tc/1 : {decode_µs, data} = :timer.tc(fn -> decode_from_csv(file) end) {encode_µs, json} = :timer.tc(fn -> Jason.encode_to_iolist!(data) end) etc.

Is there a reason to stream the data into memory? If not try using the non streamed API, which should be faster.

Phoenix json API extremely slow when serving CSV files

Questions & Help Questions

kokolegorille November 16, 2021, 11:52am 7

You can use Benchee…