Phoenix json API extremely slow when serving CSV files

Indeed I used "[" and "]" for the brackets, and ?, for the comma, it is not very consistent :slight_smile:

I guess when sending a chunk I thought we send some content to the browser here, while building the JSON chunks I thought here we are building some IO data.

Also there is the Stream.intersperse/2 function that would make the whole code cleaner when inserting commas between records instead of using take(1) and then drop(1).

The code would look simpler :

  defp csv_stream(file) do
    headers = headers(file)

    file
    |> File.stream!()
    |> CSV.parse_stream(skip_headers: true)
    |> Stream.map(fn x -> Enum.zip(headers, x) |> Map.new() |> Jason.encode_to_iodata!() end)
    |> Stream.intersperse(",")
  end

But note that for my 50 MB JSON it takes ~4s instead of ~3s. because you send each comma as its own chunk.