Phoenix json API extremely slow when serving CSV files

?, means: "give me the codepoint of the comma (in this case it returns 44, which you can try on iex: iex> ?,). You can read about the usages of IO data here: IO — Elixir v1.12.3

So this function does (as I understand it):

    stream
    # Take the first line from the CSV (since we used "skip_headers" this is the first line of data)
    |> Stream.take(1)
    # 1. Merge the headers of the csv with the data we just read into tuples of {HEADER, DATA}
    # 2. Make a new map from that list of tuples
    # 3. Encode the map as json and output it as io_data instead of as string
    |> Stream.map(fn x -> Enum.zip(headers, x) |> Map.new() |> Jason.encode_to_iodata!() end)
    # We now have a list of io_data lists, so something like: `[ [12, 31, 55], [13, 14, 99], ... ]`
    # which goes as the first argument into Stream.concat/2
    |> Stream.concat(
      # The second argument to the concat function
      stream
      # But this time we remove the first line...
      |> Stream.drop(1)
      # ... map over _each other line_ in the csv...
      |> Stream.map(fn x ->
        # ... and then prepend it with a comma in codepoint format
        [?,, Enum.zip(headers, x) |> Map.new() |> Jason.encode_to_iodata!()]
      end)
    )

All this will result in something like this {"hello": "world"}, {"hello": "foo"}, {"hello": "bar"} (notice that we don’t have a comma at the first position?). The send_csv function wraps that into [ and ] which makes it a valid json array. This whole thing with the code points is probably done for performance reasons in this case and I personally have not had the need to use this form. This is relatively advanced stuff for a relatively advanced problem.

So please don’t beat yourself up about this. We’re all here to learn and to give back where we can! This forum, or rather it’s management and the people in it, might just be the best feature of Elixir in general - and that says a lot. :slight_smile: