I am building a json REST API with Phoenix that (among other services) has to convert static CSV files from a repository and “serve” them as response in json format; that will be used downstream by other services in our project. Some of these csv files are quite big, ranging for a several hundred kb…

I used the 10000_sales.csv file from NimbleCsv , I duplicated the content until the total size was 20M (19.2 Mio). It has 15 columns. Now with this code using chunks it takes 2.8 seconds with wget to download the JSON data (size is 50M), although in the browser it loads forever. defmodule XsvWeb.Pa…

My suggestion is to use @lud 's solution to see if it is good enough or not. If you still want more speed, you can write a native csv to json converter in a command line tool using rust or go, reading from the FS and writing to stdout. That should be doable in 100 LOC. Then use a port to run the co…

[image] dimitarvp: Passing that to the function I gave you above will yield a list of maps in the shape visible in the processor function. You could have explained this from the beginning. It does not matter my (humble) experience on Elixir. I was assigned to do this task, given a deadline a…

I didn’t mean to be rude and sorry if it read that way – my point is that without some level of understanding we cannot help you because you might not understand and not be able to apply the suggestions given to you. So then how can we help you? What I suggested to you is: integrate all processing …

Totally agree, @ dimitarvp . Please understand that I am under much pressure about this task. I didn’t have years to study Elixir, just a couple of weeks, but it convinced me. I’m coming from the OOP world (C#, Java, php) so the functional paradigm still feels strange to me. I’m still trying to giv…

[image] quda: Please understand that I am under much pressure about this task. I understand and sympathize. Been there, done that myself. Send us a link to a sample CSV file – or just tell us how much columns it has. I am sure we can think of something to help you in short order. Also which …

Just want to echo this, using ports and running a binary from Go or Rust (or whatever floats your boat) is an excellent way of doing data manipulation in an easy and understandable way.

@quda after reading this thread I really think what you have is an architecture problem and I’m basically echoing @lud here but just to clarify that what you’re describing would probably be not a good idea regardless of the language/technology. Short version: Don’t parse CSV->JSON, simply write J…

@lud and @dimitarvp you were both right. I re-wrote the controller using the @lud ’s example and guess what? It works out-of-the-box, like a charm. curl -x "" localhost:4000/api/D3/env.ERA638.csv &> /dev/null % Total % Received % Xferd Average Speed Time Time Time Current …

?, means: "give me the codepoint of the comma (in this case it returns 44, which you can try on iex: iex> ?,). You can read about the usages of IO data here: IO — Elixir v1.12.3 So this function does (as I understand it): stream # Take the first line from the CSV (since we used "skip_heade…

Elixir Programming Language Forum

Phoenix json API extremely slow when serving CSV files

Questions & Help Questions

dimitarvp November 17, 2021, 1:10pm 39

Awesome explanation, kudos.