I am building a json REST API with Phoenix that (among other services) has to convert static CSV files from a repository and “serve” them as response in json format; that will be used downstream by other services in our project. Some of these csv files are quite big, ranging for a several hundred kb…

I used the 10000_sales.csv file from NimbleCsv , I duplicated the content until the total size was 20M (19.2 Mio). It has 15 columns. Now with this code using chunks it takes 2.8 seconds with wget to download the JSON data (size is 50M), although in the browser it loads forever. defmodule XsvWeb.Pa…

For the prior benchmarks I used csv files from here: https://www.stats.govt.nz/large-datasets/csv-files-for-download/ Geographic units, by industry and statistical area: 2000–2021 descending order – CSV is ~ 120MB when unzipped. But I am not sure how the number of columns affects the runtime. The …

@lud you’re right about (not) beating php. Very disappointed. We were asked to migrate/refactory the old backend for APIs, developed 7 years ago on php5/nginx because it recently started to have performance issues: too many concurrent calls from the downstream API consumers. The cpu and memory wer…

EDIT: @lud ’s code is more complete than mine, I only demonstrated how to avoid passing huge collections around. [image] lud: @quda Could you give us a big anonymized file so we can try stuff? Seconded, without that we can just theorize until the end of time. I agree with @LostKobrakai tha…

I find Elixir and Phoenix a joy to work with. Individual requests are generally very fast for common purposes. But here you are doing a lot of string manipulations and iterating over large chunks of data, witch is not what Elixir is designed to do fast, unfortunately. Honestly, I couldn’t imagine …

Thanks for suggestion. I don’t understand what this processor function is [image] dimitarvp: when is_function(processor, 1)

How much Elixir did you learn before coming to the forum? Do you know the basics of functional programming? I personally prefer to give people good advice after they have done some homework and not before. Still, let me add to my example so it is at least complete: # # The function signature `[fir…

[image] lud: Lots of PHP libraries are bloated with overengineered, convoluted, interface-heavy APIs but the raw performance is there (to my taste at least). Exactly ! This php backend became an unmanageable beast in the last years (thousands of internal commits made by people are not working…

My suggestion is to use @lud 's solution to see if it is good enough or not. If you still want more speed, you can write a native csv to json converter in a command line tool using rust or go, reading from the FS and writing to stdout. That should be doable in 100 LOC. Then use a port to run the co…

[image] dimitarvp: Passing that to the function I gave you above will yield a list of maps in the shape visible in the processor function. You could have explained this from the beginning. It does not matter my (humble) experience on Elixir. I was assigned to do this task, given a deadline a…

I didn’t mean to be rude and sorry if it read that way – my point is that without some level of understanding we cannot help you because you might not understand and not be able to apply the suggestions given to you. So then how can we help you? What I suggested to you is: integrate all processing …

Phoenix json API extremely slow when serving CSV files

Questions & Help Questions

quda November 16, 2021, 4:10pm 27

Thanks for suggestion. I don’t understand what this processor function is