@quda after reading this thread I really think what you have is an architecture problem and I’m basically echoing @lud here but just to clarify that what you’re describing would probably be not a good idea regardless of the language/technology.
Short version:
- Don’t parse CSV->JSON, simply write JSON files to the server and serve those.
- If you can, send the CSV file directly to the client and parse it there.
- Depending on if you actually need the whole file at once in the client, you could implement a pagination on large files.
Long version:
- As @lud mentioned: Why write CSV files to the server if what you need is JSON? Nothing - not even C or Rust - would be faster at parsing CSV and writing JSON than Nginx serving some static JSON files. So if you must use JSON I would write those files directly as JSON and not convert them on the fly in the request.
If you need authentication you can, of course, also just serve the file contents with phoenix. It would be a tad slower than Nginx but much, much faster than parsing and transforming the file. (This would probably also solve the performance problem of the server because you wouldn’t compute something in every request). - If you can, you should really just serve the CSV files and parse them on the client. Browser JS engines are really, really fast when dealing with data structures and you simply cannot render all data from 20MB files as HTML at once. That would kill the browser.
- You probably want some pagination for those files in order to only send a subset of the data. I have only once had the missfortune of having to send such an amount of data to the client and process it there. Most of the times you want to chunk it up and load more data on demand.
