Hi, in our Team at work we have Hackathons and for next year I am thinking about to introduce Elixir. As we have todo with data processing in our daily work I planning to propose a small app that should import data in parallel from a huge csv file (let say for example a list of persons) into a data…

Now that you posted the entire code, here’s exactly what’s going on. You are spawning 18 parallel processes and each of them is trying to insert 500 records one by one which means 9000 DB transactions. Unless you configure your pool with 9000 connections then an incomplete ingestion is to be expect…

[image] Iex.new: For which use cases make it sense to do not do validation? In your current code. It tries to do two things that conflict: (1) work with the records in batches – via the Stream.chunk_every(500) call, and (2) insert records one by one. You don’t do that. You either: Use St…

I wonder if postgres can handle this abuse better.

Likely yes but with that code the odds are against any DB.

thank you very much! , that’s solve my issue. I get no errors anymore logged. Now I have to find out why the process still creating records although the end of the file was reached. The file containing products: cat file.csv | wc -l 10000000 Total products created in the database: mysql> select…

You can also split the CSV file on smaller chunks and experiment with them.

Can’t know but obviously empty the DB before starting a run. That is what I do before each run. I also clear the log before to check that the amount of INSERT is equal to the amount of data records created into the database :slight_smile:

As said above, start chopping that CSV file on smaller chunks and experiment. I can’t help you further unless you give me full access to the file, inform me of your exact MySQL version, what Erlang / Elixir versions you use, and give me access to the code. You’ll have to track this down yourself b…

Oh btw, one final thing you can try just so we’re 100% sure: Don’t do parallelism. Just stream the CSV file records and insert them one by one and see if you get 10M records. BTW the amount above is mind-blowingly different… you expect 10M records but get almost 110M? Something is super weird, man…

I have uploaded the file separately as advised. The file is available for download until 12.01.2024: products.csv I have now created a repo with the code I use. Thank you so much for your help!

Elixir Programming Language Forum

Data processing in Parallel (Hackathon project)

Questions & Help Questions

dimitarvp January 5, 2024, 7:50am 33

Can’t know but obviously empty the DB before starting a run.