Hi, in our Team at work we have Hackathons and for next year I am thinking about to introduce Elixir. As we have todo with data processing in our daily work I planning to propose a small app that should import data in parallel from a huge csv file (let say for example a list of persons) into a data…

Now that you posted the entire code, here’s exactly what’s going on. You are spawning 18 parallel processes and each of them is trying to insert 500 records one by one which means 9000 DB transactions. Unless you configure your pool with 9000 connections then an incomplete ingestion is to be expect…

I have updated my configuration based on what you adviced but I still have some DBConnection.ConnectionError. I was able to create ca 5 899 575 records from a total of 100 000 000. config.exs: pool_size: 10, queue_target: 10_000 mysql config: max_connections=1000 ingest_csv.ex : max_concurr…

Hm, that is strange and should not be the case. What are the errors this time around?

pool_size is related to your cpu… how many cores? It should be something around 2 x cores There is a formula, but I don’t remember it exactly

I have a CPU AMD Ryzen 7 pro 5850U 8 core for Lenovo laptop. I run the app locally in WSL.

[image] Iex.new: [image] Iex.new: MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError) I still have errors like: MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError)

yes, I will try to find something in the Mysql log or somewhere else. I will come back :slight_smile:

Ah, I don’t mean that. Surely there must be more errors in the Elixir’s console / log? But in any case I at my wits end here because I’ve done work similar to yours, at least 7 times in the last 7.5 years and was always able to fix even the occasional missed inserts.

Just now noticed that pool_size and max_concurrency are the same. As already mentioned above, I advise you to always have max_concurrency be little less than pool_size. I’d bump pool_size to 20 and then set max_concurrency to 18.

Elixir Programming Language Forum

Data processing in Parallel (Hackathon project)

Questions & Help Questions

dimitarvp December 28, 2023, 7:36am 20

Only that? No other details?