Hi, in our Team at work we have Hackathons and for next year I am thinking about to introduce Elixir. As we have todo with data processing in our daily work I planning to propose a small app that should import data in parallel from a huge csv file (let say for example a list of persons) into a data…

Now that you posted the entire code, here’s exactly what’s going on. You are spawning 18 parallel processes and each of them is trying to insert 500 records one by one which means 9000 DB transactions. Unless you configure your pool with 9000 connections then an incomplete ingestion is to be expect…

These numbers are crazy. First of all, give MySQL a more lax limit: 100k is too much. Make it something like 1000 or 2000. Furthermore, even 100 parallel tasks was already a lot before because you can likely saturate your CPU or even network (depending on what machine you’re running the workload). …

I have updated my configuration based on what you adviced but I still have some DBConnection.ConnectionError. I was able to create ca 5 899 575 records from a total of 100 000 000. config.exs: pool_size: 10, queue_target: 10_000 mysql config: max_connections=1000 ingest_csv.ex : max_concurr…

Hm, that is strange and should not be the case. What are the errors this time around?

pool_size is related to your cpu… how many cores? It should be something around 2 x cores There is a formula, but I don’t remember it exactly

I have a CPU AMD Ryzen 7 pro 5850U 8 core for Lenovo laptop. I run the app locally in WSL.

[image] Iex.new: [image] Iex.new: MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError) I still have errors like: MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError)

yes, I will try to find something in the Mysql log or somewhere else. I will come back :slight_smile:

Ah, I don’t mean that. Surely there must be more errors in the Elixir’s console / log? But in any case I at my wits end here because I’ve done work similar to yours, at least 7 times in the last 7.5 years and was always able to fix even the occasional missed inserts.

Data processing in Parallel (Hackathon project)

Questions & Help Questions

Iex.new December 28, 2023, 7:23am 19

I still have errors like:

MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError)