Hi, in our Team at work we have Hackathons and for next year I am thinking about to introduce Elixir. As we have todo with data processing in our daily work I planning to propose a small app that should import data in parallel from a huge csv file (let say for example a list of persons) into a data…

Now that you posted the entire code, here’s exactly what’s going on. You are spawning 18 parallel processes and each of them is trying to insert 500 records one by one which means 9000 DB transactions. Unless you configure your pool with 9000 connections then an incomplete ingestion is to be expect…

If you don’t want to store future and current tasks state you can just get away with NimbleCSV and Task.async_stream/3 , very easily: defmodule IngestCSV do alias NimbleCSV.RFC4180, as: YourCSV def load(path) do path |> Path.expand() |> File.stream!(read_ahead: 524_288) |> YourC…

@dimitarvp , thank you! Yesterday I was able to implement what I wanted to do with the solution you proposed :slight_smile: In my process I check with the lib file_system if a new file was detected in a specific directory and if yes then it will proceed. Each line of the CSV file is correctly impor…

Glad you made it work! I can offer you something that can accelerate the code further: you can put Stream.chunk_every(500) after YourCSV.parse_stream() and then the Task.async_stream will accept a batch of 500 records (not a single record). And then you can do batch inserts – provided you don’t do…

thanks for the hint ! I have tried with 100 000 000 rows but only 62 733 738 have been imported. I got some errors like: 01:14:22.102 [info] MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError) client #PID<0.726857.0> exited 01:14:22.096 [error] MyXQL.Connection (#PID<…

If the MySQL instance itself is telling you “too many connections” maybe you should look into its own config – maybe it is not configured to accept that many? I mean you can allow Elixir to connect to 100 separate MySQL connections (on the same server) but if MySQL is configured to e.g. allow maximu…

I’m curious if you fix the connection errors. Please let us know.

[image] Iex.new: MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError) Until now I hadn’t managed :frowning: Resources I found to help me to try to solve the issue: Mysql Documentation: https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_…

First of all, don’t use a huge pool of connections, start with 10, having a bigger pool doesn’t mean necessarily that the queries will run faster.

These numbers are crazy. First of all, give MySQL a more lax limit: 100k is too much. Make it something like 1000 or 2000. Furthermore, even 100 parallel tasks was already a lot before because you can likely saturate your CPU or even network (depending on what machine you’re running the workload). …

Elixir Programming Language Forum

Data processing in Parallel (Hackathon project)

Questions & Help Questions

Iex.new December 21, 2023, 3:57pm 10

Thank you @dimitarvp !