Hi, in our Team at work we have Hackathons and for next year I am thinking about to introduce Elixir. As we have todo with data processing in our daily work I planning to propose a small app that should import data in parallel from a huge csv file (let say for example a list of persons) into a data…

Now that you posted the entire code, here’s exactly what’s going on. You are spawning 18 parallel processes and each of them is trying to insert 500 records one by one which means 9000 DB transactions. Unless you configure your pool with 9000 connections then an incomplete ingestion is to be expect…

I have a CPU AMD Ryzen 7 pro 5850U 8 core for Lenovo laptop. I run the app locally in WSL.

[image] Iex.new: [image] Iex.new: MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError) I still have errors like: MyXQL.Connection (#PID<0.302.0>) disconnected: ** (DBConnection.ConnectionError)

yes, I will try to find something in the Mysql log or somewhere else. I will come back :slight_smile:

Ah, I don’t mean that. Surely there must be more errors in the Elixir’s console / log? But in any case I at my wits end here because I’ve done work similar to yours, at least 7 times in the last 7.5 years and was always able to fix even the occasional missed inserts.

Just now noticed that pool_size and max_concurrency are the same. As already mentioned above, I advise you to always have max_concurrency be little less than pool_size. I’d bump pool_size to 20 and then set max_concurrency to 18.

I wish you all a happy and healthy new year! I have configured the pool size as advised to 20 and the max_concurrency to 18. This seems to have no effect as I still have the same issue as before. config/config.exs import Config config :file_watch_example, :ecto_repos, [Store] config :file_watc…

Do not insert 500 records in a loop. Maybe just remove the Stream.chunk_every and Enum.map calls and and just insert records one by one. I think the way you’re doing it now increases contention for DB connections. Working with batches of records in your scenario only makes sense if you’re going to …

thank you, I will try that. For which use cases make it sense to do not do validation?

Elixir Programming Language Forum

Data processing in Parallel (Hackathon project)

Questions & Help Questions

kokolegorille December 28, 2023, 1:21pm 23

You can try pool_size = 16