Hi there, I have a problem with ecto timeouts (so it seems). In a Elixir application I wrote a module for importing data. It gets a CSV file, goes through that file line by line (with a Stream.map()), looks if this dataset already exists in the database, updates it or inserts it as a new dataset. P…

Are you running in a Docker container? I discovered a bug in the the official Elixir 1.8.1 container, which is backed by Erlang 21.3, which caused SSL problems like these in production. I had to switch to a custom image running Erlang 21.2.7.

This looks strange to me: your last statement in such a processing pipeline should always be using Enum; Stream just returns a function. Are you sure any actual work is done by this code in production?

Sorry - it‘s an Enum.map, not a Stream.map Edit: ok, I’m at the PC again. Sorry, I simplified the code a bit, in fact it is more like this: Repo.transaction(fn -> stream |> Stream.map(&create_or_replace/1) |> Enum.reduce(…) end, timeout: :infinity) The reduce counts and throws away the suc…

Is it the transaction that times out or the individual query? As far as I can remember DB lessons during university, individual queries during a transaction might take longer than without the transaction due to the doublebookkeeping of indexes or linear scan of things that are only visible in the t…

There are some tools to leverage high import, for example GenStage, which will provide back pressure support. I would also reach for insert_all, because it’s more efficient to send one query with 10’000 insert than 10’000 single insert. Enum.chunk_by can be useful to treat smaller pieces. But all…

The timeouts appear all over the place in the application during the import job, in parts of the application which have nothing to do with the import and are not wrapped in the transaction. But the queries in the transaction seem not to time out, I did not see the connection errors there.

insert_all is not a viable solution in this case, I either update existing rows or insert new rows, but not all in one table. For example one CSV row can result in 4 rows in 4 different tables.

Instead of inserting new rows, I keep them in a list, and proceed all in one go. If You have different tables, You still might keep multiple lists (one per table), and proceed with multiple insert_all. I had similar constraint (huge textfile, multiple tables, over 1’000’000 records) with update or…

Hm. Race conditions on the data are a non-issue in this case. But I will overhaul my import pipeline, thanks for your input… That said, I guess I found the underlying cause for this problem. Your hint with concurrently importing the data gave me the idea: the CSV library uses workers to parallelize…

Dammit. The errors appeared again, random connection drops by Postgrex: 11:38:50.415 [error] Postgrex.Protocol (#PID<0.425.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed 11:38:52.415 [error] Postgrex.Protocol (#PID<0.431.0>) disconnected: ** (DBConnection.ConnectionError) ssl…

Elixir Programming Language Forum

Timeout errors with long running transactions

Questions & Help Questions

cjk March 15, 2019, 6:29am 4

Nothing, nobody? Am I missing some important information?