Hi there,
I have a problem with ecto timeouts (so it seems). In a Elixir application I wrote a module for importing data. It gets a CSV file, goes through that file line by line (with a Stream.map()), looks if this dataset already exists in the database, updates it or inserts it as a new dataset. Pretty basic.
To make that operation restartable I wrap this whole process in a database transaction. The amount of datasets is pretty large and it can take up to an hour.
I know of Ecto timeouts, and thus I start that transaction with timeout: :infinity to avoid timeout problems. It works in development, but in prod I get strange errors:
16:02:34.435 [error] Postgrex.Protocol (#PID<0.423.0>) disconnected: ** (DBConnection.ConnectionError) ssl send: closed
or
** (exit) an exception was raised:
** (DBConnection.ConnectionError) ssl send: closed
(ecto_sql) lib/ecto/adapters/sql.ex:624: Ecto.Adapters.SQL.raise_sql_call_error/1
(ecto_sql) lib/ecto/adapters/sql.ex:557: Ecto.Adapters.SQL.execute/5
(ecto) lib/ecto/repo/queryable.ex:147: Ecto.Repo.Queryable.execute/4
(ecto) lib/ecto/repo/queryable.ex:18: Ecto.Repo.Queryable.all/3
(ecto) lib/ecto/repo/queryable.ex:66: Ecto.Repo.Queryable.one/3
(termitool) lib/termitool/meta/meta.ex:332: Termitool.Meta.get_user_by/1
(termitool) lib/termitool/meta/meta.ex:439: Termitool.Meta.get_by_username/1
(termitool) lib/termitool/meta/meta.ex:465: Termitool.Meta.username_password_auth/2
Basically connection errors at random places in the application. I can avoid this problem by setting timeout: :infinity in the repo configuration, but this seems wrong and dangerous to me.
The code is basically:
Repo.transaction(fn ->
stream
|> Stream.map(fn {row, idx} -> update_or_create_row(row) end)
end,
timeout: :infinity)
I am using a stream because that’s what I get from the CSV parsing library.
What am I doing wrong?
Best regards,
CK
