Data processing in Parallel (Hackathon project)

Questions & Help Questions
, ,
28

In your current code.

It tries to do two things that conflict: (1) work with the records in batches – via the Stream.chunk_every(500) call, and (2) insert records one by one.

You don’t do that.

You either:

  1. Use Stream.chunk_every(500) and Repo.insert_all(rows) (which skips validation).

OR

  1. Do NOT use Stream.chunk_every(500) and do NOT use Enum.map(rows, ... Repo.insert(one_record)). Just do Repo.insert(row) directly.

I’ll follow up with a bigger comment explaining why you can’t ingest all records. Looking at your complete code it’s now perfectly clear. Incoming.