That is likely that to be the answer to your question. If I understand the wider problem you are receiving stuff, filtering duplicates, grabbing additional information and inserting the collated result. If you are getting duplicates at the insert end then maybe a reasonable solution would maybe be to not insert and instead use an update with upsert so that each message processed is idempotent and any duplicates have no real effect, you still have an extra API call being made so it depends on the implications of that. The MongoDB API I’m thinking is db.collection.update() (mongosh method) - Database Manual - MongoDB Docs though I have no experience with it in the Elixir specific clients.
Edit: you could maybe solve the API call problem with tailable cursors to augment the additional information on inserted records?
