Right, your message processing should all be idempotent.
The way we achieve this is by tagging each message with a globally unique id. We use GitHub - elixir-toniq/hlcid · GitHub for this but you could also use a flake id, uuidv4, etc. We assign these ids as early as we can. Each consumer can then use those ids to enforce its delivery semantics: at most once or at least once. For most operations we choose to use at least once semantics + idempotence to achieve exactly once delivery semantics. For scenarios where you guarantee idempotence (most commonly sending an RPC to a service you don’t control and can’t support at least once delivery) we have to use At Most Once delivery.
In order to achieve either of these schemes, you’ll need a way to store which ids you’ve already processed. If you need at most once processing, you can check to see if the id has already been stored and if it hasn’t you immediately store it, then proceed to do whatever action you need to take. If you end up replaying that message, either due to a crash, error, restart, something upstream re-published the message or some other transient failure, you’ll end up skipping the operation, even if the operation had previously failed.
If you want At Least Once processing, you check to see if the id is stored, do your work, and then store the id. This assumes that the operations that you’re conducting are also idempotent. For instance, incrementing a counter isn’t an idempotent operation. So, instead you’ll need to use something like a Set and add your id to the set. Then when you want the count you can take the cardinality of the Set.
I gave a talk on this a while ago: Kafka, The Hard Parts which might be useful. After working on these types of systems over the past few years, my opinion is that stream processing systems are incredibly fragile. We’ve spent a lot of time building internal libraries that support this stuff but unfortunately I haven’t been able to get them open sourced yet. IMO the ecosystem is really lacking a solid answer around kafka specifically. I think Elixir and Erlang are really well suited for data pipeline problems, but the tooling around kafka and other ingestion isn’t really there yet. If you need to do this for production you should be ready to support a lot of your own tooling. Otherwise you should probably just use kstreams, flink, wallaroo, storm, etc.
