I’m a heavy Kafka user, and I have been bouncing between Brod, KafkaEx, an BroadwayKafka for a long time.
Until now I have been using Brod, which was a bit abandoned until a month ago, when it was donated to the community by the company that started the project.
Now mainly zmstone and k32 are maintaining the library but they still have too much work to do. Meanwhile, Kafka_ex has also many PRs opened, and lot of work to be done. In both cases, I really think maintainers needs help (and I would love to at least send them a ko-fi).
My problem is that I currently need lz4 compression and Exactly once delivery, and neither of them are there yet. Also I’ve some issues with the consumer group balancer in brod, because our running Kafka version uses an old balancing protocol.
Recently I started looking at the Python, Ruby, Go and Js libraries. Official ones (from Confluent) depends on GitHub - confluentinc/librdkafka: The Apache Kafka C/C++ library · GitHub (C/C++) which has a lot of support.
Of course, there is also one for the BEAM (GitHub - silviucpp/erlkaf: Erlang kafka driver based on librdkafka · GitHub). I’ve used it in development environment, and until now, it works perfectly well. For example, I experimented consuming with a topic with 12 partitions, with 1 node, it automatically spawns 12 processes, I run another node and automatically balances 6 and 6 processes (That is handled by the Kafka protocol).
Someone could be worried by nifs but as erlkaf readme says: librdkafka driver is async, so it doesn’t block.
So I think, for now, I will be migrating to erlkafka. But I will keep looking for brod and kafka_ex, hoping they get more support.
