Yes, I’m totally aware of this. As I mentioned here, I can only speak to plans for the consumer side right now, and I’m very interested in learning about the current issues the community faces with Kafka to try to create a better experience. Thanks for sharing it!
The issue of “back pressure” keeps coming up, and I’m still working to fully understand it, especially since Kafka consumers inherently support back pressure by processing messages in pull mode. This means consumers should only request new records once they’ve fully handled the previous set, so if upstream services become slow, consumption naturally slows down as well.
However, given that back pressure still is a big pain, I believe that Kafka consumers shouldn’t merely slow down in proportion to upstream services; ideally, they would slow down even further to allow upstream services some breathing room to recover.
Several people have noted that bottlenecks often appear in upstream services like databases before Kafka consumption itself becomes an issue. With that in mind, here are a few approaches I’m considering:
-
The auto-acking behavior in some Kafka libraries might not be ideal for these scenarios. If a database is slow or times out, the Kafka consumer should slow down rather than auto-acknowledging and risking overload.
-
Providing a “rate limiting” option for consumers could allow manual control over processing rates, helping to prevent Kafka consumers from overwhelming upstream services during production peaks.
-
Tracking consumption times for each consumer could enable an adaptive rate-limiting feature. For instance, if a consumer usually processes a batch of 10 records within 1 second but suddenly takes 5 seconds, we could reduce the consumption rate to give upstream services a chance to recover. Exposing these metrics to users could also be valuable.
Let me know if I’m properly understanding your back pressure issues, and please feel free to share any others. I’d love to take them into account as I design the consumer features for Klife.
