I had a similar situation a long time ago, and I solved it by doing an adaptive buffering with two processes. One process enqueues incoming items, while another performs the write. The writer reports to the queue that it’s available. Then, as soon as the first item arrives, the queue process sends it to the writer, and marks that the writer is busy.
Now, while the writer is busy, the queue stores new items in its internal structure. Then, when the writer is done and reports back to the queue, the queue can send all the items to the writer at once, and the writer can store them at once. In this way, the writer is profiting from the fact that writing N items at once is way faster than N writes of a single item.
I’m not familiar with GenStage, but I feel that this should be doable using two stages (consumer-producer for the queue, and consumer for the writer).
The nice benefit of this approach is that buffering is adaptable. If writing is faster than the rate of incoming messages, then there’s no buffering. Otherwise, the buffer will expand to accommodate the incoming rate.
Another nice benefit is that in the queue process you can do all sorts of trickery to handle overload, such as remove old items when the queue becomes large, reject new items if the queue is too large, or eliminate duplicate requests. I did all of those things in that project, and the result was pretty stable and resilient against all kinds of bursts, failures, latency increases and other problems.
Again, I’m not familiar with GenStage, so not sure if there’s an out-of-the-box solution for that, or you need to work a bit for it, but I believe that it should be possible with GenStage.
