GenStage: Bounded buffering and serialization

Consider each stage as a unit of concurrency / fault tolerance / back-pressure rather than of code organization. As as extreme, imagine you need to perform three operations, a, b and c on every event in the system. Putting each operation in a separate GenStage that are connected together is not the way to go. If your goal is code isolation, then Elixir modules and functions are the proper tool.

The reason I am saying this is that I am not sure I would have the buffer as separate stage. It could simply be part of the producer. I also wouldn’t necessarily mix all producers and consumers together, especially because items from the same producer needs to be processed in order.

From your description, the simplest solution seems to be N GenServers, one for each kind of producer that keeps the buffer and processes the events as they come. The processing can happen on Task. Alternatively, you can keep N pairs of GenStage, with a single producer and a single consumer, connected directly, each pair isolated from each other.