One opportunity for us is to make service similar to Zookeeper. Consider the following diagram which depicts how Kafka and Zookeeper interact, and it is similar for other systems such as Apache Storm.
Brokers, producers and consumers use Zookeeper to manage and share state
It is a type of coordination service and it’s exactly what OTP should take care of for us. I was thinking about @cjbottaro’s Tempest library. Once this library wants to scale across many machines, it will need some coordination between the processes in order to handle failures and retries and so on.
The key here is that the business need will drive the innovation; or, “scratch your own itch.” That is, if we can start to solve real problems by brewing our own Elixir data processing systems, then we have a real project to work from.
