I largely have similar experience with Kubernetes in particular to you in Elixir. It gets even more tricky if you have autoscaling and the pods get resized horizontally and vertically.
I tend to report on an ongoing basis, from each node, the amount of nodes you see in the cluster. We report that using a custom Telemetry metric that gets sent to DataDog. If different pods keep reporting different number, it means something is wrong and we raise an alert. I think it can be further automated to mark the pod as broken using liveness probe and get it to shut down/restart in such case.
I don’t think the PubSub should be used to send any mission-critical information. We pretty much use it nowadays to send the message the “something happened, the UI must be updated”. We don’t send important payloads over the wire, rather than we send just a notice. And on top of that we have polling, so that if a message is in fact missed, a scheduled refresh will happen in worst case scenario in 30s or so and the UI will update anyway to the user, just with a slight delay.
We use Oban more and more. Oba Pro can replace Quantum, and we have not been seeing the same problems we had where duplicate scheduled jobs are executed. Oban does not rely on cluster being established, instead it synchronizes everything through PostgreSQL, to ensure that the job executes on only one of the nodes at a time.
We have observed quite a big delay between when the pod starts, and when it connects to the cluster. We don’t want Oban jobs to be executed in that time, and we also don’t want some other things to happen. Instead of starting Oban in the application.ex, we start a custom supervisor in application callback module, and a watchdog GenServer process. The Watchdog process checks if we see successfull connection to database, and if we managed to connect to the cluster, by checking if we see other nodes. Only then we start Oban and some other processes to increase the likelihood that the messages sent out from background jobs do not end up in “ether” as you said.
We got rid of all the cluster-unique GenServer processes, be it :global , or through Swarm or Horde, this has proven to be too much of a hassle and too unstable in a multi-node, quite rapidly changing number of pods connected over cloud connection. Stuff like passing over data was unreliable and we never got it 100% right, instead again we rely on Oban unique jobs for the most part and centralised communication via Postgres.
This is probably not what you were looking for in terms of answer but that’s what I figured out works in an enviroment like Kubernetes with autoscaler, which is quite unpredictable. The Erlang clustering implementation is really the best suited to several servers connected through a cable directly. I believe folks who are serious about having multi-dozen-nodes of Erlang clustered doing mission-critical stuff have their own clustering layers, that’s what WhatsApp does and others I believe.
