So I have been investigating the feasibility of employing Erlang’s native distributed application architecture (Distributed Applications — Erlang System Documentation v29.0.2) for a project. I have done some research but I do lack hands on experience.
I am mostly interested in built in failover and takeover mechanisms, I think it will work just fine in my case. The problem are network splits.
I will be running the cluster in the cloud (EC2) and, I am afraid, the connectivity between those two nodes is not as guaranteed as it would be if the nodes were connected by a physical cable in the same date center. The connectivity will and does go down on occasion. The default dist_ac’s behavior is less then ideal:
- nodes disconnect from each other
- they keep running as if they were single nodes in the cluster.
To make things even worse, re-joining the cluster afterwards does not resolve the situation and failover / takeover no longer take effect.
I can detect that a node was disconnected from the cluster, but that’s about it I can do. I can’t find a way to recover from this situation - other than shutting down and starting the whole cluster from scratch.
Any recommendations you may have?
