Best options for Elixir-only persistent distributed cache?

Depends on how you deploy.

You can build a distributed cache that synchronizes agressively, and when you shut down a node to roll a new node, the stopping node stops accepting external input but will wait till all its currently processing cache items are synced on other nodes, and the new node will not start to accept input before being in sync with all other nodes.

k8s can help with that, but it’s probably a lot of pain to do properly.

Otherwise you can use PubSub : all cache writes go to pubsub and are distributed to all other nodes. When a new node start, it subscribes to pubsub without processing messages yet, it selects another node and gets a full copy of the cache to bootstrap the cache locally, and then processes incoming messages. Writes must be time-stamped to ignore old writes (aso because the node you are copying from is already listening to messages sent during the boostrap sequence and the new node will receive them too. If you start listening after copying the cache you will miss some messages.).

All of this looks hard to get right, are you sure you cannot use Redis?