Leaving a BEAM cluster (basically) unattended for years - Yuri Oliveira | ElixirConf US

[ElixirConf]

Leaving a BEAM cluster (basically) unattended for 2 years - Yuri Oliveira | ElixirConf US 2025

Leaving a BEAM cluster (basically) unattended for 2 years - Yuri Oliveira | ElixirConf US 2025

This talk was recorded at ElixirConf US 2025. If you’re curious about our upcoming event, check https://elixirconf.com✨
We implemented one of our services as an Erlang cluster with a team of 2 and then left it basically unattended for the next 2 years with a team of one that wasn’t on the initial team. What happened? How do we monitor it? Is it growing by itself?

As mentioned in previous conference talks, the cluster runs SQLite and Elixir to deal with video stream data mapping, and it had 53 nodes on 2023-07-10.

It’s now made of 100+ nodes, 25k+ live streams, and is still running SQLite.

In the talk, we’ll talk about maintenance challenges and how hard it is to keep such a cluster up and running.

Outline

• Recap why we moved from Cassandra to SQLite
• State of the Art on 2023-07-10

• of streams

• of nodes

• of writes, reads per second

• team size
• Team changed
• OOM kills
• Netsplits
• PG connections going down when the Cloud provider reboots the DB
• State of the Art in 2025
• same metrics as above
• Challenges
• Updating OS (Ubuntu LTS 20 to 24)
• SOC 2
• Updating Elixir/OTP versions (1.13.4 now)
• Keeping costs low (storage, compute usage)

Comments welcome! View the elixirconf tag for more ElixirConf talks!