- Have you built systems where processes live for days/weeks with accumulated state?
I have experience with this. For a GenServer with a long-lived session, it’s quite easy to keep the accumulated state in the process. However, if the GenServer crashes, it cannot recover its state if that state only exists in the process. In that case, the worker will simply restart with the initial state from init/1.
For simple cases, I would move the important state to an ETS table or some kind of database so that the process can recover it after a crash.
- How do you handle hot code changes in production for stateful GenServers?
In production, this can be quite complicated because you need to handle state migration between OTP releases.
IMO, I would avoid running the application and Phoenix on the same VM/node. Instead, move the long-lived/stateful processes to another node so they can run independently from the Phoenix node.
Then you can use hot code upgrades more easily. When the state structure changes, you can migrate the existing state in the code_change/3 callback while upgrading the code.
