The product I work on leans heavily on Kubernetes. We typically expect OS processes to crash and cause pods to be restarted and that hooks into all sorts of monitoring and alerting. Our Elixir app starts a lot of supervised processes. If one of those processes crashes repeatedly, the supervisor jus…

In you Application.start/2 callback you usually return a pid. If that pid stops running (and your application is declared permanent, which is the default) then the application will shut down and issue a shutdown for the whole VM. So if you‘re not seeing failing recovery of supervisors bubble up to …

[image] LostKobrakai: So if you‘re not seeing failing recovery of supervisors bubble up to that root pid (and stop the VM), you probably haven‘t setup your supervision tree correctly. Hmm, good hint. Now that you’ve said that, I kinda feel the culprit is that our pods basically run something …

Maybe superfluous but another way to handle this on kubernetes could be liveness probes. See something like this: GitHub - joeapearson/elixir-k8s-probe: Provides configurable HTTP liveness and readiness endpoints for Elixir applications, intended to be used to support Kubernetes probes. · GitHub Yo…

Oofa, so is there any way to do this? The liveness probes don’t work in our case because there is no “web” aspect to it. The process in question in just a Postgres CDC processor (kinda like a stream processor). Our specific problem is that we process large JSON payloads (up to 800 mb). It takes a …

[image] cjbottaro: So really I wanna be able to mark some process as critical and if it crashes, just exit the whole OS process. Is this possible?? You could tinker with :max_restarts/:max_seconds on supervisors, but depending on the tree depth that might not be the most efficient way to go a…

[image] cjbottaro: Our specific problem is that we process large JSON payloads (up to 800 mb). It takes a long time to process that amount of data. It takes several minutes before it crashes. So the supervisor happily restarts the process ad infinitum. If your task is some kind of batch proce…

We want things like Postgres connection pools (Ecto repos), Xandra connection pools, Redis connections pools, etc… to be started under a supervisor. That’s the kinda stuff we want restarted if there is a transient blip in service to any of those things. So I’m not sure how to say “start up most of …

My 2 points above weren’t meant to be 2 exclusive options, but 2 things to do together, apologies for the confustion. I think I understand your problem better now. [image] cjbottaro: So I’m not sure how to say “start up most of our app under the normal supervision tree, but this one process is…

Looks like that it’s maybe time to switch to a NIF-based JSON parser and/or use a streaming one? Options: GitHub - rusterlium/juicy: A fast JSON parser for Elixir with some extra features · GitHub (last updated 5 years ago though) GitHub - talentdeficit/jsx: an erlang application for consuming, pr…

Elixir Programming Language Forum

Supervisors in a container/pod world

Questions & Help Questions

LostKobrakai October 30, 2022, 8:08am 4

That would be it it seems.