KubernetesProbes - liveness/readiness probes for Phoenix, built on OTP's shutdown sequence

News Announcing
, ,
1

We built this at Pandosearch to replace an older library of ours, traffic_drain_plug, after a review turned up a handful of real bugs in it. We also realised it was ~260 lines wrapping graceful_stop to reimplement machinery the BEAM already ships for free. More detail on exactly what was wrong and why is in the traffic_drain_plug README - we kept that repo (deprecated) around specifically as a writeup of what was wrong.

kubernetes_probes is the replacement, built directly on OTP’s native shutdown sequence instead of a custom signal handler and mutable hook list. Two pieces:

  • KubernetesProbes.Plug — first plug in your endpoint. Answers /probe/liveness (200 while up) and /probe/readiness (200 while ready, 503 while draining or not ready).
  • KubernetesProbes.Drainer — a GenServer to be added as the last child of your application supervisor. On SIGTERM, OTP tears down children in reverse start order, so this terminates first. Its terminate/2 flips readiness to 503 via :persistent_term and sleeps for the drain window then OTP terminates your Endpoint, Repo, etc.

Usage

# mix.exs
{:kubernetes_probes, "~> 0.1"}

# lib/my_app/application.ex
children = [
  MyApp.Repo,
  MyAppWeb.Endpoint,
  # Must be last - terminates first on shutdown
  {KubernetesProbes.Drainer, wait: 20_000}
]

# lib/my_app_web/endpoint.ex
plug KubernetesProbes.Plug

# optional: gate readiness on something real, e.g. DB connectivity
plug KubernetesProbes.Plug, ready?: &MyApp.repos_ready?/0

Point your Deployment’s probes at /probe/liveness and /probe/readiness (paths can be configured) and set terminationGracePeriodSeconds to the drain window plus a few seconds of slack (30s covers the 20s default).

We have been running this in production for a while and are happy with the result, apart from one bug we found where an internal client kept the connection open (hence the 0.1.1 release).

Currently at 0.1.1 - MIT-0, on Hex and GitLab. Happy to hear if this is useful for anyone else’s setup. Feedback, bug reports or MRs more than welcome.

2 Likes