Skip to content

kubernetes: add opt-in backoff after a failed poll - #222

Open
ceolinrenato wants to merge 1 commit into
bitwalker:mainfrom
ceolinrenato:kubernetes-poll-backoff
Open

ceolinrenato wants to merge 1 commit into
bitwalker:mainfrom
ceolinrenato:kubernetes-poll-backoff

Conversation

@ceolinrenato

@ceolinrenato ceolinrenato commented Nov 4, 2026 •

Copy link
Copy Markdown

Summary of changes

The "why": when the Kubernetes API server is degraded, polling it at the fixed :polling_interval keeps adding load exactly when it can least afford it. We hit this in production: our pods run different sets of OTP applications on their Erlang nodes, so we had one topology per application and N pollers on the same interval. A failing control plane got worse because every one of them kept retrying at the fixed interval, and recovery took longer than the outage itself.

We now run a custom strategy that does a single poll for all of them and backs off after a failure, among other things. The backoff is generic and does not depend on the rest of that strategy, so I thought it was worth contributing back upstream. It does not change the cost of a single poll, it just gives the control plane room to recover between them.

The change is opt-in so nothing changes for existing configurations:

  • kubernetes_poll_backoff unset (default): fixed interval, identical to today.
  • kubernetes_poll_backoff: [base_ms: 100, max_ms: 30_000, jitter: 0.2]: capped exponential backoff with jitter added after each failed poll.
  • true uses the defaults (50 / 60_000 / 0.2).

A failed poll already preserves the last known membership; this only changes when the next poll happens. The attempt counter travels on the scheduled message, so State.meta keeps its current shape and no public struct changes.

The curve lives in Cluster.Strategy.Kubernetes.Backoff (same internal-module pattern as TLS and EndpointSlice) so it can be unit tested without sleeping.

Checked locally against CONTRIBUTING.md: mix format --check-formatted, mix deps.unlock --check-unused, mix hex.audit, test and prod compiles with --warnings-as-errors, mix test --warnings-as-errors (171 passing), mix docs --warnings-as-errors and mix hex.build.

Separate idea: selectors for heterogeneous node basenames

Not part of this PR. It comes out of the same incident as the backoff, but it is a separate idea and a larger change.

kubernetes_node_basename is fixed per topology, so a deployment where pods run different sets of OTP applications on their Erlang nodes needs one topology per application. That is what turned one poller into N during the incident above. An alternative would be to let a selector drive a heterogeneous topology: select every application at once and derive each node's basename from a label on the pod.

# One selector covers every application; the node basename comes from a pod label.
pods = list_pods(namespace, label_selector: "app in (web,jobs,grpc,kafka)")

nodes =
  for pod <- pods, ready?(pod) do
    basename = pod.labels["app"]
    :"#{basename}@#{pod.ip}"
  end

One topology and one poll would then cover several basenames, instead of one topology and one poller per application. This needs its own design discussion, so I'm only raising the idea here. #65 proposed a similar thing at the config level and was closed unmerged; linking it as prior art.

Checklist

  • New functions have typespecs, changed functions were updated
  • Same for documentation, including moduledocs
  • Tests were added or updated to cover changes
  • Commits were squashed into a single coherent commit
  • Notes added to CHANGELOG file which describe changes at a high-level

Polling a degraded Kubernetes API server at a fixed interval keeps adding load exactly when it can least afford it, which slows recovery. Add an opt-in kubernetes_poll_backoff option that applies capped exponential backoff with jitter after a failed poll, keeping the last known membership in the meantime. Without the option the strategy keeps the fixed polling interval.

The backoff curve lives in Cluster.Strategy.Kubernetes.Backoff so it can be tested without sleeping.
@ceolinrenato
ceolinrenato marked this pull request as draft November 4, 2026 20:35
@ceolinrenato
ceolinrenato marked this pull request as ready for review November 4, 2026 20:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant