Skip to content

fix(controller): prune superseded inactive worker versions - #577

Open
mitchross wants to merge 2 commits into
temporalio:mainfrom
mitchross:fix/prune-superseded-inactive-versions
Open

fix(controller): prune superseded inactive worker versions#577
mitchross wants to merge 2 commits into
temporalio:mainfrom
mitchross:fix/prune-superseded-inactive-versions

Conversation

@mitchross

@mitchross mitchross commented Sep 9, 2026

Copy link
Copy Markdown

A rollout superseded before its version becomes Current or Ramping stays Inactive forever. The controller scales its Deployment to zero, but only Drained and NotRegistered versions enter deletion, leaking both the Kubernetes Deployment and Temporal version record. This is the Inactive case discussed in #498.

Extend the existing controller deletion path:

  • Nominate superseded Inactive versions only after Kubernetes has observed zero desired/actual replicas and no terminating replicas; protect current and target versions.
  • Count running pinned workflows for the exact deployment version using both legacy dot and current colon encodings. Errors, missing responses, and nonzero counts retain the version.
  • Use normal DeleteVersion without skipping drainage. Temporal refusal retains the Deployment for retry; NotFound permits Kubernetes cleanup.
  • Retain rendered resources when an Inactive version cannot be deleted.

No CRD or new retirement clock is introduced. The existing sunset delays are defined for Drained versions. Unused Inactive versions are removed after the checks above succeed.

The documented operational limit is explicit: stop sending pinned overrides to versions being retired. Visibility is eventually consistent and the count is not atomic with deletion. Temporal's existing Drained status also does not account for newly pinned overrides. This change does not claim to provide a server-side exclusion against concurrent workflow starts or override updates.

Validation:

  • go test ./... -count=1 with envtest assets and Helm dependencies passes.
  • Targeted integration tests against Kubernetes envtest and real Temporal 1.31.2 pass: superseded never-active version removal, pinned override protection followed by removal after closure, and normal Drained sunset.
  • The same unused-version integration test fails on the unmodified controller after 90 seconds because its Deployment remains.
  • Unit coverage includes observed/terminating replicas, current/target guards, visibility failures, server refusal, NotFound retry, and retained rendered resources.

The complete TestIntegration suite passes (475 seconds), covering Manual, AllAtOnce, Progressive, rollback, version limits, resource templates, and deletion. No production deployment has occurred.

@CLAassistant

CLAassistant commented Sep 9, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@mitchross mitchross changed the title fix(controller): prune superseded inactive versions fix(controller): prune superseded inactive worker versions Sep 10, 2026
@mitchross
mitchross marked this pull request as ready for review September 10, 2026 22:10
@mitchross
mitchross requested review from a team, eniko-dif and jlegrone as code owners September 10, 2026 22:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants