Repository navigation
bug(kubernetes): stale boundary reconciliation never converges for suspended sandboxes #3525
Description
This was generated by AI during triage.
Agent Diagnostic
- Skills loaded:
diagnose, OpenShell repositorycreate-github-issue, and cluster inspection workflows - OpenShell version tested:
0.0.117-dev.211+g4cd5e5478 - Latest release checked:
v0.0.116; the issue reproduces on a newer development build containing the RFC 0012 architecture - Known fixes reviewed: release notes for
v0.0.116, RFC 0012 implementation PR feat(isolation): implement the RFC 0012 sandbox architecture #2942, and the staleSuspendedphase fix in fix(gateway): let Ready win over stale Suspended in derive_phase #2933 - Possible duplicates reviewed: searched all OpenShell issues and merged PRs for
stale boundary,workload fence generation, and the exact warning text; no matching issue was found. Stop→start leaves sandbox stuck at Starting on Agent Sandbox v1beta1 (stale Suspended condition read as Stopped) #2932 concerns phase derivation from an upstream stale condition and is not this reconcile loop. - Findings: seven already-suspended Kubernetes sandboxes with obsolete namespace-fence generations produced exactly seven warnings and seven suspension attempts per reconciliation interval. The gateway emitted 140 identical warnings in 20 minutes with no errors. Current
mainchecks the namespace fence generation before calculating whether the sandbox should run, then unconditionally merge-patches the Sandbox to suspend it when the generations differ. The suspension patch does not make the generation comparison converge. - Remaining reason for filing: the fail-closed transition succeeds, but reconciliation continues performing the same transition indefinitely for terminally suspended objects.
Description
Actual behavior: After the namespace workload fence generation changes, retained Sandbox CRs whose workloads are already suspended remain generation-mismatched. Every reconciliation pass logs:
namespace workload fence generation changed; suspending stale boundary
and calls the dependency-failure suspension path again. In the observed cluster, seven suspended sandboxes generated seven warnings per minute indefinitely—140 warnings in 20 minutes—and prompted repeated Kubernetes merge-patch requests despite already being suspended.
The relevant reconciliation order on current main is:
- Ensure/read the workload NetworkPolicy.
- Compare its namespace-fence generation with the Sandbox CR.
- Suspend and
continueon mismatch. - Only afterward calculate
desired_running.
Consequently, a suspended object cannot reach the later suspended-state reconciliation path when this comparison remains false.
Expected behavior: A namespace-fence mismatch must still fail closed, but the resulting suspended state should converge. Subsequent reconciliation passes should not repeatedly patch an already-suspended Sandbox or emit transition-level warnings unless the state changes or a repair genuinely fails.
Reproduction Steps
- Deploy the RFC 0012 Kubernetes split workload/supervisor architecture.
- Create a sandbox and retain its Sandbox CR, boundary resources, and workspace when suspending it.
- Roll out a namespace workload fence with a new generation, as during a gateway/chart architecture upgrade.
- Wait for more than one gateway reconciliation interval.
- Inspect gateway logs and Sandbox API writes.
- Observe that every interval logs
namespace workload fence generation changed; suspending stale boundaryand re-enters the suspension helper for the same already-suspended Sandbox.
Environment
- OS: GKE nodes running Ubuntu 24.04, Linux amd64
- Kubernetes:
v1.35.7-gke.1222000 - Agent Sandbox controller:
v0.5.0, v1beta1 API - Docker: not applicable; Kubernetes compute driver
- OpenShell:
0.0.117-dev.211+g4cd5e5478 - Deployment: Helm
0.0.0-dev, RFC 0012 separate workload and supervisor pods, Kubernetes NetworkPolicy fence enabled - Latest release checked: yes,
v0.0.116; the tested development build is newer - Possible duplicates checked: yes; no matching issue found
Logs
# Aggregate over 20 minutes from the gateway pod:
stale_boundary_warnings=140
errors=0
# Repeats once per suspended sandbox on every interval:
WARN openshell_driver_kubernetes::driver: namespace workload fence generation changed; suspending stale boundary sandbox_id="<redacted>"Acceptance Criteria
- A stale boundary still fails closed when a namespace workload fence generation mismatch is first detected.
- Once the Sandbox is suspended, subsequent reconciliation passes converge without repeated suspension patches.
- Transition-level warnings are emitted once per state transition or are otherwise rate-limited/aggregated, not once per object per interval forever.
- Active sandboxes with mismatched workload or supervisor generations retain the existing fail-closed behavior.
- A Kubernetes regression test runs multiple reconciliation passes against an already-suspended, generation-mismatched Sandbox and verifies stable state without repeated mutations.
Activity
drew commentedon Sep 22, 2026
Implementation Plan
Issue type:
fixComplexity: Low
Confidence: High — clear path
Summary
Make namespace-fence generation reconciliation state-aware: active or bootstrapping sandboxes still fail closed on a stale fence, while a stable sandbox whose desired state is already suspended treats that transition as converged and does not emit another transition warning or PATCH. Add request-observing Kubernetes-client tests that run reconciliation more than once and prove both convergence and preserved active fail-closed behavior.
Scope
crates/openshell-driver-kubernetes/src/driver.rs: derive desired-running/bootstrap state before the namespace-fence generation branch, gate the suspension transition appropriately, and add lifecycle predicate plus mocked Kubernetes API reconciliation coverage.Implementation Steps
Test Plan
operatingMode, v1alpha1replicas, and bootstrap preservation incrates/openshell-driver-kubernetes/src/driver.rs.tower::service_fnKubernetes client pattern indriver.rs; execute two reconciliation passes for a suspended stale boundary and inspect/count mutation requests, then verify active stale boundaries still suspend.Risks & Open Questions
Documentation Impact
/proc, binary execution, or inter-process visibility and behaves the same under SELinux and AppArmor.Revision 1 — initial plan
drew commentedon Sep 22, 2026
Implementation Complete
PR: #3570
What was built
Kubernetes reconciliation now treats stale namespace-fence generations as actionable only while a sandbox is running or bootstrapping. Stable suspended sandboxes converge without repeated warning or mutation, while active stale boundaries continue to fail closed.
Tests
mise run testpassedDocs updated
The issue will auto-close when the PR is merged.
github-actions commentedon Oct 7, 2026
This issue has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity. Comment or remove the state:stale label to keep it open.