Skip to content

fix(ci): wait for controller reconciliation before Kubernetes conformance - #4676

Open
pimlock wants to merge 1 commit into
mainfrom
4675-kubernetes-conformance-startup/pimlock
Open

pimlock wants to merge 1 commit into
mainfrom
4675-kubernetes-conformance-startup/pimlock

Conversation

@pimlock

@pimlock pimlock commented Oct 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

Kubernetes conformance currently starts when the gateway connects, even if Agent Sandbox is still recovering after guest boot. This can fail the first sandbox creation before its scenario assertions run. Wait for controller-owned Pod creation before starting conformance, and preserve controller startup evidence when preparation or tests fail.

Related Issue

Closes #4675.

Changes

  • Verify reconciliation with a temporary Sandbox and its controller-owned Pod. A scheduling gate prevents the Pod from running or pulling an image.
  • Retry temporary API failures, remove the probe on success or failure, and skip conformance with diagnostics if preparation fails.
  • Collect controller state, leader leases, events, and bounded current and previous logs. Retain both startup and recent log sections.
  • Document test preparation and diagnostics. Production provisioning deadlines and ownership checks remain unchanged.

Testing

  • Full nix run .#tmachine -- test ubuntu-k3s k3s conformance: all eight archived tests passed in 298.4 seconds on an ARM64 Ubuntu 24.04 guest, K3s 1.29.15, and Agent Sandbox 0.5.0. Runtime artifacts came from be9d7a0561cfd55b1ca426f0217e283a4e28b6b8; the conformance sources match this branch. The probe waited 10.9 seconds through real controller recovery, then cleaned up.
  • Isolated checks for delayed leadership, stopped-controller cleanup and conformance skipping, and retry after a temporary apply failure.
  • Final diagnostic collector executed in the VM; a 400 KB log fixture retained startup/recent markers and command failure status within the output limit.
  • Ansible syntax, ShellCheck, and diff whitespace checks.
  • mise run pre-commit.
  • mise run ci: failed in the unchanged VM-driver test driver::tests::stopping_a_sandbox_lets_the_supervisor_drain_before_closing_liveness, at the fixture's child.try_wait().unwrap().is_none() assertion. The test also failed when rerun alone. This PR does not modify VM-driver code.

The local VM reproduced a controller startup certificate error followed by restart and leader election. It does not establish the original CI failure's exact cause. Original CI used x86-64; local VM validation used ARM64.

Checklist

  • Follows Conventional Commits.
  • Commits are signed off for DCO.
  • Documentation updated; runtime architecture is unchanged.

…ance

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(ci): Kubernetes conformance starts before controller reconciliation recovers

1 participant