bug(ci): Kubernetes conformance starts before controller reconciliation recovers #4675

User Story

As an OpenShell contributor validating Kubernetes sandboxes, I need conformance to start after the workload controller can reconcile resources, so startup recovery does not fail unrelated scenarios.

Problem Statement

The installed-artifact conformance playbook waits for gateway connectivity after each guest boot, but does not check Agent Sandbox reconciliation. Gateway connectivity and controller Deployment availability do not establish that the controller can create workload Pods.

The first sandbox creation timed out after 30 seconds waiting for its gated workload Pod in PR #4320's run. The other seven scenarios passed. An independent PR #4549 run showed the same first-create timeout. The unchanged #4320 rerun passed.

A fresh local tmachine boot reproduced a controller recovery gap. The controller exited on Kubernetes API certificate verification, restarted twice, and then waited for leadership. A reconciliation probe waited 10.9 seconds after gateway connectivity before observing its Pod. This establishes a startup preparation gap, but does not prove the original CI failure had the same exact cause.

Impact / Why This Matters

Startup recovery can fail the first conformance scenario before its assertions run. Rerunning the lane is the current workaround, but consumes CI resources and does not establish readiness. Existing diagnostics omit controller state and logs, and late collection can lose startup evidence.

Acceptance Criteria

• Kubernetes conformance verifies controller-owned Pod creation after each guest boot before running scenarios.
• The readiness check is bounded and its temporary Pod cannot run or pull an image.
• Success and failure clean up the probe; readiness failure skips conformance and collects diagnostics.
• Diagnostics retain controller state, leader leases, events, and bounded startup and recent controller logs.
• Relevant installed-artifact Kubernetes conformance passes without changing production provisioning deadlines or ownership checks.

Reproduction Steps

1. Prepare OpenShell's installed-artifact inputs for the host architecture.
2. Run nix run .#tmachine -- test ubuntu-k3s k3s conformance to install OpenShell and boot a fresh test guest.
3. Compare gateway connectivity with controller restart state, leader leases, and first workload Pod creation. The delay is intermittent.

Environment

• Tested OpenShell artifacts: be9d7a0561cfd55b1ca426f0217e283a4e28b6b8.
• Deployment: OpenShell Helm installation on K3s v1.29.15+k3s1, Agent Sandbox v0.5.0.
• Local reproduction: Ubuntu 24.04 ARM64 guest under tmachine/QEMU/KVM.
• Original CI: Ubuntu K3s x86-64 lane.