Skip to content

fix(ci): restore default e2e coverage with a Kubernetes recovery waiver - #4424

Open
elezar wants to merge 3 commits into
mainfrom
codex/e2e-conformance-only/el
Open

elezar wants to merge 3 commits into
mainfrom
codex/e2e-conformance-only/el

Conversation

@elezar

@elezar elezar commented Oct 12, 2026 •

Copy link
Copy Markdown
Member

Summary

Restore the default Rust E2E coverage when CI exports an empty feature override, and use e2e-conformance-only for intentional conformance-only runs. Temporarily exclude only canonical_main_connect_recovers_its_ssh_transport in the Kubernetes nextest profile while #4445 investigates input-owner cleanup after reconnect. Keep the existing Kubernetes port-forward transport and the test's coverage on other drivers.

The empty-feature behavior underlying this CI coverage gap was introduced by #2925, merged on September 1, 2026. That PR changed the runners from ${VAR:-default} to ${VAR-default} and made an empty feature list skip Rust E2E tests after conformance. #2260, merged on September 7, then wired the Kubernetes workflow's empty default feature input into that override. Together, these changes caused the default Kubernetes lane to run conformance only while reporting success.

Related Issue

No separate implementation issue is required for this localized CI fix, which does not change product behavior or public APIs. The missed coverage is visible in this Kubernetes job.

Temporary Kubernetes-only test waiver: #4445. This PR does not close the investigation.

Changes

  • Treat unset and empty feature overrides as driver defaults. Support e2e-conformance-only in the Docker, Podman, Kubernetes, and VM runners and consume it before invoking Cargo.
  • Migrate the intentional Docker, Podman, and Kubernetes external-driver conformance-only tasks to the explicit selector. Keep the VM external-driver task running conformance and Rust tests.
  • Add an exact-name default-filter exclusion for the one recovery test in profile.e2e-kubernetes. Keep the other 96 restored default Kubernetes tests enabled, along with recovery coverage on other driver profiles.
  • Document the selector and the temporary waiver in TESTING.md; update workflow input help. No kind configuration, Service type, or gateway transport changes are included.

Testing

  • mise run pre-commit on the final branch.
  • Nextest list validation against an isolated fixture confirms the exact recovery name is excluded only in e2e-kubernetes; a similarly named neighbor and an ordinary lifecycle test remain selected. The recovery name is selected under both ci and default profiles.
  • Mocked selector dispatch for all four drivers covers unset, empty, conformance-only, default, custom features, and conformance failure propagation. VM checks exercise dispatch after provisioning.
  • Shell syntax, task selector parsing, and git diff --check.
  • Final GitHub Kubernetes run pending: expected default selection is 96 tests, with the recovery test recorded as skipped.
  • Full local mise run ci: the earlier attempt with fresh Go caches hit Rust test linking because system Z3 is unavailable (ld: library 'z3' not found).

Checklist

  • Follows Conventional Commits.
  • Commits are signed off for DCO.
  • Testing documentation updated.

@github-actions

Copy link
Copy Markdown

@elezar elezar added the test:e2e Requires end-to-end coverage label Oct 12, 2026
@github-actions

Copy link
Copy Markdown

Label test:e2e applied for 63699cd. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

@elezar
elezar changed the base branch from main to codex/conformance-after-parity October 12, 2026 14:57
Base automatically changed from codex/conformance-after-parity to main October 12, 2026 15:46
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar elezar changed the title fix(ci): make conformance-only e2e selection explicit fix(ci): restore default e2e coverage and stabilize kind connections Oct 12, 2026
@elezar
elezar enabled auto-merge October 12, 2026 20:33
Signed-off-by: Evan Lezar <elezar@nvidia.com>
@elezar elezar changed the title fix(ci): restore default e2e coverage and stabilize kind connections fix(ci): restore default e2e coverage with a Kubernetes recovery waiver Oct 13, 2026

@matthewgrossman matthewgrossman left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I verified the restored coverage in the latest CI run: both Kubernetes jobs (v1beta1 and v1alpha1) report 96 tests run: 96 passed, 4 skipped, including the single profile.e2e-kubernetes.default-filter exclusion. The "Final GitHub Kubernetes run pending" testing item can be checked off.

I also confirmed that the intentional conformance-only lanes still run only conformance. The workflow exports an empty OPENSHELL_E2E_KUBERNETES_FEATURES, but the mise task's e2e-conformance-only value takes precedence; the kubernetes-external-driver-e2e log shows conformance targets only. No other workflow or task relies on the old empty-value behavior.

Optional: the branch predates #4081, so this run did not include the new k3s-ha conformance lane. The changes don't overlap, but a rebase would give a green run alongside it before merge.

Non-blocking suggestions inline.

run_e2e() {
run_conformance
if [ -z "${E2E_FEATURES}" ]; then
if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the original gap was a lane that skipped its Rust tests while reporting success. With this change the default is correct, but a future workflow or task change could narrow the run again without anyone noticing. Could we make the skip visible here and in the other three runners? For example, echo "::notice::Skipping Rust e2e tests (e2e-conformance-only)". A follow-up could also fail default lanes that execute zero Rust tests. The mocked selector checks described in the PR body would also be valuable as a committed test.

Comment thread TESTING.md
| `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility |
| `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar |

The Docker, Podman, Kubernetes, and VM runners execute CLI

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this paragraph covers all four runners, but it sits under the Kubernetes task-variants table, where Docker, Podman, and VM users are unlikely to look. Consider moving it to a shared e2e section.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:e2e Requires end-to-end coverage

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants