Skip to content

fix(ci): restore default e2e coverage with a Kubernetes recovery waiver #4424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
elezar wants to merge 3 commits into
base: main
Choose a base branch
from
Open

fix(ci): restore default e2e coverage with a Kubernetes recovery waiver #4424

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
930f026
fix(ci): make conformance-only e2e selection explicit
elezar Oct 12, 2026
2a48390
fix(ci): use mapped NodePort for kind gateway sessions
elezar Oct 12, 2026
6c13fca
fix(ci): waive SSH transport recovery only on Kubernetes
elezar Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .config/nextest.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ failure-output = "immediate-final"
slow-timeout = { period = "30s", terminate-after = 4 }

[profile.e2e-kubernetes]
# Temporary waiver: port-forward reconnect can attach read-only while the old
# SSH session retains stdin ownership. Keep recovery coverage on other drivers.
# Remove this exclusion after resolving https://github.com/NVIDIA/OpenShell/issues/4445.
default-filter = "not test(/^canonical_main_connect_recovers_its_ssh_transport$/)"
fail-fast = false
status-level = "slow"
final-status-level = "fail"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/e2e-kubernetes-test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ on:
type: string
default: ""
kubernetes-features:
description: "Cargo feature list for the Kubernetes e2e crate (uses script defaults when empty)"
description: "Cargo feature list for Kubernetes e2e (defaults when empty; e2e-conformance-only skips Rust tests)"
required: false
type: string
default: ""
Expand Down
18 changes: 18 additions & 0 deletions TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,6 +336,12 @@ mise run e2e:kubernetes:credential-drivers

### Kubernetes E2E (`e2e/rust/e2e-kubernetes.sh`)

The `e2e-kubernetes` nextest profile temporarily excludes only
`canonical_main_connect_recovers_its_ssh_transport` while
[#4445](https://github.com/NVIDIA/OpenShell/issues/4445) investigates delayed SSH
input-owner release through port-forwarding. The test remains available to other
driver profiles; the remaining Kubernetes tests stay enabled.

Kubernetes e2e tests deploy an OpenShell gateway into a real Kubernetes cluster
via Helm and run the Rust e2e suite against it. On vanilla Kubernetes the harness
reaches the gateway through `kubectl port-forward`; on OpenShift it instead uses a
Expand Down Expand Up @@ -476,6 +482,18 @@ Available task variants:
| `e2e:kubernetes:v1alpha1` | Agent Sandbox v1alpha1 compatibility |
| `e2e:kubernetes:external-driver` | External Kubernetes driver sidecar |

The Docker, Podman, Kubernetes, and VM runners execute CLI

@matthewgrossman matthewgrossman Oct 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this paragraph covers all four runners, but it sits under the Kubernetes task-variants table, where Docker, Podman, and VM users are unlikely to look. Consider moving it to a shared e2e section.

conformance before their Rust tests. Set the driver's feature variable
(`OPENSHELL_E2E_DOCKER_FEATURES`, `OPENSHELL_E2E_PODMAN_FEATURES`,
`OPENSHELL_E2E_KUBERNETES_FEATURES`, or `OPENSHELL_E2E_VM_FEATURES`) to
`e2e-conformance-only` to skip the Rust tests. Unset or empty values select
the driver's default Rust features. The selector is handled by the runner;
it is not a Cargo feature.

```shell
OPENSHELL_E2E_KUBERNETES_FEATURES=e2e-conformance-only mise run e2e:kubernetes
```

Kubernetes e2e environment variables:

| Variable | Purpose |
Expand Down
7 changes: 4 additions & 3 deletions e2e/rust/e2e-docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ set -euo pipefail

ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
E2E_TEST="${OPENSHELL_E2E_DOCKER_TEST:-}"
E2E_FEATURES="${OPENSHELL_E2E_DOCKER_FEATURES-e2e-docker}"
# Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests.
E2E_FEATURES="${OPENSHELL_E2E_DOCKER_FEATURES:-e2e-docker}"
DEFAULT_WORKLOAD_MANIFEST="${ROOT}/e2e/gpu/images/.build/workloads.yaml"
RUN_WITH_GATEWAY_COMMAND="__openshell_run_docker_e2e"
# shellcheck source=e2e/support/conformance.sh
Expand All @@ -20,7 +21,7 @@ source "${ROOT}/e2e/support/conformance.sh"
run_e2e() {
e2e_run_openshell_conformance "Docker"

if [ -z "${E2E_FEATURES}" ]; then
if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then
return 0
fi

Expand All @@ -44,7 +45,7 @@ if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then
exit 0
fi

if [ -n "${E2E_FEATURES}" ]; then
if [ "${E2E_FEATURES}" != "e2e-conformance-only" ]; then
CONTAINER_ENGINE=docker bash "${ROOT}/tasks/scripts/e2e-build-workload.sh"
fi

Expand Down
5 changes: 3 additions & 2 deletions e2e/rust/e2e-kubernetes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,8 @@ RUN_WITH_GATEWAY_COMMAND="__openshell_run_kubernetes_e2e"
# shellcheck source=e2e/support/conformance.sh
source "${ROOT}/e2e/support/conformance.sh"

E2E_FEATURES="${OPENSHELL_E2E_KUBERNETES_FEATURES-e2e,e2e-host-gateway,e2e-kubernetes}"
# Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests.
E2E_FEATURES="${OPENSHELL_E2E_KUBERNETES_FEATURES:-e2e,e2e-host-gateway,e2e-kubernetes}"

# Fixed output path of the `e2e-kubernetes` nextest profile (`.config/nextest.toml`).
JUNIT_XML="${ROOT}/results/e2e-kubernetes.xml"
Expand Down Expand Up @@ -90,7 +91,7 @@ run_suite() {

run_e2e() {
run_conformance
if [ -z "${E2E_FEATURES}" ]; then
if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then

@matthewgrossman matthewgrossman Oct 14, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: the original gap was a lane that skipped its Rust tests while reporting success. With this change the default is correct, but a future workflow or task change could narrow the run again without anyone noticing. Could we make the skip visible here and in the other three runners? For example, echo "::notice::Skipping Rust e2e tests (e2e-conformance-only)". A follow-up could also fail default lanes that execute zero Rust tests. The mocked selector checks described in the PR body would also be valuable as a committed test.

return 0
fi

Expand Down
7 changes: 4 additions & 3 deletions e2e/rust/e2e-podman.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ set -euo pipefail
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
E2E_TEST="${OPENSHELL_E2E_PODMAN_TEST:-}"
E2E_TEST_SET="${OPENSHELL_E2E_PODMAN_TEST_SET:-}"
E2E_FEATURES="${OPENSHELL_E2E_PODMAN_FEATURES-e2e-podman}"
# Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests.
E2E_FEATURES="${OPENSHELL_E2E_PODMAN_FEATURES:-e2e-podman}"
DEFAULT_WORKLOAD_MANIFEST="${ROOT}/e2e/gpu/images/.build/workloads.yaml"
RUN_WITH_GATEWAY_COMMAND="__openshell_run_podman_e2e"
# shellcheck source=e2e/support/conformance.sh
Expand Down Expand Up @@ -71,7 +72,7 @@ fi

if [ "${1:-}" = "${RUN_WITH_GATEWAY_COMMAND}" ]; then
e2e_run_openshell_conformance "Podman"
if [ -z "${E2E_FEATURES}" ]; then
if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then
exit 0
fi

Expand All @@ -97,7 +98,7 @@ if [ -n "${OPENSHELL_GATEWAY_ENDPOINT:-}" ] && [ -z "${OPENSHELL_BIN:-}" ]; then
export OPENSHELL_BIN="${ROOT}/target/debug/openshell"
fi

if [ -n "${E2E_FEATURES}" ]; then
if [ "${E2E_FEATURES}" != "e2e-conformance-only" ]; then
CONTAINER_ENGINE=podman bash "${ROOT}/tasks/scripts/e2e-build-workload.sh"
fi

Expand Down
6 changes: 5 additions & 1 deletion e2e/rust/e2e-vm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,8 @@ GATEWAY_BIN="${OPENSHELL_GATEWAY_BIN:-${ROOT}/target/debug/openshell-gateway}"
DRIVER_BIN="${OPENSHELL_VM_DRIVER_BIN:-${ROOT}/target/debug/openshell-driver-vm}"
CLI_BIN="${OPENSHELL_BIN:-${ROOT}/target/debug/openshell}"
E2E_TEST_OVERRIDE="${OPENSHELL_E2E_VM_TEST:-}"
E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES-e2e-vm}"
# Unset or empty selects Rust defaults; e2e-conformance-only skips Rust tests.
E2E_FEATURES="${OPENSHELL_E2E_VM_FEATURES:-e2e-vm}"
SANDBOX_IMAGE="${OPENSHELL_SANDBOX_IMAGE:-nvcr.io/nvidia/base/ubuntu:24.04}"

# The VM driver places `compute-driver.sock` under `[openshell.drivers.vm].state_dir`.
Expand Down Expand Up @@ -383,6 +384,9 @@ e2e_export_gateway_restart_metadata \
export OPENSHELL_PROVISION_TIMEOUT="${SANDBOX_PROVISION_TIMEOUT}"

e2e_run_openshell_conformance "VM"
if [ "${E2E_FEATURES}" = "e2e-conformance-only" ]; then
exit 0
fi

# Seed the catalog once for the whole lane. The profiles live in the gateway
# for as long as it runs, and the import is create-only: a second import of
Expand Down
6 changes: 3 additions & 3 deletions tasks/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -247,12 +247,12 @@ run = "bash e2e/no-compute-driver-gateway.sh"

["e2e:docker:external-driver"]
description = "Run Docker conformance with a driver-free gateway and external Docker driver binary"
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_DOCKER_FEATURES = "" }
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_DOCKER_FEATURES = "e2e-conformance-only" }
run = "e2e/rust/e2e-docker.sh"

["e2e:podman:external-driver"]
description = "Run Podman conformance with a driver-free gateway and external Podman driver binary"
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_PODMAN_FEATURES = "" }
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_PODMAN_FEATURES = "e2e-conformance-only" }
run = "e2e/rust/e2e-podman.sh"

["e2e:vm:external-driver"]
Expand All @@ -262,7 +262,7 @@ run = "e2e/rust/e2e-vm.sh"

["e2e:kubernetes:external-driver"]
description = "Run Kubernetes conformance with a driver-free gateway and external Kubernetes driver"
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_KUBE_BUILD_IMAGES = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "" }
env = { OPENSHELL_E2E_EXTERNAL_COMPUTE_DRIVER = "1", OPENSHELL_E2E_KUBE_BUILD_IMAGES = "1", OPENSHELL_E2E_KUBERNETES_FEATURES = "e2e-conformance-only" }
run = "e2e/rust/e2e-kubernetes.sh"

["e2e:docker"]
Expand Down
Loading