Repository navigation
test(suites): move the conformance, oci-image, provider-refresh and podman suites to testkit - #4774
Draft
matthewgrossman wants to merge 20 commits into
Conversation
Add tests/testkit as its own Cargo workspace: testkit, testkit-macros, and testkit-spec. testkit reads a target file, connects to the gateway through openshell-sdk, creates and cleans up sandboxes, and runs the installed CLI. #[testkit::test] requires a stable ID and a doc comment and declares capability requirements. testkit-ready waits for a target's gateway and CLI before tests run. Nothing uses testkit yet. The lockfile is seeded from the root lockfile so shared crates resolve to the same versions. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Nothing else builds testkit, so check it in Branch Checks and the matching mise tasks to catch openshell-sdk changes that break it. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Cleanup shared one two-minute budget across all of a test's sandboxes, so a slow delete left the rest with no time and could leak them on a shared gateway. Delete them concurrently, each with the full timeout. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Nothing reads test IDs or capability requirements until a runner, waivers, or result history exist, so remove the id and requires arguments and the target file's capabilities section. Both can return as optional attribute arguments without changing existing tests. With them gone, testkit-spec only checked the function shape and read the doc comment, so fold it into testkit-macros. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Replace the target file (OPENSHELL_TEST_TARGET) with the CLI's own gateway selection: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the active gateway. A registered gateway's endpoint, auth mode, and mTLS material come from its CLI config directory. cx.cli runs OPENSHELL_BIN against the same gateway and workspace as the SDK. Suites now run unchanged under e2e/with-docker-gateway.sh and e2e/with-podman-gateway.sh, and environments need no extra file. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…like the root `cargo --locked` only checks a lockfile against its own manifest. A workspace outside the root that builds root crates through path dependencies, such as the testkit suites building `openshell-sdk`, has its own Cargo.lock, so the two lockfiles can resolve a shared crate to different versions without any check failing. That is how aws-lc-sys 0.46 was picked up in #4629, which does not link against the AWS-LC that Nix provides. check_cargo_lockfile_alignment.py fails when an aligned workspace locks a registry crate that the root lockfile also contains at a version the root lockfile does not have. With --fix it re-locks the workspace starting from the root lockfile. The script and its tests come from #4629 (a37ebe9). The aligned workspaces are tests/testkit and tests/suites/sandbox instead of the old conformance suites. The check is wired into the lockfile task and CI in the next commit. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Give new end-to-end tests a home and a CI lane. New tests go in testkit; the old suites take fixes only. - tests/suites/sandbox is a testkit suite workspace. It starts with the long_running_exec_delete test, adapted from the sandbox-lifecycle suite in #4629. Its README holds the guidance on what a test may assert. - The suite builds into a Nix test archive with `--locked`, and a tmachine testsuite and playbook run it against the gateway the installer registered. testkit finds that gateway the way the CLI does, so no target file is needed. The playbook waits for the gateway first, so an unavailable gateway is one environment failure rather than N test failures. It runs nextest serially with a slow-timeout that terminates hung tests, and fails the lane when fewer tests run than expected. - Branch E2E runs it in a `testkit-integration` job on Ubuntu with Docker and the Debian package. It isn't part of the Core E2E result yet. - The alignment check runs in `rust:lockfiles:check` and the Cargo lockfiles CI job, with its tests in `mise run test`, and enrolls tests/testkit and the new suite. A PR that bumps a shared crate in the root Cargo.lock now needs `uv run tasks/scripts/check_cargo_lockfile_alignment.py --fix`. - `mise run e2e:testkit` (and `e2e:testkit:podman`) run the suite against an ephemeral local gateway. - fmt and clippy cover the new workspace in Branch Checks and the mise tasks. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Port the lifecycle, file-transfer, policy-advisor, and CLI smoke checks from tests/suites/conformance/cli to testkit suites under tests/suites/sandbox. Tests that need a persistent workspace live in their own test file so playbooks can exclude them. Add Context::track and Context::adopt so sandboxes created through the CLI are cleaned up. Adapted from #4629 by Evan Lezar. Co-authored-by: Evan Lezar <elezar@nvidia.com> Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Move the oci-image feature tests to tests/suites/sandbox/oci-image. Add Context::track_sandbox and Context::delete_sandboxes to testkit so a test that creates sandboxes through the CLI can register them for cleanup and delete them before removing the images they use. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…ed sandboxes Add Context::cli_env for commands that read secrets from the environment, and Context::track_sandbox so cleanup deletes a sandbox created through the CLI. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…s to testkit Move the provider-refresh Keycloak test and the Podman user-namespace test onto testkit as the tests/suites/providers and tests/suites/podman workspaces, and align their lockfiles with the root. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…odman suites to testkit
Cut the release lanes over to suites written on testkit. The new crates come
from the ports in this series; the cutover rewires everything that ran the old
ones.
- Conformance lanes (Branch E2E, Release Dev, release qualification) run the new
`sandbox` testsuite: the lifecycle, file-transfer and policy-advisor crates in
tests/suites/sandbox from one nextest archive. The `policy-advisor` testsuite
stays as a focused filter over the same archive.
- oci-image, provider-refresh and driver-podman keep their testsuite names and
playbooks and run the ported crates from their own archives
(sandbox-oci-image, providers-refresh-keycloak, podman-userns).
- The conformance playbook becomes the generic suites/nextest.yaml. It also sets
nextest timeouts and fails the lane when fewer tests run than the testsuite's
`suite_expected_tests`. The oci-image, provider-refresh and Podman playbooks
get the same count check.
- `mise run e2e` and the driver e2e scripts run the sandbox suites through
e2e/support/conformance.sh instead of the old conformance crate.
`mise run e2e:oci-image` runs the ported crate.
- Docs point at tests/suites/sandbox, providers and podman.
The old tests/suites/{conformance,features,drivers} workspaces and the
openshell-conformance crate are no longer referenced by CI, Nix or mise tasks.
The ports adapt the sandbox-lifecycle, file-transfer and policy-advisor suites
from #4629.
Co-authored-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Nothing runs them after the cutover to testkit. Remove tests/suites/conformance, tests/suites/features, tests/suites/drivers and the openshell-conformance crate they used, along with its AGENTS.md row and Cargo.lock entry. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
matthewgrossman
added this pull request to stack #4776
October 20, 2026 23:38
|
🌿 Preview your docs: https://nvidia-preview-pr-4774.docs.buildwithfern.com/openshell |
…t results tmachine keys its disk layers on a hash of the Ansible tree, and actions/cache does not save after an exact key hit, so the fixed cache key rebuilt the changed layers on every run. Key the cache on the playbooks and fall back to the newest entry for the same runner. The sandbox-lifecycle playbook now has nextest write a JUnit report and fetches it into artifacts/tmachine-diagnostics, which the integration runner already uploads whether or not the lane passes. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com> # Conflicts: # tests/ansible/playbooks/suites/sandbox-lifecycle.yaml
…workspace - testkit README: drop the duplicated `track_sandbox` bullet and describe `track`, `track_sandbox`, `adopt` and `delete_sandboxes` once. - TESTING.md: the local suite command now excludes oci-image, which needs a container engine, and names the sandbox suites and `sandbox` testsuite instead of the deleted conformance ones. - Run fmt and clippy for tests/suites/providers and tests/suites/podman in the mise tasks and Branch Checks, as already done for tests/suites/sandbox. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the old conformance, oci-image, provider-refresh, and Podman userns suites onto testkit, cuts the release lanes over, and deletes the old suites. This is step 4 of the test harness plan, stacked on the first testkit lane.
The sandbox-lifecycle, file-transfer, and policy-advisor ports start from @elezar's draft #4629.
Related Issue
Part of #3954
Changes
tests/suites/sandbox:lifecycle,file-transfer,policy-advisor,oci-image.tests/suites/providers:refresh-keycloak.tests/suites/podman:userns.sandboxtestsuite (14 tests). oci-image, provider-refresh, and driver-podman keep their testsuite names and run the ported crates. A genericsuites/nextest.yamlplaybook replaces the conformance one, and every migrated playbook fails the lane when fewer tests run thansuite_expected_testsinconfig.nix.mise run e2eand the driver e2e scripts run the sandbox suites.mise run e2e:oci-imageruns the ported crate.tests/suites/{conformance,features,drivers}andcrates/openshell-conformance.cx.track,cx.adopt,cx.track_sandbox,cx.delete_sandboxes, andcx.cli_envfor sandboxes and environment the tests create through the CLI.Old test to new test
smokelifecycle::cli_create_exec_list_deletesandbox_lifecycle(stop/start keeps workspace, stopped can be deleted)persistent_workspace_survives_stop_start,stopped_sandbox_can_be_deleted,cli_stop_start_deletefile_transferround_trip, git_filtering, path_safetyupload_download_round_trip,upload_respects_gitignore,download_stays_inside_workspacepolicy_advisorpolicy_local, mechanistic_proposal, new_hostname_proposalpolicy_local_accepts_proposals,denied_address_produces_draft_rule,denied_hostname_produces_draft_rulerevoked_refresh_grant_requires_user_reauthorizationNew from #4629 with no old counterpart:
long_running_exec_delete,run_to_completion_exit_delete,default_command_sandbox_is_listed_ready,restart_stopped_sandbox.Dropped: the three unit tests of the
rule gettext parser, because the new tests assert on typed draft chunks. The CLI surfacescreate --policy <file>,settings set/get --json,policy get --full, and therule gettext format are no longer exercised; the new tests use the equivalent gRPC calls. oci-image CLI calls now share testkit's 5-minute timeout, where create used to get 10.Only
persistent_workspace_survives_stop_startneeds a driver with persistent workspaces. Docker, Podman, Kubernetes, and the VM driver keep/sandboxacross stop and start, so no lane excludes it.Testing
test:e2e).Checklist
Stacked on the first testkit lane PR.
🤖 Generated with Claude Code