test(testkit): add the testkit library for end-to-end suites

[matthewgrossman]

Summary

Adds tests/testkit/, a support library for SDK-driven end-to-end suites, as its own Cargo workspace. This is a small first step to get the crate into main and checked in CI. Nothing uses it yet. A follow-up PR moves the existing suites onto it, starting with tests/suites/conformance, and each old suite is deleted when its replacement lands.

The code comes from @elezar's draft #4629 (head 673928804). This PR takes only the testkit portion, adapted to current main and cut down to what's needed today (see "Deferred" below).

Related Issue

Part of #3954

Changes

• tests/testkit/: the testkit and testkit-macros crates, plus the testkit-ready binary, taken from test(conformance): run conformance suites from test plans in tmachine environments #4629.
  • A test is an async fn marked #[testkit::test] with a required doc comment.
  • Tests find the gateway the same way the openshell CLI does: OPENSHELL_GATEWAY_ENDPOINT, then OPENSHELL_GATEWAY, then the active gateway. A registered gateway's endpoint and mTLS material come from its CLI config directory. cx.cli runs OPENSHELL_BIN against the same gateway and workspace. Each test logs the gateway it used.
  • Cleanup deletes a test's sandboxes concurrently, each with its own timeout.
  • Cargo.lock is re-seeded from the root lockfile, because main's openshell-sdk has more dependencies than when test(conformance): run conformance suites from test plans in tmachine environments #4629 was cut.
• Branch Checks and the rust:lint, rust:format, rust:format:check, and test:rust mise tasks now run fmt, clippy, and nextest for the testkit workspace. This follows test(conformance): run conformance suites from test plans in tmachine environments #4629's cf72f1108, restricted to testkit.
• tests/testkit/README.md covers writing a test, what Context provides, and running against a gateway. TESTING.md, AGENTS.md, and CONTRIBUTING.md point to it.

Deferred

Dropped from #4629's testkit to keep this first step small. Each can come back without changing tests written in the meantime:

• Stable test IDs (id = "t-…"). Until a runner, waivers, or result history need them, the nextest name identifies a test.
• Capabilities and requires = [...]. These belong with the runner, which selects tests per environment.
• The testkit-spec crate, for listing tests without compiling them. It returns with the runner's catalog.
• The target file (OPENSHELL_TEST_TARGET). It held a copy of the CLI's registered gateway plus capabilities, so testkit now reads the CLI's gateway config directly. Step 3 then needs no openshell_client role change.
• Guidance on what a conformance test may assert. It moves to the conformance suites.

Also left out for later work:

• the runner, plans, environments, waivers, and matrix
• the tmachine Session library, and the Ansible and config.nix changes
• the new suites and e2e-suites.yml
• the lockfile alignment script, until testkit is built into test archives
• changes to existing e2e lanes

Known limitations:

• Cleanup only deletes sandboxes created through cx.sandbox(...). Tests must delete anything else they create. This is documented in the README.
• testkit supports plaintext and mTLS gateways, not OIDC or edge-token auth.

Testing

• cargo fmt --manifest-path tests/testkit/Cargo.toml --all -- --check
• cargo clippy --locked --manifest-path tests/testkit/Cargo.toml --all-targets -- -D warnings
• cargo nextest run --locked --config-file .config/nextest.toml --manifest-path tests/testkit/Cargo.toml (23 passed)
• tasks/scripts/check-cargo-lockfiles.sh covers tests/testkit/Cargo.lock
• mise run license:check
• Unit tests added/updated (if applicable)
• E2E tests added/updated (if applicable). Not applicable: no suite uses testkit yet.

Against a real gateway

To check that testkit works end to end, I built a throwaway suite crate outside the repo. It depends on this branch's testkit and contains the six sandbox_lifecycle tests from #4629, plus a test that creates three sandboxes, checks that cx.cli(["sandbox", "list"]) shows them, and leaves them for cleanup. I ran it unchanged under e2e/with-docker-gateway.sh on macOS with Docker Desktop. The wrapper's own OPENSHELL_GATEWAY and OPENSHELL_BIN were the only configuration:

e2e/with-docker-gateway.sh sh -c 'testkit-ready 120 && cargo nextest run --manifest-path <suite>/Cargo.toml --test-threads 1'

testkit-ready resolved the wrapper's mTLS gateway, reported it healthy, and confirmed the CLI reaches it. All 7 tests passed in about 15s, and openshell sandbox list showed nothing left behind.

On Docker Desktop, sandboxes need either Docker Desktop's host networking turned on (Settings → Resources → Network) or supervisor_network_mode = "bridge". Otherwise the supervisor cannot reach a gateway on the Mac's loopback. For this run I set bridge mode with a local, uncommitted change to the wrapper's gateway config.

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-pr-4724.docs.buildwithfern.com/openshell