test: define and track OpenShell testing strategy #3954

Description

Track OpenShell's testing strategy: behavioral contracts, test ownership, execution infrastructure, CI and release qualification, and contributor guidance. Conformance remains a central workstream within this broader scope.

This issue succeeds #2183 as the active conformance tracker. #2183 remains the completed initial design milestone and #2925 the initial implementation. Focused implementations and migrations retain their own issues and PRs.

Strategy discussion

RFC 0016 — OpenShell testing strategy is assigned to this issue at the maintainer's request. Existing PR #3460 is being repurposed as the RFC discussion, replacing its proposed target-state rewrite of TESTING.md and CI.md.

The RFC separates behavioral contracts and test routing from provisioning, installation, execution, and CI selection. It incorporates general conformance, feature-specific, driver-specific, disruption, and load/scale testing. SDK-specific compatibility remains in #3028 / #3238.

Tracked Work

Strategy and documentation

• docs(rfc): propose OpenShell testing strategy #3460 — RFC 0016: testing strategy and unresolved policy decisions
• docs(conformance): route agents to portable test guide #3945 — contributor routing and conformance authoring guidance, aligned with the RFC
• Define TESTING.md as the contributor entry point, tests/CONFORMANCE.md as conformance policy, suite READMEs as authoring/run guides, and CI.md as implemented workflow reference
• Define contract maturity, promotion/change review, version compatibility, and the meaning of a complete conformance result
• Keep SDK conformance docs(rfc): propose SDK conformance testing #3238 aligned with shared terminology and provisioning boundaries

Foundation and execution

• spike: driver conformance suite for compute drivers #2183 — initial shared driver conformance design
• test(e2e): run conformance in gateway lanes #2925 — initial gateway-lane integration
• test(conformance): select leaf scenarios by capability #3768 — independently selectable scenario leaves; distinguish scenario IDs from API capabilities
• test(e2e): remove schema parity campaign #3864 — remove completed schema-parity campaign
• test(conformance): run driver suites with cargo #3866 — Cargo-based conformance execution and standalone conformance CLI removal
• Consolidate the shared conformance library under tests/suites/conformance after runner removal
• Add effective capability discovery through the API and machine-readable CLI output

Behavioral coverage and migration

• test(podman): run existing Podman behavioral e2e coverage in CI #3712 — migrate all e2e-podman-eligible intent into appropriate suites; retire the interim feature and tmachine suite
• test(e2e): add cross-driver image-to-sandbox conformance #2422 — cross-driver image-to-sandbox contracts
• Extend lifecycle, policy, networking, provider, and file-operation coverage
• Explore portable security probes, including bypass prevention and core-dump protection
• Establish disruption conformance with portable assertions and environment-specific actuators
• Give load/scale coverage a separate execution and reporting path

Driver and environment coverage

• ci(windows): report full MXC conformance #3769 — advisory mock-MXC results on Windows x64 and ARM64
• test(e2e): separate VM gateway provisioning from test execution #2314 — separate VM provisioning from scenario execution
• Clarify default workspace behavior in Kubernetes operator mode #2971 — default-workspace behavior in Kubernetes operator mode
• feat(ha): harden gateway peer transport and expand failover conformance #3529 — HA and failover coverage
• ci(qualification): add Ubuntu 24.04 k3s scenario #3455 — Ubuntu k3s qualification environment
• Run general conformance on every available supported driver; track unavailable environments as coverage gaps

CI, release qualification, and reporting

• Define required versus advisory checks and branch/release execution matrices
• Separate candidate installation validation from behavioral conformance
• Start run attribution with the OpenShell SHA and tmachine configuration
• Document how contributors and operators execute and interpret suites

Related work

• feat(cli): add identity-aware sandbox deletion waiting #3944 — identity-aware deletion waiting exposed by lifecycle conformance
• fix(mxc): preserve JSON arguments in PowerShell 5.1 (NVBug 6843165) #3893 — PowerShell argument preservation for MXC runners
• test(e2e): add API and CLI conformance profiles #2182 and test(e2e): add API conformance framework #2873 — older API-conformance proposals to assess for supersession, preserving unique useful assertions

Completion criteria

• The testing strategy and documentation ownership are agreed.
• Test-family boundaries and capability/result semantics are documented.
• Required coverage and environment gaps are visible in CI and release qualification.
• Contributors have a documented path to classify, add, and review scenarios.
• Remaining implementations and migrations have focused linked issues or PRs.

Approval of the RFC does not itself complete this tracking issue.