Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/branch-checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -141,6 +141,7 @@ jobs:
shell: nix develop -c bash -euo pipefail {0}
run: |
cargo fmt --all -- --check
cargo fmt --manifest-path e2e/support/rust/Cargo.toml --all -- --check
cargo fmt --manifest-path e2e/suites/conformance/Cargo.toml --all -- --check
cargo fmt --manifest-path e2e/rust/Cargo.toml --all -- --check
cargo fmt --manifest-path examples/governance-interceptor/Cargo.toml --all -- --check
Expand Down Expand Up @@ -197,6 +198,7 @@ jobs:
shell: nix develop -c bash -euo pipefail {0}
run: |
cargo clippy --locked --workspace --all-targets -- -D warnings
cargo clippy --locked --manifest-path e2e/support/rust/Cargo.toml --workspace --all-targets -- -D warnings
cargo clippy --locked --manifest-path e2e/suites/conformance/Cargo.toml --workspace --all-targets -- -D warnings
cargo clippy --locked --manifest-path e2e/rust/Cargo.toml --all-targets -- -D warnings
cargo clippy --locked --manifest-path examples/governance-interceptor/Cargo.toml --all-targets -- -D warnings
Expand Down Expand Up @@ -237,6 +239,7 @@ jobs:
OPENSHELL_TELEMETRY_ENABLED: "false"
run: |
cargo nextest run --locked --profile ci --workspace --features openshell-server/test-support
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path e2e/support/rust/Cargo.toml
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path e2e/suites/conformance/Cargo.toml --package openshell-test-conformance-cli -E 'binary(policy_advisor) & test(draft_assertion::tests::)'
cargo nextest run --locked --config-file .config/nextest.toml --profile ci --manifest-path examples/supervisor-middleware-content-guard/Cargo.toml

Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,6 @@ Do not rely on this file for a full inventory. The detailed public and contribut
| Path | Components | Purpose |
|------|-----------|---------|
| `crates/openshell-cli/` | CLI binary | User-facing command-line interface |
| `crates/openshell-conformance/` | Conformance test support | Command execution, polling, diagnostics, and cleanup |
| `crates/openshell-server/` | Gateway server | Control-plane API, sandbox lifecycle, auth boundary |
| `crates/openshell-sandbox/` | Sandbox runtime | Capability-free workload launcher, process identity, and seccomp-mediated I/O |
| `crates/openshell-supervisor/` | Supervisor runtime | Gateway session, policy evaluation, credentials, and upstream networking |
Expand Down Expand Up @@ -61,7 +60,8 @@ Do not rely on this file for a full inventory. The detailed public and contribut
| `python/openshell/` | Python SDK | Python bindings and CLI packaging |
| `sdk/typescript/` | TypeScript SDK | Native Connect client, curated sandbox API, and generated protobuf types |
| `proto/` | Protobuf definitions | gRPC service contracts |
| `e2e/suites/` | Behavioral E2E suites | Directly authored conformance stories |
| `e2e/suites/` | Behavioral E2E suites | Conformance stories; other suites remain in tests/suites |
| `e2e/support/rust/` | Shared Rust test tooling | CLI runner, binary resolution, output parsing, and port utilities |
| `tests/` | Test provisioning | Target configuration, artifact construction, and Ansible setup |
| `deploy/` | Docker, Helm, K8s | Dockerfiles, Helm chart, manifests |
| `docs/` | Published docs | MDX pages, navigation, and content assets |
Expand Down
4 changes: 2 additions & 2 deletions CI.md
Original file line number Diff line number Diff line change
Expand Up @@ -578,8 +578,8 @@ jobs to the required status list while they remain in observation mode.
## Conformance source checks

Branch Rust checks format and lint the separate
`e2e/suites/conformance` workspace.
The Rust test jobs include the conformance runner in the root workspace and select the pure policy assertion
`e2e/suites/conformance` workspace and shared tooling in `e2e/support/rust`.
The Rust test jobs run the shared tooling and select the pure policy assertion
tests in the conformance CLI package with the CI nextest profile, without a gateway. Gateway-backed CLI tests retain their driver E2E and installed-artifact
integration lanes; source unit-test jobs do not execute those entry points.

Expand Down
11 changes: 0 additions & 11 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

24 changes: 13 additions & 11 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,14 @@ mise run ci # Everything: lint, compile checks, and tests
- End-to-end tests exercise a configured OpenShell target through a client.
- Benchmarks measure performance or scale separately from behavioral correctness.

Conformance stories under `e2e/suites/conformance/` verify required public behavior.
The [conformance guide](e2e/suites/conformance/README.md) owns their authoring and
execution guidance. Driver and feature suites remain under `tests/suites/`;
existing mixed suites remain under `e2e/rust/`. `tests/` also supplies provisioning
and artifact construction.
End-to-end suites are grouped by intent: conformance verifies
required public behavior, feature suites need a named external service or special
gateway configuration, and driver suites verify driver or host integration.
The [conformance guide](e2e/suites/conformance/README.md) owns conformance
requirements, authoring, and execution guidance. Existing mixed suites remain
under `e2e/rust/` while their behavior is migrated. `tests/` supplies provisioning
and artifact construction independently of the suite layout. Driver and feature
suites remain under `tests/suites/` pending their relocation.

## Test Layout

Expand All @@ -29,7 +32,7 @@ crates/*/src/ # Inline #[cfg(test)] modules
crates/*/tests/ # Rust integration tests
python/openshell/ # Python unit tests (*_test.py suffix)
e2e/python/ # Python E2E tests (test_*.py prefix)
e2e/rust/ # Rust CLI E2E tests
e2e/rust/ # Legacy Rust CLI E2E tests awaiting migration
e2e/suites/conformance/ # Conformance stories
tests/suites/ # Existing feature and driver suites
tests/ # Target provisioning and artifact construction
Expand Down Expand Up @@ -227,11 +230,10 @@ When more than one test needs this behavior, put the transport in the shared
Rust e2e harness and require callers to use it instead of duplicating DNS,
HTTP `Host`, TLS SNI, and mTLS handling.

Conformance command execution, polling, diagnostics, and cleanup remain in
`crates/openshell-conformance`. Story implementations live directly under
`e2e/suites/conformance/cli/tests`. `mise run test:rust` includes runner and pure
policy assertion tests without requiring a gateway. Existing Rust e2e tests
retain their own harness.
Shared Rust test tooling lives under `e2e/support/rust`, including the CLI runner,
binary resolution, output parsing, and port utilities. Conformance scenarios live
under `e2e/suites/conformance/cli/tests`. `mise run test:rust` includes the tooling
and scenario unit tests without requiring a gateway.

Suites:

Expand Down
21 changes: 0 additions & 21 deletions crates/openshell-conformance/Cargo.toml

This file was deleted.

10 changes: 0 additions & 10 deletions crates/openshell-conformance/README.md

This file was deleted.

12 changes: 12 additions & 0 deletions e2e/rust/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions e2e/rust/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,7 @@ path = "tests/gpu.rs"
required-features = ["e2e-gpu"]

[dependencies]
openshell-e2e-support = { path = "../support/rust" }
base64 = "0.22"
bollard = "0.20"
bytes = "1"
Expand Down
6 changes: 3 additions & 3 deletions e2e/rust/src/harness/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@

//! Shared test harness modules for CLI e2e tests.

pub mod binary;
pub use openshell_e2e_support::binary;
pub mod cli;
pub mod container;
pub mod gateway;
pub mod host_process;
pub mod output;
pub mod port;
pub use openshell_e2e_support::output;
pub use openshell_e2e_support::port;
pub mod sandbox;
58 changes: 2 additions & 56 deletions e2e/suites/conformance/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 6 additions & 6 deletions e2e/suites/conformance/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,10 @@ The current areas are `file_transfer`, `lifecycle`, `policy_advisor`, and `smoke
Each area compiles into one test binary; individual stories remain selectable by
their module prefix. `main.rs` contains only module declarations.

CLI execution, polling, diagnostics, and cleanup remain in
`crates/openshell-conformance`. Story modules implement their tests directly;
Cargo and nextest provide discovery and selection. The runner crate contains no
scenario registry. Existing Rust e2e tests retain their separate harness.
Shared CLI execution, polling, diagnostics, and cleanup live in
[`e2e/support/rust`](../../support/rust/README.md), alongside the utilities used by
the existing Rust e2e tests. Story modules implement their tests directly; Cargo and nextest provide discovery and selection. Product crates must
not depend on the test tooling.

For example, run only file-transfer round trips against a prepared gateway:

Expand All @@ -32,7 +32,7 @@ OPENSHELL_BIN=/absolute/path/to/openshell \
Run shared tooling and scenario unit tests:

```shell
cargo test --locked --package openshell-conformance
cargo test --locked --manifest-path e2e/support/rust/Cargo.toml
cargo test --locked --manifest-path e2e/suites/conformance/Cargo.toml \
--package openshell-test-conformance-cli --test policy_advisor draft_assertion::tests::
```
Expand Down Expand Up @@ -115,7 +115,7 @@ and waiver handling remain implementation work.
## Migration scope

The conformance workspace lives under `e2e/suites/conformance/`; feature and
driver suites remain under `tests/suites/`, alongside target provisioning. Existing
driver workspaces remain under `tests/suites/` until a separate relocation. Existing
assertions and test identities are preserved. The legacy mixed suites under
`e2e/rust/` still need classification and migration by behavioral intent.

Expand Down
2 changes: 1 addition & 1 deletion e2e/suites/conformance/cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ repository = "https://github.com/NVIDIA/OpenShell"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
tempfile = "3"
openshell-conformance = { path = "../../../../crates/openshell-conformance" }
openshell-e2e-support = { path = "../../../support/rust" }
tokio = { version = "1.43", features = ["macros", "process", "rt", "time"] }

[lints]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use super::helpers::{
delete_sandbox, download, exec, fs_error, git, git_init, prepare_sandbox, require_absent,
require_exists, require_text, upload, upload_result,
};
use openshell_conformance::OpenShellRunner;
use openshell_e2e_support::OpenShellRunner;
use std::fs;
use std::path::Path;

Expand Down
4 changes: 2 additions & 2 deletions e2e/suites/conformance/cli/tests/file_transfer/helpers.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
// SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
// SPDX-License-Identifier: Apache-2.0

use openshell_conformance::CommandResult;
use openshell_conformance::OpenShellRunner;
use openshell_e2e_support::CommandResult;
use openshell_e2e_support::OpenShellRunner;
use std::fs;
use std::path::Path;
use std::process::Stdio;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use super::helpers::{
delete_sandbox, download, download_result, exec, fs_error, prepare_sandbox, require_absent,
require_text,
};
use openshell_conformance::OpenShellRunner;
use openshell_e2e_support::OpenShellRunner;
use std::fs;
use std::path::Path;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use super::helpers::{
LARGE_FILE_SIZE, delete_sandbox, download, exec, fs_error, prepare_sandbox, require_text,
upload, upload_result,
};
use openshell_conformance::OpenShellRunner;
use openshell_e2e_support::OpenShellRunner;
use std::fs;
use std::path::Path;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
use super::helpers::{
create_running_sandbox, run_lifecycle_command, wait_for_absence, wait_for_phase,
};
use openshell_conformance::OpenShellRunner;
use openshell_e2e_support::OpenShellRunner;

/// Verify a stopped sandbox can be deleted.
#[tokio::test]
Expand Down
Loading
Loading