Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/branch-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,7 @@ jobs:
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"cdi"},
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/release-dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ jobs:
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"cdi"},
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/release-qualification.yml
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ jobs:
integration-inputs-artifact-id: ${{ needs.prepare-integration.outputs.integration_inputs_artifact_id }}
test-matrix: >-
[
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"cdi"},
{"environment":"fedora-podman-rootful","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"fedora-podman-rootless","installer":"rpm","testsuite":"provider-refresh"},
{"environment":"ubuntu-docker-rootful","installer":"deb","testsuite":"oci-image"},
Expand Down
71 changes: 71 additions & 0 deletions TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -330,6 +330,77 @@ nix run .#tmachine -- test fedora-podman-rootless binaries e2e-podman
nix run .#tmachine -- test fedora-podman-rootless binaries driver-podman
```

The `cdi` feature suite tests device injection without GPU hardware in the
`ubuntu-docker-rootful` environment. Build the candidate artifacts, then run:

```shell
nix run .#build-artifacts
nix run .#tmachine -- test ubuntu-docker-rootful deb cdi
```

If candidate binaries, images, and packages are already built, rebuild just the
CDI test archive with `nix run .#build-artifacts-cdi-tests`.

The `binaries` installer also supports this suite. Docker environment setup
creates the named scenario fixtures and a CDI spec that defines one device per
scenario, selected as `tmachine.example/cdi-scenario=<scenario>`. Persistent
spec sources live under `/var/lib/openshell-test-inputs/cdi/specs`; a
systemd-tmpfiles configuration recreates them in `/var/run/cdi` on every VM
boot. This preserves the fixtures across tmachine's setup, installation, and
test disk layers and makes them available to every suite using the Docker
environment. The tests need no GPU hardware.

Each test creates a sandbox that selects one scenario explicitly with `--gpu 1`
and the Docker `cdi_devices` driver option, then checks what the workload
observes or that creation is rejected. Selection by count or default, and the
`nvidia.com/gpu` inventory, are not covered here. Caller driver configuration
is enabled temporarily in the installed gateway's active TOML. The test
playbook restores the configuration and restarts the gateway even after test
failure; the CDI fixtures remain part of the environment. Docker must support
CDI. Branch and release feature-test matrices run this suite on Docker.

| Scenario | Injected resource | Asserted today | Expected with PR #4204 |
| --- | --- | --- | --- |
| `mount-under-read-only-baseline` | File at `/usr/share/tmachine-cdi/baseline-read-only-file`, under a directory the sandbox already makes read-only | Read succeeds, write denied. | Unchanged. |
| `read-only-file` | File at `/tmachine-cdi/read-only/file` | Read and write denied. | Read succeeds outside the baseline grants; write fails. |
| `read-only-directory` | Directory at `/tmachine-cdi/read-only/directory` | Reading its marker and creating files denied. | Reading its marker succeeds; write fails. |
| `library-parent` | Marker named `/tmachine-cdi/library/libfixture.so.1` | Read and write denied. | Resolver derives the library parent; the marker is not an actual shared library. |
| `broad-library-parent` | Marker named `/libtmachinefixture.so.1` | Read and write denied. | Launch rejects expansion to `/`. |
| `writable-file` | Writable file at `/tmachine-cdi/writable/file` | Read and write denied. | Launch rejects it even with exact or ancestor write grants. |
| `writable-directory` | Writable directory at `/tmachine-cdi/writable/directory` | Write denied. | Launch rejects it even with exact or ancestor write grants. |
| `omitted-access-file` | Bind without `ro` or `rw` | Read and write denied. | Effective writable access causes launch rejection. |
| `device-node` | Null character device at `/dev/tmachine-cdi-null` | Non-root, write denied. | Non-root read/write succeeds; an exact authored read-only entry fails launch. |
| `device-node-with-gid` | Group-restricted null device at `/dev/tmachine-cdi-group-null` | Non-root with the device's group, write denied. | The generated supplementary GID permits non-root access. |
| `unix-socket` | Echo socket at `/tmachine-cdi/socket/echo.sock` | Sibling file in the host directory is not visible. | Socket communication succeeds without deriving parent filesystem grants. |

Every scenario also supplies a spec-level environment marker and a read-only
mount at `/usr/share/tmachine-cdi/common/spec-marker`, allowing checks of
spec/device edit merging without depending on derived grants. The device group
is allocated in the disposable guest. A systemd socket unit restores the echo
endpoint after reboot; the existing tmpfiles configuration restores the
scenario spec. Negative scenarios must be selected individually, not combined
with positive cases.

The role publishes each scenario's expectations as `scenarios.json` next to the
fixtures. The `scenario_*` tests load them and assert the behavior recorded in
`vars/scenarios.yaml`, which is the sandbox's behavior today. A change that
alters a scenario's behavior, such as #4204 deriving grants from the spec,
updates those values in the same diff. To record the current behavior, run the
tests with `OPENSHELL_CDI_CAPTURE=1` set for the test process; they then print
what each scenario observed instead of asserting.

To add a scenario, add an entry to `cdi_scenarios` in
`tests/ansible/roles/cdi_fixtures/vars/scenarios.yaml` with the host fixtures
it needs, its CDI edits, and its `expect` (the checks on the workload, or a
rejection), then list it in `cdi_scenarios!` in
`tests/suites/features/cdi/tests/injection.rs`. A new spec file belongs in
`cdi_fixture_spec_files` in the role defaults, which the setup and test
playbooks share.

Directory absence, daemon configuration refresh, and restart tests require
orchestration around these fixtures. Spec projection secrecy is a separate
workload assertion. Hardware CUDA initialization remains outside this suite.

Print the exact tmachine archive selection as a shell `PODMAN_CI_TESTS` array:

```shell
Expand Down
4 changes: 4 additions & 0 deletions flake.nix
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,10 @@
type = "app";
program = "${artifacts.testArchives}/bin/build-artifacts-test-archives";
};
build-artifacts-cdi-tests = {
type = "app";
program = "${artifacts.cdiArchive}/bin/build-openshell-cdi-test-archive";
};
build-artifacts-test-images = {
type = "app";
program = "${artifacts.testImages}/bin/build-artifacts-test-images";
Expand Down
11 changes: 3 additions & 8 deletions tasks/scripts/test-packaging-assets.sh
Original file line number Diff line number Diff line change
Expand Up @@ -91,15 +91,10 @@ assert_contains "$spec" '%files prover'
assert_contains "$spec" '%{_bindir}/%{name}-prover'
assert_not_contains "$spec" '%%S/openshell/tls'

# Installed-package qualification must select the candidate trusted runtime
# images through the same environment file consumed by preflight and startup,
# without generating operator-owned gateway TOML.
# Qualification retains the package-owned service's environment file. The
# selected configuration and runtime images are exercised by tmachine's
# installed-package integration lanes.
assert_contains "$package_qualification" 'dest: "{{ openshell_gateway_home }}/.config/openshell/gateway.env"'
assert_contains "$package_qualification" 'OPENSHELL_COMPUTE_DRIVER={{ openshell_gateway_driver }}'
assert_contains "$package_qualification" 'OPENSHELL_SANDBOX_RUNTIME_IMAGE=docker.io/openshell/sandbox:tmachine'
assert_contains "$package_qualification" 'OPENSHELL_SUPERVISOR_IMAGE=docker.io/openshell/supervisor:tmachine'
assert_not_contains "$package_qualification" 'OPENSHELL_GATEWAY_CONFIG='
assert_not_contains "$package_qualification" '/var/lib/openshell-qualification/gateway.toml'

# Direct tmachine gateway services select trusted runtime images through the
# process environment. Driver TOML remains available for settings that do not
Expand Down
12 changes: 12 additions & 0 deletions tests/ansible/playbooks/cdi.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

---
- name: Prepare hardware-free CDI fixtures in the Docker environment
hosts: all
gather_facts: false
pre_tasks:
- name: Wait for SSH
ansible.builtin.wait_for_connection:
roles:
- cdi_fixtures
105 changes: 105 additions & 0 deletions tests/ansible/playbooks/features/cdi/tests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

---
- name: Test CDI injection without GPU hardware
hosts: all
gather_facts: false
vars_files:
- ../../../roles/cdi_fixtures/defaults/main.yaml
vars:
cdi_test_root: /var/lib/openshell-feature-tests/cdi
tasks:
- name: Wait for SSH
ansible.builtin.wait_for_connection:

- name: Resolve installed gateway context
ansible.builtin.include_role:
name: openshell_test_gateway

- name: Require environment CDI specs after boot
ansible.builtin.stat:
path: "/var/run/cdi/{{ item }}"
loop: "{{ cdi_fixture_spec_files }}"
register: cdi_environment_specs

- name: Require CDI fixtures supplied by the Docker environment
ansible.builtin.assert:
that: [item.stat.isreg | default(false)]
fail_msg: "Docker environment did not recreate CDI spec {{ item.item }} after boot"
loop: "{{ cdi_environment_specs.results }}"

- name: Save gateway configuration before enabling caller device selection
become: true
ansible.builtin.slurp:
src: "{{ openshell_test_gateway.config_path }}"
register: cdi_gateway_config

- name: Run CDI tests with caller device selection enabled
block:
- name: Enable caller driver configuration for CDI tests
become: true
ansible.builtin.blockinfile:
path: "{{ openshell_test_gateway.config_path }}"
insertafter: '^\[openshell\.drivers\.docker\]$'
marker: "# {mark} OpenShell CDI test fixture"
block: allow_driver_config = true

- name: Restart gateway with caller device selection enabled
ansible.builtin.include_role:
name: openshell_test_gateway
tasks_from: restart.yaml

- name: Create CDI test directory
become: true
ansible.builtin.file:
path: "{{ cdi_test_root }}"
state: directory
owner: tmachine
group: tmachine
mode: "0700"

- name: Install CDI test archive
become: true
ansible.builtin.unarchive:
src: "{{ openshell_cdi_test_bundle }}"
dest: "{{ cdi_test_root }}"
owner: tmachine
group: tmachine

- name: Run CDI feature tests
ansible.builtin.command:
argv: [cargo-nextest, nextest, run, --archive-file, "{{ cdi_test_root }}/tests.tar.zst", --workspace-remap, "{{ cdi_test_root }}", --no-fail-fast, --no-capture]
environment:
OPENSHELL_GATEWAY: "{{ openshell_test_gateway.name }}"
OPENSHELL_BIN: "{{ openshell_test_cli.stdout }}"
register: cdi_test_result
changed_when: false
failed_when: false

- name: Show CDI test result
ansible.builtin.debug:
var: cdi_test_result

- name: Collect gateway diagnostics after test failure
ansible.builtin.include_role:
name: openshell_test_gateway
tasks_from: journal.yaml
when: cdi_test_result.rc != 0

- name: Require CDI test success
ansible.builtin.assert:
that: [cdi_test_result.rc == 0]
fail_msg: CDI feature tests failed

always:
- name: Restore gateway configuration
become: true
ansible.builtin.copy:
content: "{{ cdi_gateway_config.content | b64decode }}"
dest: "{{ openshell_test_gateway.config_path }}"

- name: Restart gateway with its original configuration
ansible.builtin.include_role:
name: openshell_test_gateway
tasks_from: restart.yaml
10 changes: 10 additions & 0 deletions tests/ansible/roles/cdi_fixtures/defaults/main.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

---
cdi_fixture_root: /var/lib/openshell-test-inputs/cdi

# Specs the role installs into /var/run/cdi on every boot. The CDI feature
# playbook checks the same list, so a new spec is added here only.
cdi_fixture_spec_files:
- openshell-tmachine-scenarios.yaml
13 changes: 13 additions & 0 deletions tests/ansible/roles/cdi_fixtures/files/openshell-cdi-echo.socket
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
[Unit]
Description=OpenShell tmachine CDI Unix socket fixture

[Socket]
ListenStream=/var/lib/openshell-test-inputs/cdi/scenarios/socket/echo.sock
SocketMode=0666
Accept=yes
RemoveOnStop=yes

[Install]
WantedBy=sockets.target
14 changes: 14 additions & 0 deletions tests/ansible/roles/cdi_fixtures/files/openshell-cdi-echo@.service
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
[Unit]
Description=OpenShell tmachine CDI echo connection

[Service]
ExecStart=/bin/cat
StandardInput=socket
StandardOutput=socket
DynamicUser=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
49 changes: 49 additions & 0 deletions tests/ansible/roles/cdi_fixtures/tasks/main.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0

---
# Persistent sources are copied into /var/run/cdi by systemd-tmpfiles on every
# boot, including the setup, installation, and test disk layers.
- name: Check fixture paths before creating CDI specs
become: true
ansible.builtin.stat:
path: "{{ item }}"
follow: false
loop: "{{ cdi_fixture_spec_files | map('regex_replace', '^', '/var/run/cdi/') }}"
register: cdi_fixture_paths

- name: Require a guest without existing fixture paths
ansible.builtin.assert:
that: [not item.stat.exists]
fail_msg: "Refusing to replace existing CDI fixture path {{ item.item }}"
loop: "{{ cdi_fixture_paths.results }}"

- name: Create CDI spec directories
become: true
ansible.builtin.file:
path: "{{ item }}"
state: directory
owner: root
group: root
mode: "0755"
loop:
- "{{ cdi_fixture_root }}"
- "{{ cdi_fixture_root }}/specs"

- name: Prepare named CDI scenarios
ansible.builtin.include_tasks: scenarios.yaml

- name: Recreate volatile CDI specs on every VM boot
become: true
ansible.builtin.template:
src: tmpfiles.conf.j2
dest: /etc/tmpfiles.d/openshell-tmachine-cdi.conf
owner: root
group: root
mode: "0644"

- name: Populate CDI specs for the current boot
become: true
ansible.builtin.command:
argv: [systemd-tmpfiles, --create, /etc/tmpfiles.d/openshell-tmachine-cdi.conf]
changed_when: true
Loading
Loading