test(oci): add a shared Docker and Podman suite for image USER and WORKDIR by matthewgrossman · Pull Request #3931 · NVIDIA/OpenShell

matthewgrossman · 2026-09-30T06:01:31Z

Summary

Adds oci-image, a shared feature suite that checks how an image's USER and WORKDIR set the sandbox identity and workspace. It runs through the installed CLI on Docker rootful, Podman rootful, Podman rootless, and ARM64 Docker. It replaces Docker-only and Podman-only e2e checks that couldn't catch a regression on the other runtime.

Closes #4346. Related to #2526.

The suite

tests/suites/features/oci-image runs four scenarios:

a named USER with a custom WORKDIR: identity, working directory, unchanged root-owned image files, writes from the main process and sandbox exec, and upload/download relative to the workspace
a numeric USER with private (0700) WORKDIR parents, under a policy with no process section, so the image USER is the only identity source
an image with no WORKDIR: falls back to the managed /sandbox
a writable/unwritable WORKDIR pair: the control succeeds, and the unwritable one fails before the command runs. The suite reads the WorkspaceValidationFailed condition reason from openshell sandbox get --output json, so another provisioning failure can't pass

Every scenario also checks that the workload has no capabilities, has NoNewPrivs set, and cannot read the sandbox credentials, and that a custom WORKDIR is not a mount.

The numeric-USER scenario also inspects the driver's containers through the gateway's engine: the workload runs the image ID inspected before creation, starts directly as the OCI identity, has no network, and has no TLS or supervisor-state mounts, and the supervisor is a separate, non-root container.

Wiring: tmachine testsuite, test archive, and playbook; oci-image entries in the branch, release-dev, and release-tag matrices, including ARM64 Docker.

Other changes

Workspace rejections are reported. The supervisor exits with reserved status 78 when the sandbox rejects the image working directory, and the Docker and Podman drivers report that exit as WorkspaceValidationFailed with a fixed message. A workload that exits with 78 itself is reported to the driver as 1.
Docker monitor: logs supervisor output instead of putting it in the condition message, and removes the supervisor container when a stop overlaps diagnostics collection.
Legacy e2e: deletes custom_image.rs. Trims podman_oci_identity.rs to the Podman-specific supervisor checks: the companion runs as the resolved OCI identity (Docker uses a fixed unprivileged user) and has no effective capabilities.

Follow-ups

test(oci): cover image USER and policy identity resolution in the oci-image suite #4431: add the bug: sandbox from a USER root image fails with an unexplained "UID or GID zero" error #4030 identity scenarios (USER root with and without a policy override, and no USER). The root rejection first needs Podman to report IdentityResolutionFailed.
test(conformance): move scenario files into Cargo tests #4120 relocates all suites, including this one, from tests/suites to e2e/suites. If test(conformance): move scenario files into Cargo tests #4120 merges first, this PR moves the suite before merging. The remaining Podman-specific podman_oci_identity checks can move into a Podman driver suite as part of that work.
The Docker-only nested-volume test in driver_config_volume.rs stays as it is. Moving it needs driver-config mounts enabled on the tmachine gateways.

Testing

mise run pre-commit on 2bb7f665e; fmt and clippy on the suite and the e2e crate
Suite passed on local rootful Podman before the shared container and condition-reason checks were added
CI on d181609a7: Branch Checks and Core E2E pass, including all four oci-image lanes
CI on the current head, which adds the capability, credential, and mount checks, the shared container inspection checks, and the structured condition-reason check
mise run e2e:oci-image locally on macOS Docker Desktop: sandbox creation fails before the scenarios run because the supervisor cannot reach the gateway; CI is the first real run of the latest changes

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable): TESTING.md

copy-pr-bot · 2026-09-30T06:01:34Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

github-actions · 2026-09-30T06:02:11Z

🌿 Preview your docs: https://nvidia-preview-pr-3931.docs.buildwithfern.com/openshell

matthewgrossman · 2026-09-30T06:12:43Z

/ok to test 2fed032

github-actions · 2026-09-30T06:16:33Z

Label test:e2e applied for 2fed032. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway, sandbox, and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

elezar · 2026-09-30T07:13:07Z

This is great! Thanks.

One expectation I had (related to #3712) would be that at least some of the tests in podman_oci_identity.rs would also move / be replaced by the new feature-specific tests. My agent seems to think that there may still be some tests there that are driver-agnostic (at least for Docker and Podman) in their intent -- even though they may rely on runtime specifics to verify behaviour.

I don't think that that file should be removed entirely because it also includes tests for other behaviour.

elezar · 2026-10-01T14:42:57Z

The shared Docker/Podman OCI suite would also be a useful place to pin the image identity contract discussed in #4030. Could we extend the scenarios to cover:

Explicit USER root, with no policy identity override: sandbox creation fails with IdentityResolutionFailed before the workload command runs.
No image USER and no policy identity: the workload runs as UID/GID 1000:1000.
Explicit USER root with a valid non-root policy user/group override: the workload runs as the policy-selected identity. Use a compatible workspace so this tests identity resolution independently of workspace permissions.

Each case should run on the existing Docker rootful, Podman rootful, and Podman rootless lanes. These assertions can cover the current behavior without waiting for #4030's diagnostic changes. Detailed rejection provenance belongs in resolver tests, and once-only interactive failure output needs a real PTY test alongside the #4030 fix.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…idationFailed The supervisor exits with the reserved status when the sandbox rejects the image working directory. Docker maps that exit during readiness and monitoring and records it before removing the supervisor container, and Podman maps the supervisor companion's exit instead of the workload's. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Unrelated provisioning failures can no longer pass the rejection scenario. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

The supervisor no longer passes on a workload's own exit status 78, so drivers cannot misreport it as a workspace rejection after a runtime restart. The rejection is recognized once at agent start and carried as a typed error. Docker reports the same fixed condition message as Podman and logs the supervisor output instead, and its monitor removes the supervisor container when a stop overlaps diagnostics collection. The OCI image suite registers images for cleanup only after they build. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…image-suite Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Report a recorded Docker workspace rejection once, where supervisor readiness fails, and match its fixed message exactly. Read the Podman supervisor exit status once per inspection. Choose the supervisor exit status in one function and derive the workspace error type. The OCI image suite shares step helpers, checks that a custom workspace does not also create /sandbox, and its playbook waits for a connected gateway and collects failure diagnostics through the shared role. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

elezar

The shared OCI suite and workspace-rejection reporting look good. Branch Checks and Core E2E pass on d181609, including the Docker rootful DEB, Podman rootful/rootless RPM, and ARM64 Docker OCI lanes.

When updating #4120, we'll also need to move the OCI tests added here from tests/suites/features/oci-image to e2e/suites/features/oci-image, adopt the shared e2e/support/rust tooling, and update the archive/workflow paths. That migration and the previously discussed #4030 identity scenarios are follow-ups and do not block this PR.

…image-suite Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

…hecks The oci-image feature suite now checks, on Docker and Podman, that the image user has no capabilities, has no_new_privs set, and cannot read the sandbox control credentials, and that a custom WORKDIR is not a mount. The checks that need container inspection (pinned image ID, network=none, supervisor user and capabilities) move to a small test in the Podman driver suite. Remove the legacy podman_oci_identity e2e test. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Restore the Podman e2e test removed in the previous commit and drop only the checks the shared oci-image suite now runs on Docker and Podman: workload capabilities, no_new_privs, credential readability, and the workspace and /sandbox mount checks. The pinned image ID and the workload/supervisor isolation checks stay, as does the Podman driver suite unchanged. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Carry the Docker control supervisor's failure reason in a typed error instead of matching status text, build the fixed workspace message where the wait result is classified, and drop the fake Docker server test. Inline image cleanup ordering in the suite, read the container engine once, share sandbox naming, and stop granting a literal /sandbox in the image identity policy. Add e2e:oci-image mise tasks and use them from the ARM64 Docker workflow and TESTING.md. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

Inspect the containers the driver created through the gateway's engine in the shared suite: the workload runs the inspected image ID as the OCI identity without a network or supervisor credential mounts, and the supervisor is a separate non-root container. Keep only the Podman-specific supervisor identity and capability checks in podman_oci_identity. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-10-12T20:40:42Z

@elezar Agreed. Most of the remaining podman_oci_identity checks were Podman-only in how they inspect containers, not in what they expect. The oci-image suite already builds images with the gateway's engine, so as of 7fdecf3 it also inspects the containers the driver created, on Docker and Podman. It checks that the workload runs the image ID inspected before creation, starts directly as the OCI identity, has no network, and has no TLS or supervisor-state mounts. It also checks that the supervisor is a separate, non-root container.

podman_oci_identity.rs stays, with only the Podman-specific supervisor checks: the companion runs as the resolved OCI identity (Docker uses a fixed unprivileged user) and has no effective capabilities (read from Podman's EffectiveCaps, which Docker's inspect doesn't have).

Read the WorkspaceValidationFailed condition reason from sandbox get --output json instead of matching CLI output, and restore the Docker driver's original workspace failure helpers. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman · 2026-10-12T22:29:57Z

@elezar this is ready for another review. Since your last review on d181609a7, I merged main and made these changes:

Workspace rejection check: the unwritable-WORKDIR scenario now reads conditions[].reason from openshell sandbox get --output json and requires WorkspaceValidationFailed, instead of searching the CLI output.
podman_oci_identity checks moved into the shared suite: the suite now inspects the driver's containers on Docker and Podman. It checks the pinned image ID, that the workload starts as the OCI identity, no network or credential mounts, and a separate non-root supervisor. podman_oci_identity.rs keeps only the Podman-specific supervisor checks.

The driver code is unchanged from what you reviewed, and the rest is test cleanup. CI on 2bb7f665e passes, including all four oci-image jobs.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar

The shared OCI suite and workspace-rejection reporting look good. The expected fixture image ID is now captured before sandbox creation, with image cleanup preserved on inspection failure. Branch Checks and Core E2E pass on fde1cc5, including the Docker rootful DEB, Podman rootful/rootless RPM, and ARM64 Docker OCI lanes.

The previously discussed #4030 identity scenarios remain nonblocking follow-ups. When updating #4120, we'll also need to migrate the OCI suite added here to e2e/suites/features/oci-image, adopt e2e/support/rust, and update its archive, task, and workflow references. That migration does not block this PR.

This was referenced Sep 30, 2026

fix(podman): create managed workspace volumes owned by the workload identity #3932

Merged

feat(podman): honor OCI image working directories #3933

Merged

matthewgrossman added this pull request to stack #3936 September 30, 2026 06:06

matthewgrossman mentioned this pull request Sep 30, 2026

bug: rootful Podman managed workspace is unwritable for non-root workloads #3937

Closed

4 tasks

elezar mentioned this pull request Sep 30, 2026

test(podman): run existing Podman behavioral e2e coverage in CI #3712

Open

9 tasks

matthewgrossman added the test:e2e Requires end-to-end coverage label Sep 30, 2026

elezar reviewed Sep 30, 2026

View reviewed changes

Comment thread tests/suites/features/oci-image/tests/oci_image.rs Outdated

matthewgrossman changed the title ~~test(oci): share OCI image checks across Docker and Podman~~ fix(oci): share OCI image checks across runtimes and report unusable WORKDIRs Sep 30, 2026

matthewgrossman force-pushed the podman-workdir/a-oci-image-suite branch from 33f5e2d to 9bf8773 Compare September 30, 2026 18:33

matthewgrossman mentioned this pull request Sep 30, 2026

refactor(sandbox): remove unreachable root-side identity and workspace code #3979

Merged

7 of 8 tasks

matthewgrossman removed this pull request from stack #3936 September 30, 2026 18:46

matthewgrossman changed the base branch from main to podman-workdir/0-remove-dead-workspace-code September 30, 2026 18:52

This was referenced Sep 30, 2026

fix(podman): create managed workspace volumes owned by the workload identity #3981

Merged

feat(podman): honor OCI image working directories #3982

Merged

matthewgrossman changed the base branch from podman-workdir/0-remove-dead-workspace-code to podman-workdir/c-oci-workdir September 30, 2026 18:53

matthewgrossman added this pull request to stack #3983 September 30, 2026 18:53

matthewgrossman changed the title ~~fix(oci): share OCI image checks across runtimes and report unusable WORKDIRs~~ test(oci): share OCI USER and WORKDIR checks across Docker and Podman Sep 30, 2026

matthewgrossman force-pushed the podman-workdir/a-oci-image-suite branch from 9bf8773 to e65a902 Compare September 30, 2026 19:54

matthewgrossman force-pushed the podman-workdir/a-oci-image-suite branch from e65a902 to f184d63 Compare September 30, 2026 23:20

elezar mentioned this pull request Oct 1, 2026

bug: sandbox from a USER root image fails with an unexplained "UID or GID zero" error #4030

Open

6 tasks

matthewgrossman force-pushed the podman-workdir/a-oci-image-suite branch from f184d63 to 7658c9f Compare October 1, 2026 19:15

elezar requested review from a team and derekwaynecarr as code owners October 8, 2026 10:07

matthewgrossman added 3 commits October 8, 2026 09:11

test(oci): remove migrated legacy workspace helpers and checks

9642f7e

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

test(oci): require WorkspaceValidationFailed for unwritable WORKDIR

2b1c5d2

Unrelated provisioning failures can no longer pass the rejection scenario. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman mentioned this pull request Oct 8, 2026

test(oci): move OCI image USER/WORKDIR checks into a shared tmachine feature suite #4346

Closed

7 tasks

matthewgrossman changed the title ~~test(oci): share OCI USER and WORKDIR checks across Docker and Podman~~ fix(oci): report workspace rejections and share OCI USER and WORKDIR checks across Docker and Podman Oct 8, 2026

matthewgrossman added 2 commits October 9, 2026 13:33

Merge remote-tracking branch 'origin/main' into podman-workdir/a-oci-…

2ce1cdb

…image-suite Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

elezar previously approved these changes Oct 12, 2026

View reviewed changes

matthewgrossman added 2 commits October 12, 2026 08:53

Merge remote-tracking branch 'origin/main' into podman-workdir/a-oci-…

255f73d

…image-suite Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

matthewgrossman dismissed elezar’s stale review via e69b2e4 October 12, 2026 16:01

matthewgrossman mentioned this pull request Oct 12, 2026

test(oci): cover image USER and policy identity resolution in the oci-image suite #4431

Closed

matthewgrossman changed the title ~~fix(oci): report workspace rejections and share OCI USER and WORKDIR checks across Docker and Podman~~ test(oci): add a shared Docker and Podman suite for image USER and WORKDIR Oct 12, 2026

matthewgrossman mentioned this pull request Oct 12, 2026

feat(podman): honor OCI image working directories #3801

Closed

7 tasks done

matthewgrossman mentioned this pull request Oct 12, 2026

test(e2e): add cross-driver image-to-sandbox conformance #2422

Open

7 tasks

test(oci): capture fixture image identity before sandbox creation

fde1cc5

Signed-off-by: Evan Lezar <elezar@nvidia.com>

elezar approved these changes Oct 13, 2026

View reviewed changes

elezar added this pull request to the merge queue Oct 13, 2026

Merged via the queue into main with commit 8c67c27 Oct 13, 2026
116 checks passed

elezar deleted the podman-workdir/a-oci-image-suite branch October 13, 2026 13:12

Navigation Menu

Repository navigation

test(oci): add a shared Docker and Podman suite for image USER and WORKDIR - #3931

test(oci): add a shared Docker and Podman suite for image USER and WORKDIR#3931
elezar merged 18 commits into
mainfrom
podman-workdir/a-oci-image-suite

matthewgrossman commented Sep 30, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

matthewgrossman commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

elezar commented Sep 30, 2026

Uh oh!

Uh oh!

elezar commented Oct 1, 2026

Uh oh!

elezar left a comment

Uh oh!

matthewgrossman commented Oct 12, 2026

Uh oh!

matthewgrossman commented Oct 12, 2026 •

edited

Loading

Uh oh!

elezar left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

matthewgrossman commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

The suite

Other changes

Follow-ups

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

matthewgrossman commented Sep 30, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026

Uh oh!

elezar commented Sep 30, 2026

Uh oh!

Uh oh!

elezar commented Oct 1, 2026

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

matthewgrossman commented Oct 12, 2026

Uh oh!

matthewgrossman commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elezar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

matthewgrossman commented Sep 30, 2026 •

edited

Loading

matthewgrossman commented Oct 12, 2026 •

edited

Loading