feat(network): support additional destination CAs

[jhjaggars]

Summary

Add a global network-supervisor configuration for additional destination CA certificates so sandbox egress can trust private PKI without replacing default roots or changing gateway control-plane trust.

This series adopts the mechanism merged to main in #3996 (from #3781) and extends it to every first-party compute driver: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path.

Related Issue

Fixes #3295

Changes

• Add [openshell.supervisor.network].additional_ca_bundle (single pre-assembled PEM file) with strict, bounded startup validation and fail-closed capability-based driver handling.
• Keep effective-config preflight side-effect-free; stage gateway-owned trust material only after the selected driver passes compatibility checks.
• Extend the shared network supervisor with additive rustls and child-process trust while preserving user TLS variables in direct mode and keeping proxy CA and gateway mTLS trust separate.
• Stage the normalized bundle through read-only Docker/Podman mounts, Kubernetes immutable content-addressed ConfigMaps, and VM overlays, all at the upstream mount path.
• Record and validate the expected trust generation at the gateway/driver boundary so mutable or stale runtime material fails closed.
• Reconcile stopped sandboxes whose recorded trust generation differs from the restarted gateway snapshot while retaining sandbox identity and supported durable workspace contents.
• Restrict Kubernetes ConfigMap RBAC to the operations required for immutable generation creation and validation; do not grant list, watch, patch, update, or delete for this feature.
• Enforce one deployable, bounded trust-bundle limit across gateway, Kubernetes, VM, and supervisor consumers.
• Reject additional_ca_bundle in operator driver JSON config so the global gateway setting stays the single source; standalone drivers self-source the same bundle from --sandbox-additional-ca-bundle with identical validation.
• Add shared cross-driver additional-CA coverage for private/public trust, hostname enforcement, invalid staged material, callback isolation, and removal/restart behavior.
• Update gateway reference, architecture, RFC, Helm, compute-driver, and cluster-debugging documentation.

Why lifecycle reconciliation is included

Additional CA sources are startup-only gateway configuration. Updating the source file or Helm source ConfigMap does not hot-reload running supervisors. To roll certificates, an operator updates the source, restarts or redeploys the gateway, and explicitly stop/starts each affected sandbox.

Each runtime records the immutable trust generation it was created with. Without reconciliation, ordinary stop/start cannot reliably move an existing sandbox to the new generation: Docker and Podman retain old container metadata, Kubernetes retains the old content-addressed ConfigMap mount, and VM retains the old writable overlay. Replacing only the staged file would conflict with the recorded digest and correctly fail closed.

The only existing manual alternative is to delete and recreate each affected sandbox, or create a replacement sandbox and move work to it. Sandbox deletion normally removes its driver-owned mutable filesystem storage—Docker/Podman workspace volumes, the Kubernetes sandbox PVC, or the VM writable overlay. Operators would therefore need to back up and restore required /sandbox contents, use external persistent storage, and re-establish any sandbox-specific state. The reconciliation implemented here avoids that migration by rebuilding stale runtime resources during stop/start while preserving the sandbox record and supported durable workspace data.

This lifecycle work can be split into a follow-up only if destructive replacement plus explicit data migration is an acceptable interim certificate-rotation procedure.

Testing

• mise run pre-commit passes
• Workspace formatting and Clippy with warnings denied pass
• Targeted Rust tests pass across core, sandbox, supervisor-network, Docker, Kubernetes, Podman, VM, server, and gateway packages
• Helm tests pass: gateway and workspace chart suites including Rust loader validation of the rendered gateway TOML
• Unit and process-level tests added/updated
• Cross-driver E2E coverage added and feature combinations compile
• Rootless Podman manually verified against a private-PKI GitLab endpoint: TLS failed without the additional CA and succeeded with it while default public trust remained enabled
• Full mise run test task run as a single validation command
• Docker, Kubernetes combined/sidecar, and VM E2E runtime lanes (configured for CI; not run locally)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture and user-facing configuration documentation updated

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[jhjaggars]

 [openshell.supervisor.network]
   additional_ca_cert_paths = [
     "/tmp/openshell-e2e-podman.McovxM/additional-ca/ca.crt",
   ]

[politerealism]

Summary

The design is sound and directly addresses both #3781 (Podman TLS-trust regression) and #3295 (general additional-destination-CA feature): trust material is scoped to the supervisor container only, root-store construction is fixed for both the bundled-ca-roots and native-store variants, validation is fail-closed end-to-end, and the gateway-mTLS trust boundary is provably untouched (new destination_ca_cannot_authenticate_gateway regression test). However, the branch as submitted (eb2db3f65) doesn't compile its own test suite, so CI can't pass yet.

Blocking

cargo test -p openshell-driver-podman -p openshell-driver-docker -p openshell-server (and mise run pre-commit's rust:lint step) fail to compile, all four attributable to this PR's own commits:

• crates/openshell-driver-podman/src/container.rs:1833 — new test additional_destination_trust_is_delivered_only_to_supervisor_with_digest is missing the pre-existing required field rootless on IsolationSpecInput.
• crates/openshell-driver-podman/src/container.rs:2020 — existing rootful_specs = build_isolation_specs(...) wasn't updated with the new required field network_trust.
• crates/openshell-driver-docker/src/tests.rs:135 — sets daemon_version: "28.0.0".to_string(), but DockerDriverRuntimeConfig has no such field.
• crates/openshell-server/src/compute/mod.rs:13576 — new test start_sandbox_request_uses_optional_wire_field_six_for_durable_snapshot is missing the pre-existing required field expected_runtime_identity on StartSandboxRequest.

openshell-driver-kubernetes, openshell-driver-vm, and openshell-gateway build and test cleanly in isolation, so this is localized to four struct literals, not systemic.

Non-blocking notes

• deploy/helm/openshell/templates/clusterrole.yaml:82-97 / role.yaml:65-70: RBAC grants unscoped get on ConfigMaps (in addition to create), since resourceNames can't restrict dynamically digest-named generations. That lets the gateway service account read any ConfigMap in the bound scope, not just its own CA generations. Reasoning is documented in-line and in the debug-openshell-cluster skill update, but worth a second look given it's a real (if bounded) privilege widening.
• crates/openshell-supervisor-network/src/run.rs:212-219: the additional-CA load path fails closed correctly but doesn't emit a ConfigStateChangeBuilder OCSF event the way the adjacent TLS-termination code does at run.rs:401-429. Not required by AGENTS.md's OCSF scope (that's openshell-sandbox-specific), but would be a nice consistency improvement for operator visibility.

Testing

cargo fmt --all -- --check passes. cargo test -p openshell-supervisor-network -p openshell-core -p openshell-supervisor passes (1344/1345, one pre-existing unrelated flake). Docker/Kubernetes/VM e2e lanes haven't been run (not available in my review environment) and remain genuinely unverified — recommend running those before merge, in addition to fixing the compile errors above.

[github-actions]

This pull request has had no activity for 14 days and is now marked stale.

[TaylorMutch]

gator-agent

Blocked

Issue #3295 now has state:accepted; I checked the linked issue and confirmed that the failed acceptance run predates that update. PR #3292 is open, ready for review, mergeable, and passing DCO, but Branch Checks and Helm Lint are waiting for the current-head test mirror. Independent Gator code review has not run because this process gate remains blocked.

Action required: a maintainer must rerun PR Issue Acceptance, apply test:e2e for the network-trust and cross-driver lifecycle changes, and post /ok to test e94faaa35d251edb5bdb2bab9e9adf1ae54120fd as a separate command-only comment. Follow the E2E Label Help instructions and confirm the required workflows actually start. The operator has reserved test labels and /ok to test for a maintainer.

No test dispatch, approval, merge, or contributor-branch changes were performed.

Gator metadata

• Head SHA: e94faaa35d251edb5bdb2bab9e9adf1ae54120fd
• Base SHA: 35a3390dbc03112805cf201bb3fd99f4836a77ae
• Merge base SHA: 35a3390dbc03112805cf201bb3fd99f4836a77ae
• Patch ID: d5a7742b728213c34490bcfdcff56f494360d827
• Gator payload: 12
• Code review: not started; process gate blocked
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[mrunalp]

Thanks for this. Private-PKI egress is a real gap, and the core trust handling is careful: the bounded single read, re-encoded PEM, a content-addressed artifact that is never overwritten, drivers having to opt in, and destination trust kept separate from gateway mTLS. The concerns below are mostly about the stop/start rebuild ("reconciliation") work and upgrades.

Blocking

1. Upgrading puts running Docker/Podman sandboxes into Error, even with no CA configured

When the gateway restarts, recovery calls start_sandbox for sandboxes in a running phase. Before this PR, the driver treated an already-running container as a no-op success. Now network_trust_generation_matches() runs before that check. A missing label counts as a mismatch, and every container created before this PR has no label. So the driver returns refusing to replace a running … workload, and recovery marks the sandbox StartFailed.

• Podman: crates/openshell-driver-podman/src/driver.rs ~1837-1843
• Docker: crates/openshell-driver-docker/src/lib.rs ~2735-2740
• Recovery error path: crates/openshell-server/src/compute/mod.rs ~3884

Rotating the CA hits the same path: change the bundle, restart the gateway, and every running sandbox goes to Error. That contradicts the documented flow, where running supervisors keep their startup trust and operators stop/start each sandbox to pick up the new bundle.

2. On upgrade, every stopped Docker/Podman sandbox is rebuilt, which can lose data or block start

Because a missing label is never treated as none, the first start after upgrade rebuilds each stopped sandbox, even on gateways with no CA configured:

• Docker copies only workspace_root into the replacement (copy_docker_workspace_archive, ~4876). Anything else changed in the container's filesystem is lost, such as installed packages or files in home outside the workspace.
• That archive copy rejects any entry that isn't a regular file or directory, so symlinks and hard links are refused. Those are common in Python venvs and node_modules/.bin. It also caps the copy at MAX_WORKSPACE_ARCHIVE_BYTES (512 MiB). Affected sandboxes fail to start, every time, where stop/start used to work.
• Podman: please check images whose WorkingDir isn't /sandbox. As far as I can tell they get no managed volume and no archive copy, so their workspace isn't carried over.
• The image is re-resolved from the stored snapshot, so a tag that has moved will silently upgrade the image.

The Docker rebuild also seems unnecessary. Docker already re-spawns the supervisor from current config on every start (spawn_docker_control_process, ~2301), and the CA is never mounted into the workload, so the rebuild seems to change only the label. For Podman, recreating just the supervisor container should be enough.

Suggestion: treat a missing label as none, and/or only rebuild the supervisor side.

3. Recovery now marks sandboxes Error for config problems unrelated to their record

crates/openshell-server/src/compute/mod.rs ~3750 (also ~2198, ~4037): when driver_start_request fails during recovery, the code now calls persist_invalid_start_snapshot. On main, recovery logged a warning and skipped the sandbox. driver_start_request also fails for gateway configuration problems, for example validate_launch_signer_configured when an isolation backend is configured but the JWT signer is missing.

With this PR, those sandboxes go to Error/InvalidPersistedSandbox, with no !recoverable_error guard like the neighbouring error paths have. That reason isn't covered by is_recoverable_error_reason, so the sandboxes are skipped on every later restart, even after the operator fixes the config. The error message, which says the persisted driver snapshot is invalid, also points at the wrong cause. The test change from start_persisted_sandboxes() to the authenticated variant (~17243) seems to reflect this behaviour change.

Should fix

• Migration from fix(podman): trust an operator-supplied additional CA for direct egress #3996 (v0.1.5-pre.4):
  • [openshell.drivers.podman] additional_ca_bundle now fails with a plain serde "unknown field" error (#[serde(skip)] combined with deny_unknown_fields). A targeted message pointing to [openshell.supervisor.network] would help. The current rejection test passes an array, so it fails on the type and doesn't cover this case.
  • docs/how-it-works/sandboxes/runtimes.mdx:205 still documents the Podman driver option, and still says the workload trusts the CA.
• Standalone Podman --sandbox-additional-ca-bundle: operator_network_trust checks the operator's file directly, but verify_artifact requires mode exactly 0444. A typical 0644 CA file lets the driver start, then every create and start fails. Separately, the standalone path hashes the raw file bytes, while the gateway hashes the re-encoded bundle, so the same CA file gives two different generations.
• Podman rollback on failure (please check):
  • The cleanup paths (driver.rs ~1300-1346) look like they delete the per-sandbox TLS/proxy-auth secrets even in reuse mode. The restored original pair would then fail to start.
  • An error after both renames, for example from secret_exists(...)? at ~1745, or a failed final remove_container_preserving_volumes (~1791), would leave *-rollback-* containers that delete_sandbox never cleans up.
• Kubernetes ConfigMap validation: the exact label comparison (driver.rs:186) rejects ConfigMaps whose labels were added by mutating admission webhooks, such as Kyverno or Gatekeeper mutation. The ConfigMap is immutable and the gateway can't delete it, so sandbox creates in that namespace fail until an admin deletes it by hand. Checking that our owned labels are present (a subset match) would avoid this.
• Helm: _toml.tpl ~312 rejects gatewayConfig.openshell.supervisor.network only when additionalCaConfigMapName is set. Setting additional_ca_bundle through gatewayConfig directly renders, but no ConfigMap RBAC is granted, so Kubernetes sandbox creates fail with 403.
• CI: the new Kubernetes additional-CA row pins agent_sandbox_version: v0.5.0 (branch-e2e.yml:436). That looks like a rebase leftover, since main uses v1.0.3.

Smaller items / nits

• sandbox_runtime.rs:385 ignores the digest (_digest). Either pass it to the supervisor and verify it there, or drop the parameter and the pre-ungate re-checks. As written, they don't close the window before kubelet mounts the ConfigMap.
• CA ConfigMaps are never deleted (no ownerReferences, no delete verb). Each rotation leaves one more per namespace. The comment at ~165 about stopped PVCs needing old material doesn't seem to apply, since resume always mounts the current bundle.
• The PEM structure check accepts any upper-case label, and rustls then silently skips unknown blocks such as TRUSTED CERTIFICATE. This fails closed, but contradicts the "no valid subset is accepted" doc comment.
• e2e:
  • The public-trust check depends on https://example.com/ (additional_ca.rs:885), so it isn't hermetic.
  • The hostname check likely fails on the supervisor's intercept certificate rather than on hostname validation against the private CA.
  • The removal check accepts any non-zero curl exit.
  • 172.0.0.0/8 in allowed_ips was probably meant to be 172.16.0.0/12.
• The Docker additional-ca row is added to the default matrix, so it also runs in release-dev and release-qualification. Is that intended?
• skills/debug-openshell-cluster/SKILL.md ~542-547 names the wrong annotation (the code uses openshell.ai/additional-ca-generation). It also describes a supervisor digest argument and a network-init re-check that don't exist.
• rfc/0003-gateway-configuration/README.md still says additional_ca_cert_paths (a list), but the implementation is additional_ca_bundle (a single path).
• Unused or duplicated code:
  • TlsEnvironmentMode::FillMissing and its reassert loop have no caller.
  • normalized_pem() duplicates pem().
  • network_trust and network_trust_configured overlap.
  • The VM driver re-implements PEM normalization even though the digest already matches.
• openshell-core adds a direct libc dependency, not pinned in the workspace, just for open flags that rustix/nix already provide.
• The configuration.mdx section is heavy on internals (generations, content addressing, RBAC), and repeats material from the Helm README and the skill.
• The PR body says feat: trust additional destination CAs for sandbox egress TLS #3295 is awaiting state:accepted. It's accepted now.

Overall

The PR body offers to split out the lifecycle reconciliation, and I'd take that offer. Most of the risk above comes from the stop/start rebuild machinery, not from CA delivery. A smaller first PR could do CA delivery and validation, plus "stop/start picks up the current bundle" by recreating only the supervisor, as the Kubernetes driver already does. That would land the feature with far less upgrade risk. Rebuilds that preserve the workspace could follow, with their own upgrade tests.

[TaylorMutch]

gator-agent

Blocked

Issue #3295 is accepted, and PR Issue Acceptance and DCO now pass on the latest head. Branch Checks and Helm Lint are waiting for the current-head test mirror. The green E2E status means test:e2e is absent, not that E2E ran. Independent Gator code review has not started because the test-dispatch process gate remains blocked.

Thanks @mrunalp for identifying the upgrade and stop/start reconciliation risks. I verified your maintainer authority and checked the latest head, review history, and CI dispatch state. Your implementation feedback remains input for the eventual code review; this process check does not establish whether the latest changes address it.

Action required: a maintainer must apply test:e2e for network trust and cross-driver lifecycle coverage, then post /ok to test 7154342135224d7a841cb00d8068feabc4d0563b as a separate command-only comment. Follow the E2E Label Help instructions, approve any required workflow execution, and confirm Branch Checks, Helm Lint, Trivy Changes, and required E2E workflows actually start on the current head. Acceptance no longer needs a rerun.

Gator metadata

• Head SHA: 7154342135224d7a841cb00d8068feabc4d0563b
• Base SHA: 604b9f66f685c582114cd571ae3a10accec23ec2
• Merge base SHA: 604b9f66f685c582114cd571ae3a10accec23ec2
• Patch ID: 5dafe68e34718f726af916a4e0c75d323de06902
• Gator payload: 12
• Code review: not started; process gate blocked
• Next state: gator:blocked
• Blocked reason: test_dispatch_required

[github-actions]

Label test:e2e applied, but pull-request/3292 does not exist yet. A maintainer needs to comment /ok to test 7154342135224d7a841cb00d8068feabc4d0563b to mirror this PR. Once the mirror exists, re-apply the label or re-run Branch E2E Checks from the Actions tab.

[TaylorMutch]

/ok to test 7154342

[github-actions]

Compatibility review (advisory)

New or updated breaking changes; see the full report for all findings.

5. rust-sdk — No explicit risk acceptance found

openshell_sdk::raw::proto::compute::v1::StartSandboxRequest gains a new generated sandbox field.

• Impact: Raw Rust SDK clients, compute-driver mocks, and exhaustive patterns using the generated struct no longer compile until they supply or ignore the new field.
• Action: Remove the generated-field change from the patch train, or release it in a minor version with guidance to set sandbox: None or use ..Default::default().
• PR: feat(network): support additional destination CAs #3292

Review details

• Execution: complete
• Candidate: PR #3292 (44bf35384753a92d96fcafdac390e6c87c86c4a7)
• Baseline: 604b9f66f685c582114cd571ae3a10accec23ec2 (604b9f66f685c582114cd571ae3a10accec23ec2)
• Reviewed: rust-sdk, python-sdk, go-sdk, typescript-sdk, api-behavior, cli, configuration, policy, helm, state

Advisory review; publication gates are unchanged.

Branch check run