Skip to content

fix(podman): trust an operator-supplied additional CA for direct egress - #3996

Merged
TaylorMutch merged 2 commits into
NVIDIA:mainfrom
politerealism:3781-podman-additional-ca-minimal
Oct 15, 2026
Merged

TaylorMutch merged 2 commits into
NVIDIA:mainfrom
politerealism:3781-podman-additional-ca-minimal

Conversation

@politerealism

@politerealism politerealism commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to a destination signed by that CA fail upstream TLS establishment (curl: (52) Empty reply from server), and the only workaround is rebuilding the supervisor image with the CA baked in per upgrade or rotation.

This adds a Podman-only additional_ca_bundle setting that restores that trust without a custom image.

Related Issue

Fixes #3781.

This is a deliberately narrower, Podman-only alternative to the in-flight cross-driver implementation in #3292 (Docker/Podman/Kubernetes/VM delivery, gateway-level config schema, lifecycle rotation reconciliation), which remains the intended long-term direction — see the note posted on #3781. There are a few other things in flight that need to settle before taking on a change of #3292's size, so this fix is scoped to just what #3781 needs.

Changes

  • Add additional_ca_bundle to PodmanComputeConfig, independent of the existing proxy_ca_bundle (which requires https_proxy to be set) — this applies to direct, inspected egress regardless of whether a corporate proxy is configured.
  • Validate the bundle on the gateway host when the Podman driver starts (read and parse it, not just check non-empty), so a misconfigured path or certificate-free bundle fails gateway startup immediately instead of producing a failed sandbox once the supervisor container starts.
  • Bind-mount the bundle read-only into the supervisor container and pass its path via a new --additional-ca-bundle supervisor flag. The mount path is listed in trusted_mount() so the supervisor keeps it and the workload does not.
  • Fold it into the supervisor's upstream TLS trust store alongside (not instead of) the system CA bundle, reusing the exact additive-append mechanism already proven for the corporate proxy CA case — no changes to tls.rs's trust-store construction itself.
  • The CA is trusted for all destinations by both the supervisor and the workload (it also lands in the sandbox trust bundle), so the docs advise using a CA you control or one with name constraints.
  • Document the new setting in docs/how-it-works/gateways/configuration.mdx and docs/how-it-works/sandboxes/runtimes.mdx.

Deliberately out of scope: Docker/Kubernetes/VM delivery, gateway.toml schema, new proto fields, and lifecycle/rotation reconciliation for stopped sandboxes — all left to #3292 or a future follow-up.

Testing

  • mise run pre-commit passes
  • cargo fmt --all -- --check passes
  • cargo clippy -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --all-targets -- -D warnings passes
  • cargo test -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --lib — 528 + 232 + 128 + 1458 tests pass, 0 failures
  • Real end-to-end TLS regression test added (additional_ca_enables_trust_of_privately_signed_upstream in tls.rs): a genuine rustls TlsAcceptor/TlsConnector handshake over a real loopback connection, reproducing bug: Podman supervisor cannot trust operator-supplied CA for upstream TLS after v0.1.2 architecture change #3781's exact failure without the CA trusted and proving it succeeds once it is
  • Config→mount→argv wiring tested directly (container.rs), including the no-https_proxy case that's the actual point of this fix
  • Fail-closed validation tested for empty, nonexistent, and certificate-free bundle paths
  • build_isolation_specs() tested directly: the final supervisor spec keeps the bind mount and --additional-ca-bundle argv, and the workload spec excludes the mount

Checklist

  • Follows Conventional Commits
  • Commit is signed off (DCO)
  • Architecture/user-facing documentation updated

@copy-pr-bot

copy-pr-bot Bot commented Sep 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Since v0.1.2 the Podman supervisor runs as its own container, separate
from the workload image, so it can no longer inherit a private CA
baked into the workload's system trust store. Policy-allowed,
inspected HTTPS requests to destinations signed by that CA fail
upstream TLS establishment, and the only workaround is rebuilding the
supervisor image with the CA baked in.

Add a Podman-only additional_ca_bundle setting, independent of the
existing proxy_ca_bundle (which requires https_proxy), that bind-mounts
an operator PEM into the supervisor container and folds it into the
upstream trust store alongside the system CA bundle. Validated on the
gateway host at sandbox-create time, matching the existing corporate
proxy CA bundle's validation, so a misconfigured path or
certificate-free bundle fails closed immediately rather than producing
a failed sandbox once the supervisor container starts.

Scoped to Podman only, reusing the existing additive-bundle mechanism
without new gateway.toml schema, proto fields, or cross-driver
plumbing -- a narrower alternative to the in-flight cross-driver
implementation in NVIDIA#3292, which remains the intended long-term
direction.

Fixes NVIDIA#3781.

Signed-off-by: politerealism <burdcat17@gmail.com>
@politerealism
politerealism force-pushed the 3781-podman-additional-ca-minimal branch from c32c785 to 729061e Compare October 13, 2026 12:39
@politerealism

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (the run_sandbox arguments moved into SandboxRunConfig, so additional_ca_bundle is threaded through that struct now) and clarified in the docs that the CA is trusted for all destinations by both the supervisor and the workload. New head is 729061e.

@sjenning @elezar could one of you take a look when you have a chance, and run /ok-to-test 729061e830f5b525b655455ec64d5d4fa28894e7 so CI can run on the new commit? This is the narrower Podman-only fix for #3781.

@elezar

elezar commented Oct 13, 2026

Copy link
Copy Markdown
Member

/ok-to-test 729061e

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The additive CA trust approach makes sense, but the final Podman container split removes the CA mount from the supervisor, so enabling the setting prevents sandbox startup. Please fix the final-spec wiring and add coverage for build_isolation_specs(). The linked issue #3781 is accepted. I reviewed the full diff and surrounding provisioning code; I did not run tests locally.

Comment thread crates/openshell-driver-podman/src/container.rs
Comment thread docs/how-it-works/gateways/configuration.mdx Outdated
build_isolation_specs() keeps only trusted_mount() destinations on the
supervisor, and the additional CA path was not in that list. The bind
mount stayed on the workload while the supervisor received
--additional-ca-bundle for a file it did not have, so startup failed.

Add the path to trusted_mount() and cover the final supervisor and
workload specs with a test. Correct the docs and a code comment to say
the bundle is validated when the driver starts, and add Podman
troubleshooting guidance to the debug-openshell-cluster skill.

Signed-off-by: politerealism <burdcat17@gmail.com>
@politerealism

Copy link
Copy Markdown
Contributor Author

Thanks, both addressed in ec7691f. ADDITIONAL_CA_MOUNT_PATH is now in trusted_mount(), and a new test runs build_isolation_specs() to check the final supervisor keeps the mount and argv while the workload doesn't. The docs and code comment now say the bundle is validated when the driver starts, and I added Podman troubleshooting to the debug-openshell-cluster skill (startup validation failure, and how to check the supervisor mount and argv).

@TaylorMutch

Copy link
Copy Markdown
Collaborator

/ok to test ec7691f

@TaylorMutch TaylorMutch left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed ec7691f. Both findings are addressed: the additional CA mount now survives the final supervisor/workload split, with a regression test covering the final specs, and the validation timing plus deployment troubleshooting guidance are corrected. No remaining blocking concerns found. All executed checks and required gates passed on this head, including Rust tests on Linux x86_64, Linux ARM64, and macOS ARM64. E2E suites were skipped by workflow configuration; I did not run tests locally.

@TaylorMutch
TaylorMutch added this pull request to the merge queue Oct 15, 2026
Merged via the queue into NVIDIA:main with commit 10241ab Oct 15, 2026
78 of 79 checks passed
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 16, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 16, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 16, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 16, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 16, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 17, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 17, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
jhjaggars added a commit to jhjaggars/OpenShell that referenced this pull request Oct 17, 2026
Unify the additional destination CA series on the mechanism merged to
main in NVIDIA#3996: one operator pre-assembled PEM bundle, the
--additional-ca-bundle supervisor flag, and the
/etc/openshell/tls/additional/ca-bundle.pem staging path.

- Replace the gateway-level additional_ca_cert_paths list with a single
  additional_ca_bundle file validated and staged at gateway startup.
- Restore the upstream supervisor argument path and fold-into-roots
  behavior; drop the --network-additional-ca-bundle and
  --network-additional-ca-digest contracts.
- Keep gateway normalization bounds, content-addressed staging,
  digest-based driver reconciliation, and stop/start rotation; the
  freshness check now runs at the gateway/driver boundary instead of
  the supervisor boundary.
- Reject additional_ca_bundle in operator driver JSON config; standalone
  drivers self-source the bundle from --sandbox-additional-ca-bundle
  with the same validation.
- Extend the Kubernetes ConfigMap name digest segment to keep the full
  63-byte budget after the prefix rename.
- Box the persisted-sandbox reconciliation futures so callers stay
  under the clippy::large_futures threshold.

Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and
tests, e2e wiring, gateway configuration docs, and the cluster
debugging skill to the single mechanism.

Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Podman supervisor cannot trust operator-supplied CA for upstream TLS after v0.1.2 architecture change

3 participants