Repository navigation
fix(podman): trust an operator-supplied additional CA for direct egress - #3996
Conversation
Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to destinations signed by that CA fail upstream TLS establishment, and the only workaround is rebuilding the supervisor image with the CA baked in. Add a Podman-only additional_ca_bundle setting, independent of the existing proxy_ca_bundle (which requires https_proxy), that bind-mounts an operator PEM into the supervisor container and folds it into the upstream trust store alongside the system CA bundle. Validated on the gateway host at sandbox-create time, matching the existing corporate proxy CA bundle's validation, so a misconfigured path or certificate-free bundle fails closed immediately rather than producing a failed sandbox once the supervisor container starts. Scoped to Podman only, reusing the existing additive-bundle mechanism without new gateway.toml schema, proto fields, or cross-driver plumbing -- a narrower alternative to the in-flight cross-driver implementation in NVIDIA#3292, which remains the intended long-term direction. Fixes NVIDIA#3781. Signed-off-by: politerealism <burdcat17@gmail.com>
c32c785 to
729061e
Compare
|
Rebased onto current main (the @sjenning @elezar could one of you take a look when you have a chance, and run |
|
/ok-to-test 729061e |
TaylorMutch
left a comment
There was a problem hiding this comment.
The additive CA trust approach makes sense, but the final Podman container split removes the CA mount from the supervisor, so enabling the setting prevents sandbox startup. Please fix the final-spec wiring and add coverage for build_isolation_specs(). The linked issue #3781 is accepted. I reviewed the full diff and surrounding provisioning code; I did not run tests locally.
build_isolation_specs() keeps only trusted_mount() destinations on the supervisor, and the additional CA path was not in that list. The bind mount stayed on the workload while the supervisor received --additional-ca-bundle for a file it did not have, so startup failed. Add the path to trusted_mount() and cover the final supervisor and workload specs with a test. Correct the docs and a code comment to say the bundle is validated when the driver starts, and add Podman troubleshooting guidance to the debug-openshell-cluster skill. Signed-off-by: politerealism <burdcat17@gmail.com>
|
Thanks, both addressed in ec7691f. |
|
/ok to test ec7691f |
TaylorMutch
left a comment
There was a problem hiding this comment.
Re-reviewed ec7691f. Both findings are addressed: the additional CA mount now survives the final supervisor/workload split, with a regression test covering the final specs, and the validation timing plus deployment troubleshooting guidance are corrected. No remaining blocking concerns found. All executed checks and required gates passed on this head, including Rust tests on Linux x86_64, Linux ARM64, and macOS ARM64. E2E suites were skipped by workflow configuration; I did not run tests locally.
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Unify the additional destination CA series on the mechanism merged to main in NVIDIA#3996: one operator pre-assembled PEM bundle, the --additional-ca-bundle supervisor flag, and the /etc/openshell/tls/additional/ca-bundle.pem staging path. - Replace the gateway-level additional_ca_cert_paths list with a single additional_ca_bundle file validated and staged at gateway startup. - Restore the upstream supervisor argument path and fold-into-roots behavior; drop the --network-additional-ca-bundle and --network-additional-ca-digest contracts. - Keep gateway normalization bounds, content-addressed staging, digest-based driver reconciliation, and stop/start rotation; the freshness check now runs at the gateway/driver boundary instead of the supervisor boundary. - Reject additional_ca_bundle in operator driver JSON config; standalone drivers self-source the bundle from --sandbox-additional-ca-bundle with the same validation. - Extend the Kubernetes ConfigMap name digest segment to keep the full 63-byte budget after the prefix rename. - Box the persisted-sandbox reconciliation futures so callers stay under the clippy::large_futures threshold. Updates Docker, Podman, Kubernetes, and VM delivery, Helm chart and tests, e2e wiring, gateway configuration docs, and the cluster debugging skill to the single mechanism. Signed-off-by: Jesse Jaggars <jjaggars@redhat.com>
Summary
Since v0.1.2 the Podman supervisor runs as its own container, separate from the workload image, so it can no longer inherit a private CA baked into the workload's system trust store. Policy-allowed, inspected HTTPS requests to a destination signed by that CA fail upstream TLS establishment (
curl: (52) Empty reply from server), and the only workaround is rebuilding the supervisor image with the CA baked in per upgrade or rotation.This adds a Podman-only
additional_ca_bundlesetting that restores that trust without a custom image.Related Issue
Fixes #3781.
This is a deliberately narrower, Podman-only alternative to the in-flight cross-driver implementation in #3292 (Docker/Podman/Kubernetes/VM delivery, gateway-level config schema, lifecycle rotation reconciliation), which remains the intended long-term direction — see the note posted on #3781. There are a few other things in flight that need to settle before taking on a change of #3292's size, so this fix is scoped to just what #3781 needs.
Changes
additional_ca_bundletoPodmanComputeConfig, independent of the existingproxy_ca_bundle(which requireshttps_proxyto be set) — this applies to direct, inspected egress regardless of whether a corporate proxy is configured.--additional-ca-bundlesupervisor flag. The mount path is listed intrusted_mount()so the supervisor keeps it and the workload does not.tls.rs's trust-store construction itself.docs/how-it-works/gateways/configuration.mdxanddocs/how-it-works/sandboxes/runtimes.mdx.Deliberately out of scope: Docker/Kubernetes/VM delivery, gateway.toml schema, new proto fields, and lifecycle/rotation reconciliation for stopped sandboxes — all left to #3292 or a future follow-up.
Testing
mise run pre-commitpassescargo fmt --all -- --checkpassescargo clippy -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --all-targets -- -D warningspassescargo test -p openshell-driver-podman -p openshell-supervisor-network -p openshell-supervisor -p openshell-core --lib— 528 + 232 + 128 + 1458 tests pass, 0 failuresadditional_ca_enables_trust_of_privately_signed_upstreamintls.rs): a genuine rustlsTlsAcceptor/TlsConnectorhandshake over a real loopback connection, reproducing bug: Podman supervisor cannot trust operator-supplied CA for upstream TLS after v0.1.2 architecture change #3781's exact failure without the CA trusted and proving it succeeds once it iscontainer.rs), including the no-https_proxycase that's the actual point of this fixbuild_isolation_specs()tested directly: the final supervisor spec keeps the bind mount and--additional-ca-bundleargv, and the workload spec excludes the mountChecklist