feat: trust additional destination CAs for sandbox egress TLS #3295

Problem Statement

Operators need sandboxes to make egress TLS connections to hosts that present certificates signed by a private or self-signed CA (an internal GitLab, artifact registry, or internal API). Today the sandbox's network supervisor trusts only the bundled public roots plus whatever the sandbox image happens to ship, so the handshake to such a host fails after the policy has already allowed the connection. Operators need a way to say "additionally trust these CA certificates for sandbox destination TLS" without replacing the default public roots and without changing the trust the supervisor uses to authenticate the gateway control plane.

Draft PR #3292 implements this feature. It originally proposed a gateway-level list of CA paths; after #3996 merged (see below), the PR was reworked to adopt that accepted mechanism and extend it to every first-party compute driver. This issue records the problem, the investigation of the current code on main, and the decisions a maintainer must make so that work can be accepted or declined.

Relationship to #3781 / #3996

#3781 (closed, state:accepted) fixed the Podman-only case: PodmanComputeConfig.additional_ca_bundle, a single operator pre-assembled PEM bundle bind-mounted into the supervisor container and folded into its upstream trust store via --additional-ca-bundle (merged as #3996). That fix is correct for its scope, but it is driver-local: Docker, Kubernetes (combined and sidecar), and VM sandboxes have no equivalent, and a gateway host path cannot reach Kubernetes sandbox pods on other nodes.

This issue tracks the cross-driver extension of that accepted mechanism: one gateway-level additional_ca_bundle (single pre-assembled PEM file), validated and staged by the gateway, delivered read-only by every first-party driver to the same /etc/openshell/tls/additional/ca-bundle.pem guest path, with the supervisor surface unchanged from #3996. The spike investigation below predates #3996 and is retained as the design record for the supervisor trust stores and delivery decisions.

User Story

As an operator running OpenShell in an environment with private PKI, I want to configure a set of additional CA certificates once at the gateway so that every sandbox can reach internal HTTPS services, while public sites keep working and the sandbox cannot use those certificates to impersonate the gateway.

Impact / Why This Matters

• A policy-allowed curl https://gitlab.internal fails inside the sandbox with an UnknownIssuer TLS error after the CONNECT succeeds, which is confusing and blocks agents that need internal services.
• The current workaround is to bake the CA into every sandbox image's system bundle. That requires an image rebuild per rotation, is owned by the image author rather than the operator, and does not work in the Kubernetes sidecar topology because the sidecar reads its own image's bundle, not the agent image's.
• The other workaround, tls: skip on the endpoint, disables L7 inspection for that host and still requires the CA in the image because the supervisor overrides SSL_CERT_FILE.
• proxy_ca_bundle exists only for Podman and VM, conflates corporate-proxy trust with destination trust, and refuses to load without an https_proxy URL.

Technical Context

OpenShell keeps three distinct trust stores, and any solution must keep them distinct:

1. Supervisor to gateway mTLS. The gRPC client trusts only the mounted gateway CA and the code explicitly forbids adding native or webpki roots, because the supervisor runs inside a user-selected image. This boundary must not change.
2. Destination TLS (proxy mode). The upstream root store is the webpki-roots bundle overlaid with the system bundle found at one of four fixed paths inside the image the supervisor runs in. In builds without the bundled-ca-roots feature, the native store is used and the system bundle is ignored.
3. Corporate proxy CA. An operator PEM delivered by argv that is appended to the system bundle string, so it reaches both the upstream root store and the child trust files. It is fail-closed and requires an https_proxy pairing.

Every gateway-managed sandbox runs in proxy mode: the supervisor generates an ephemeral CA, terminates TLS from the workload, inspects HTTP, and re-encrypts upstream with the root store above. Child processes get SSL_CERT_FILE, NODE_EXTRA_CA_CERTS, REQUESTS_CA_BUNDLE, and friends pointed at supervisor-written files, unconditionally overriding user values. Block and Allow modes exist only for local file-based policies; in those modes no TLS files or env vars are set.

Affected Components

Technical Investigation

Architecture Overview

Trust material flows gateway → compute driver → supervisor. The gateway reads TOML, merges CLI args, builds a DriverStartupContext (carrying gateway-owned inputs such as GuestTlsPaths), and the selected ComputeDriverFactory turns that into a driver config. Each driver stages files into the sandbox (Docker/Podman bind mounts, Kubernetes volumes, VM overlay files) and writes the supervisor's argv. The supervisor validates its inputs at startup, emits OCSF ConfigStateChange events, and fails closed on present-but-invalid operator material.

In proxy mode, run_networking generates the ephemeral CA, reads the system bundle, appends the corporate proxy CA if configured, builds the upstream ClientConfig, and writes two files: openshell-ca.pem (additive, used by NODE_EXTRA_CA_CERTS and DENO_CERT) and ca-bundle.pem (complete, used by SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, GIT_SSL_CAINFO). The proxy MITMs endpoints with TlsMode::Auto and verifies upstream against the root store; TlsMode::Skip endpoints and all non-proxy traffic are raw tunnels where the workload does its own TLS.

Where the supervisor runs matters for which "system bundle" is consulted: the sandbox image for Docker, Podman, VM, and Kubernetes combined mode, but the supervisor image for Kubernetes sidecar mode.

Code References

Current Behavior

curl https://gitlab.internal from a gateway-managed sandbox: curl is given HTTPS_PROXY and SSL_CERT_FILE=/etc/openshell-tls/ca-bundle.pem. The CONNECT is policy-evaluated and allowed. The proxy presents an ephemeral leaf that curl trusts, then calls tls_connect_upstream with the root store from build_upstream_root_store. The private CA is in neither webpki nor the image bundle, so rustls fails with UnknownIssuer, the tunnel is torn down, and curl reports a TLS or connection-reset error after a successful CONNECT.

Workarounds today: bake the CA into the image's system bundle (fails in Kubernetes sidecar topology), set tls: skip on the endpoint (loses inspection, still needs the CA in the image because SSL_CERT_FILE is overridden), or on Podman/VM misuse proxy_ca_bundle (requires an https_proxy URL).

In Block/Allow modes there is no termination and no TLS env, so curl uses the image bundle or a user-supplied SSL_CERT_FILE unchanged.

What Would Need to Change

• Network supervisor (tls.rs, run.rs): accept a second PEM source; add its anchors after the built-in or native roots in both feature variants (the native branch currently ignores system_ca_bundle, so the feature would silently no-op there otherwise); include it in both child files; decide whether to write child files in non-proxy modes; abort startup on a present-but-invalid file rather than degrading like the current CA-write failure path.
• Sandbox binary: one new argv-only flag mirroring the upstream proxy flags, threaded through run_sandbox to run_networking.
• Process supervisor: only if precedence changes for direct mode, where OpenShell currently overrides user TLS env unconditionally.
• Gateway: a config field (global [openshell.supervisor.network] or per-driver keys), startup validation (bounded read, only certificate blocks, at least one usable anchor), and a DriverStartupContext field so factories can hand drivers a gateway-owned staged path, following guest_tls. Fail closed when the selected driver cannot propagate the material (remote endpoints, out-of-tree factories).
• Docker/Podman: one read-only bind mount to a reserved guest path plus one argv pair, copying the proxy CA mount code.
• VM: one overlay file at a VM_GUEST_* constant plus an args-file line; remove the file when unset.
• Kubernetes: the hard part. Either a gateway-managed ConfigMap (needs create RBAC that cannot be name-restricted, 1 MiB data limit, ownership checks, forced server-side apply) or an operator-named ConfigMap mounted directly (no new RBAC, mirrors oidc.caConfigMapName). Sidecar topology must mount into the container that runs the supervisor.
• Core: a reserved container path constant under TLS_ROOT or ETC_ROOT so it stays inside CONTROL_ROOTS and the /etc Landlock baseline.

Alternative Approaches Considered

Patterns to Follow

• Operator inputs arrive as argv, never env (main.rs:203-207, upstream_proxy.rs:6-13).
• Shared host and guest validation via driver_utils so acceptance never diverges.
• Fail closed on present-but-invalid values; absent means unset (upstream_proxy.rs:410-423).
• deny_unknown_fields on every new TOML struct; per-driver tables stay raw toml::Value.
• Reserved container paths in container_paths.rs; stage a gateway-owned copy under the state dir rather than bind-mounting the operator's source file (Docker's :z relabels the host file).
• OCSF ConfigStateChangeBuilder on load success and failure.
• Redacted Debug for driver configs carrying material.

Proposed Approach

Treat destination trust as an operator-owned, gateway-validated input that is separate from both gateway mTLS and the corporate proxy CA, adopting the mechanism #3996 established for Podman. The gateway reads and validates the single configured pre-assembled PEM bundle at startup, stages a gateway-owned artifact, and hands drivers a staged path rather than the operator's source file. Each driver delivers the artifact read-only to the fixed reserved guest path from #3996 and passes it to the supervisor by the existing --additional-ca-bundle argv. The supervisor folds the anchors into the default roots exactly as in #3996; freshness and tamper checks run at the gateway/driver boundary against the staged artifact's recorded generation, and stop/start reconciliation rebuilds stale runtime material (Docker/Podman container metadata, Kubernetes ConfigMap mounts, VM overlays) so certificate rotation does not require deleting sandboxes. A regression test must prove a certificate signed by the destination CA cannot authenticate the gateway. Kubernetes delivery and direct-mode env precedence are the two decisions that most affect scope. PR #3292 is a complete implementation of this approach and should be evaluated against the decisions below.

Scope Assessment

• Complexity: Medium (High if Kubernetes uses a gateway-managed ConfigMap)
• Confidence: High on the supervisor side, Medium on Kubernetes delivery
• Estimated files to change: 20-30 for all four drivers with docs, Helm, and e2e; 12-15 for a Docker/Podman/VM-first cut
• Issue type: feat

Risks & Open Questions

• Who owns destination trust: operator or sandbox creator? With bundled-ca-roots the image bundle is already overlaid and the proxy resolves hosts from the workload's /etc/hosts; whether that combination is an accepted part of the threat model should be settled before choosing option (b).
• Kubernetes delivery: gateway-managed ConfigMap (needs namespace-wide configmaps/create, and the ConfigMap becomes a namespace-writable trust root) versus operator-named ConfigMap (no new RBAC, simpler). The investigation leans toward the operator-named ConfigMap.
• Direct-mode child env: should TLS env vars be set when there is no proxy, and should user-supplied values win there? Today proxy mode overrides unconditionally. Setting REQUESTS_CA_BUNDLE to a combined file that lacks a system bundle (image with certs outside SYSTEM_CA_PATHS) would break public TLS for Python requests.
• Strictness: the proxy CA path accepts a bundle if one anchor parses, while the lenient loader silently drops bad blocks. Pick one rule for both bundles.
• Rotation: restart-only (consistent with middleware and proxy config) or file watch.
• Scope: could a first cut ship Docker/Podman/VM plus documented image-bundle guidance for Kubernetes?
• Non-bundled-ca-roots builds ignore the system bundle today; the new roots must be added there or the feature no-ops silently.
• Bundle size must respect both the Kubernetes 1 MiB ConfigMap limit and the existing 1 MiB read bound.
• TlsMode::Skip endpoints are unaffected by the root store; only the child env helps there.
• HA gateways sharing one gateway_id would race on a managed ConfigMap during rollout.

LSM Impact

Docker relabels bind mounts unconditionally with :z; Podman only when selinuxfs is mounted. z relabels the host file, so bind-mounting the operator's /etc/pki/... source would change its system label. Stage a gateway-owned copy under the state directory instead. Inside the sandbox the read happens before privilege drop and under the /etc Landlock baseline; no /proc/<pid> traversal is involved. VM overlay files need no relabel.

Documentation Impact

• docs/reference/gateway-config.mdx: new section, and clarify the distinction from proxy_ca_bundle.
• deploy/helm/openshell/README.md(.gotmpl) and values.yaml for the Kubernetes value.
• architecture/sandbox.md and architecture/compute-runtimes.md.
• skills/debug-openshell-cluster/SKILL.md (required by AGENTS.md for Helm and driver changes).
• rfc/0003-gateway-configuration if the config schema grows.

Disposition Readiness

• State: state:validated
• Assessment: The failure is reproducible by reading the root-store construction, the workarounds and their gaps are confirmed in code, the affected components and integration points are mapped, and a working candidate implementation (PR feat(network): support additional destination CAs #3292, manually verified against a private-PKI GitLab on rootless Podman) exists. A maintainer has enough to accept or decline and to choose among the design decisions above.
• Missing evidence: None for disposition. Kubernetes and VM e2e results for the candidate PR are pending CI.

Test Considerations

• Unit: root store gains anchors in both feature variants; write_ca_files ordering; config parse and deny_unknown_fields; strictness cases (empty, private-key block, oversized, non-regular file, mismatched labels); per-driver argv and mount tests like the Podman proxy CA tests; Kubernetes pod-spec rendering for both topologies; gateway-mTLS negative test in grpc_client.rs.
• Helm unittest: new suite under deploy/helm/openshell/tests/ for values, RBAC, and volume rendering.
• E2E per driver, modeled on podman_corporate_proxy.rs and vm_corporate_proxy.rs: private-CA HTTPS upstream, curl succeeds with the setting and fails without, public host still works, hostname mismatch still rejected, invalid staged material fails closed, removal plus restart drops trust.
• Existing test infrastructure: the corporate proxy e2e harness already stands up a TLS endpoint and stages operator material per driver; a shared additional-ca suite can reuse that pattern.

Created by spike investigation. state:validated means the issue is ready for human disposition; state:needs-info means specific evidence is still required. A human applies state:accepted or places the issue on the roadmap if OpenShell should pursue the work. To queue unattended agent planning, a human applies agent:plan-requested; on a direct request, the agent warns about missing expected workflow labels and continues without changing them. Candidate implementation: #3292.