feat: add shared alpha extension authentication and custom CA transport #2623

Problem Statement

OpenShell needs an alpha authentication mechanism for extension services so middleware and gateway-interceptor integrations can begin implementing against a stable contract without waiting for production-grade mTLS, multi-key rotation, or per-runtime secret delivery. The mechanism should reuse the gateway's existing Ed25519 token minting and supervisor token-refresh APIs wherever possible, and remote middleware must support TLS verification with an operator-provided CA.

This spike narrows #2430 to an integration-enabling slice. The implementation must use one shared transport/auth client abstraction for both supervisor middleware and gateway interceptors rather than introducing two security-sensitive client implementations. Related roadmap: #1904.

Technical Context

Today both extension systems create tonic channels independently. Gateway interceptors accept HTTP, HTTPS, or Unix endpoints but do not configure TLS trust or attach caller credentials. Remote supervisor middleware accepts HTTP or HTTPS; HTTPS uses platform roots only, and the same unauthenticated registration is used by the gateway for Describe/ValidateConfig and by sandbox supervisors for Describe/EvaluateHttpRequest.

OpenShell already has most of the underlying mechanics: the gateway loads one Ed25519 signing key and kid, SandboxJwtIssuer creates short-lived tokens, RefreshSandboxToken renews a sandbox's in-memory bearer credential, and AuthInterceptor replaces bearer metadata in place. The missing work is a distinct extension audience/claim contract, authorization of requested extension credentials, per-service token slots, a shared extension channel builder, and custom-CA configuration and propagation.

Affected Components

Technical Investigation

Architecture Overview

The gateway constructs the middleware registry at startup from [[openshell.supervisor.middleware]], calls every remote service's Describe, and later calls ValidateConfig during policy admission. MiddlewareRegistry::required_services copies only policy-selected registrations into GetSandboxConfigResponse. Every sandbox supervisor receives those registrations over its already-authenticated gateway channel and constructs the same remote registry for request evaluation.

Gateway interceptors follow a separate startup path: ExecutionPlan::load connects each endpoint, calls Describe, then stores cloned tonic clients for the configured bindings and optional provider-profile source. The interceptor and middleware crates both depend on openshell-core, but neither has a neutral reusable extension transport layer.

The gateway currently loads the JWT issuer after it has already connected middleware and gateway interceptors. Authenticated startup calls therefore require moving signing-material construction earlier and passing an extension token source into both client initializers.

Stable connection data and rotating credentials must remain separate:

• Custom CA PEM is public, stable trust material. It can be loaded once by the gateway, included in SupervisorMiddlewareService, hashed into config_revision, and trigger a registry rebuild when it changes.
• Bearer JWTs are secret and short-lived. They must not be placed in SupervisorMiddlewareService: the gateway hashes the full protobuf message into config_revision, and supervisors compare complete service vectors to decide whether to rebuild and reconnect the registry.

Code References

Current Behavior

1. The gateway materializes middleware registrations directly from TOML and connects them before constructing ServerState.
2. RemoteMiddlewareService::connect enables native roots for HTTPS and otherwise opens an unauthenticated channel.
3. Interceptors call their own connect_endpoint, including a private Unix connector, and store unauthenticated generated clients.
4. The gateway later loads the sandbox JWT key pair and constructs a fixed-audience SandboxJwtIssuer.
5. Supervisors authenticate to the gateway with a process-global bearer slot and call GetSandboxConfig every ten seconds. Middleware registration changes rebuild the remote registry.
6. RefreshSandboxToken renews only the sandbox-to-gateway credential. It has no request fields and returns no extension credentials.

What Would Need to Change

Shared extension core

Create openshell-extension-core as the neutral foundation consumed by both extension systems. It should own resolved endpoint validation, HTTP/HTTPS/Unix channel construction, platform or custom-CA trust, normal hostname verification, shared extension identity and claim types, bearer metadata injection, expiry-aware per-service token slots, and secret-safe Debug/error behavior. It must not depend on middleware- or interceptor-specific generated clients and must preserve conditional Unix support.

The crate boundary follows one rule: it contains protocol-neutral primitives shared by at least two extension mechanisms; subsystem-specific orchestration stays with its owning crate. Keep operator-facing and wire configuration in existing ownership layers (openshell-core config types, server TOML, and protobufs) and keep signing authority in openshell-server. This gives the dependency direction gateway-interceptors|supervisor-middleware -> openshell-extension-core without a cycle. Cargo workspace membership is automatic through crates/*; Bazel needs the corresponding crate target and dependency metadata. Document this ownership rule in the new crate README to prevent core from becoming a general extension dumping ground.

Extension token contract and minting

Add a distinct extension claim set while reusing the existing gateway signing material and encoding mechanics. The alpha claim contract should include exact audience, issuer, subject, issued-at, expiry, kid, unique token ID, caller kind (gateway or supervisor), and sandbox ID only for supervisor calls. The existing sandbox JWT and its gateway audience must remain unchanged and must never be sent to an extension.

Each interceptor and middleware registration needs a stable operator-owned extension audience or a documented deterministic derivation. Gateway-originated clients mint the same claim form locally with gateway caller identity. A single-key JWKS representation of the already-loaded public key and kid gives integrations a verification mechanism without requiring multi-key rotation in the alpha slice.

Reuse RefreshSandboxToken for distribution

Extend RefreshSandboxTokenRequest additively with requested extension registration names, not arbitrary audiences. Extend the response with secret-marked service name, token, and expiry tuples while preserving the existing gateway token fields.

The handler must resolve every requested name against server-owned registration metadata and the effective policy for the authenticated sandbox before minting. Unknown, unselected, or duplicate requests fail rather than letting the caller choose an audience. IssueSandboxToken remains the Kubernetes bootstrap exchange because required middleware is not known until after the first configuration fetch.

Supervisors learn stable service names from GetSandboxConfig, acquire their credentials over the already-authenticated cached gateway client, then connect or reconcile the registry. Subsequent refresh updates per-service slots in memory without changing stable service equality or reconnecting channels. A transient refresh failure may retain the current credential until expiry; extension calls must fail locally after expiry.

Custom CA

Add the same custom-CA path field to middleware and interceptor operator configuration. The gateway reads and validates the file during startup. Interceptors pass the bytes directly to the shared connector; middleware registration carries the public PEM bytes through SupervisorMiddlewareService, so gateway validation and supervisor evaluation use the same trust material without driver-specific mounts.

Configured custom trust should preserve ordinary certificate and endpoint-hostname verification. Following the current SDK pattern, the proposed alpha semantics are that a configured CA replaces platform roots (private-CA pinning), while omission uses platform roots. A CA supplied for HTTP or Unix transport is invalid configuration.

Initialization and reconciliation

Construct the reusable signer/token source before middleware and interceptor startup. Remote clients receive bearer slots rather than raw token strings. Middleware reconciliation must match stable service identity separately from rotating credentials, update existing slots atomically, acquire a new credential before connecting an added service, and remove slots for detached services.

Alternative Approaches Considered

Use a transport-only openshell-extension-transport crate

This name precisely describes the immediate channel, TLS, and bearer work, but becomes artificially narrow once shared extension identity, claim types, registration primitives, or future verification helpers need a neutral owner. openshell-extension-core is selected so those cross-mechanism primitives have a durable home, guarded by the explicit shared-by-two-mechanisms ownership rule.

Put rotating tokens in GetSandboxConfig

This reuses the configuration poll but mixes short-lived secret state with a response that is also user-callable. Putting token bytes inside the registration would additionally change config_revision and force registry reconnection on rotation. Conditional out-of-band fields could be made safe, but the existing sandbox-only refresh RPC is a better semantic and authorization boundary.

Add a new extension-token RPC

A dedicated RPC would produce a clean contract, but RefreshSandboxToken already authenticates the supervisor, renews secret in-memory material, and is used by every compute-driver path. Additive request/response fields provide the alpha mechanism with less API and client duplication.

Use static bearer files

Static files are simpler for gateway callers but require separate secret mounts for Kubernetes, Docker, Podman, and VM supervisors and create a second integration contract that would later be replaced. The existing authenticated refresh API avoids those driver changes.

Reuse the sandbox-to-gateway JWT

Rejected. Its audience authorizes gateway access and it represents a different trust boundary. Giving it to extension services would expose a gateway credential and eliminate exact service audience separation.

Send custom-CA paths to supervisors

Rejected. Gateway host paths do not map consistently into four sandbox runtimes. CA certificates are public; delivering resolved PEM through stable configuration avoids new mounts, file labels, and driver-specific behavior.

Patterns to Follow

• Preserve the existing SandboxJwtIssuer Ed25519 algorithm pinning, kid, issuer identity, configured TTL, and secret-safe diagnostics.
• Follow AuthInterceptor's shared in-memory bearer replacement, but use independent slots per extension service rather than the process-global gateway slot.
• Follow RefreshSandboxToken's sandbox-principal and still-valid-token authorization boundary.
• Follow openshell-sdk::transport for operator-provided CA construction and normal hostname verification.
• Mark every new token protobuf field with (openshell.options.v1.secret) = true and extend the descriptor test that verifies dedicated secret fields.
• Preserve middleware's last-known-good registry behavior and existing on_error policy semantics.
• Use existing private-CA/mTLS test helpers under crates/openshell-server/tests/common/ and the remote tonic-server test pattern in openshell-supervisor-middleware.

Proposed Approach

Introduce openshell-extension-core as the shared foundation for middleware and interceptor clients, with transport as its first internal module and an explicit cross-mechanism ownership rule. Reuse the gateway's configured Ed25519 key and token encoder to mint a separate extension claim contract, then extend RefreshSandboxToken to return only the per-service credentials authorized by the caller sandbox's effective policy. Keep stable endpoint, audience, and public CA data in configuration, but keep rotating bearer credentials in per-service memory slots outside config revision and registry equality. Publish the current public key as a single-key JWKS for alpha integrations. Defer mTLS, multi-key overlap rotation, hot key reload, HA rotation coordination, and driver-specific secret delivery to later hardening.

Alpha Scope and Deferred Hardening

In scope

• One openshell-extension-core implementation of shared transport/auth primitives used by middleware and interceptors.
• Exact-audience gateway-minted bearer JWTs for gateway and supervisor callers.
• Supervisor distribution through additive RefreshSandboxToken fields.
• Single-key JWKS verification material.
• Platform roots or operator-provided custom CA with normal hostname validation.
• Gateway-to-interceptor, gateway-to-middleware, and supervisor-to-middleware calls.
• Explicit failure on missing, unauthorized, malformed, or expired credentials.
• Targeted interoperability, TLS, authorization, reconciliation, and secret-isolation tests.

Deferred

• mTLS client authentication.
• Multi-key overlap rotation and hot signing-key reload.
• HA issuer/key rollout semantics.
• Per-runtime secret mounts or certificate delivery.
• End-user delegation or durable agent identity.
• Runtime extension registration and health management.
• Unix socket ownership/TOCTOU hardening beyond preserving current support.

Scope Assessment

• Complexity: Medium
• Confidence: High — existing minting, refresh, polling, and TLS patterns provide a clear path
• Estimated files to change: 14-20 source, test, documentation, and build files
• Estimated agent-assisted effort: 5-8 focused engineering days
• Issue type: feat

Risks & Open Questions

• Confirm the exact audience syntax: an explicit stable URI is clearer for integrations; a deterministic subsystem/name derivation reduces configuration.
• Confirm that a configured custom CA replaces platform roots, matching the current SDK, rather than augmenting them.
• Decide whether the alpha token TTL reuses gateway_jwt.ttl_secs or adds a separately bounded extension-token TTL.
• Define whether one invalid requested service rejects the entire credential batch or returns valid entries plus per-entry errors; atomic failure is simpler and safer.
• Decide whether RefreshSandboxToken should mint a fresh gateway token when the request only needs extension credentials; preserving current behavior is simplest.
• A public JWKS route needs a stable path and must not expose private material. Multi-key semantics remain intentionally deferred.
• Gateway startup order must change so signing material exists before authenticated extension Describe; startup errors must not print tokens or PEM contents.
• openshell-extension-core needs an enforced ownership rule and README because a broadly named core crate can otherwise accumulate middleware- or interceptor-specific orchestration.

Disposition Readiness

• State: state:validated
• Assessment: The current code paths, reusable primitives, authorization boundary, shared-client ownership, CA propagation, and deferred hardening are sufficiently understood for a human accept/decline decision.
• Missing evidence: None. The listed audience, CA trust-set, and TTL choices are contract decisions for disposition or implementation planning, not missing feasibility evidence.

Test Considerations

• Shared TLS tests must run a real tonic server signed by a private CA and prove success with the configured CA, plus failure with an absent/wrong CA and hostname mismatch.
• Verify both extension systems use the shared channel path and attach bearer metadata to Describe, ValidateConfig, interceptor evaluation, middleware evaluation, and profile snapshot calls where applicable.
• JWT unit tests must cover gateway versus supervisor claims, exact audience, unique jti, expiry, kid, and rejection of a sandbox-to-gateway token for an extension audience.
• Refresh authorization tests must prove a sandbox can request only middleware selected by its effective policy and cannot supply an arbitrary audience.
• Proto/config tests must cover CA-path reading, malformed or unreadable files, PEM propagation, secret annotations, and backwards-compatible empty new fields.
• Sandbox tests must prove token replacement updates a slot without changing config_revision or rebuilding the middleware registry, and expired credentials fail locally into the existing middleware on_error handling.
• Secret-hygiene tests must prove tokens are absent from Debug, errors, logs, policy/settings/provider environments, sidecar control, files, and child process environments.
• Run the affected crate tests plus sandbox e2e because supervisor transport and reconciliation change. No new driver-specific e2e is required when CA PEM and tokens are distributed through existing authenticated gateway APIs.
• LSM impact: No direct SELinux/AppArmor-specific behavior is expected. The gateway performs an ordinary configured-file read, while PEM and tokens travel in memory over existing authenticated gRPC. No new /proc, process identity, executable, file-label, or cross-domain mount behavior is introduced.

Documentation Impact

• Update docs/reference/gateway-config.mdx for the new shared audience/custom-CA/insecure fields and changed extension transport behavior.
• Update docs/extensibility/supervisor-middleware.mdx and docs/extensibility/gateway-interceptors.mdx with the shared JWT verification contract and CA examples.
• Update RFC 0009's authentication/delivery discussion and RFC 0010's currently deferred auth section.
• Update the relevant existing architecture gateway/sandbox overview rather than creating a new top-level architecture file.
• Update Helm values/templates/docs only if first-class chart-managed CA mounting is included; inline CA propagation to supervisors itself requires no compute-driver or LSM documentation changes.

Created by spike investigation. state:validated means the issue is ready for human disposition; state:needs-info means specific evidence is still required. A human applies state:accepted if OpenShell should pursue the work and places it on the roadmap separately. To queue unattended agent planning, a human applies agent:plan-requested; a direct request to an agent does not require that label.