feat(ha): harden gateway peer transport and expand failover conformance #3529

User Story

As an OpenShell cluster operator, I want gateway peer routing to require an explicitly trusted encrypted transport and to be validated across supported sandbox operations, so that HA does not weaken confidentiality or fail only on less-common workflows.

Problem Statement

PR #1868 authenticates gateway peers with Kubernetes ServiceAccount identity, but the current transport and validation contract has gaps:

• HTTPS peer clients fall back to the platform trust store when no peer CA is configured, widening the trust surface for internal peer traffic.
• A multi-replica gateway that itself serves plaintext accepts an http:// peer endpoint with only a warning, even though peer relays carry supervisor traffic and projected peer credentials.
• The HA e2e keeps a sandbox alive through scaling and pod rotation and covers create/watch, exec, upload, and download, but does not exercise the full supported set of operations that depend on owner routing, such as forwarding/service traffic and configuration changes.

Impact / Why This Matters

A deployment mistake can leave internal peer traffic unencrypted or trusted through roots unrelated to the cluster peer identity. Separately, operators can pass the current HA smoke test while encountering failures in untested session-bound or configuration workflows after scaling or rolling gateway pods. The workaround is manual cluster-specific testing and inspection, which is inconsistent and easy to omit.

Proposed Design

Define and enforce a fail-closed peer transport contract for multi-replica gateways:

• Peer routing requires TLS with an explicit peer CA or an equivalently scoped cluster trust source; it does not silently fall back to general platform roots.
• Multi-replica startup rejects plaintext peer endpoints.
• Documentation identifies the operations expected to remain available through a non-owner gateway and during gateway scaling, failover, and rolling replacement.
• HA conformance keeps a long-running sandbox active while exercising representative operations from that documented inventory through different gateway replicas.

Acceptance Criteria

• Multi-replica gateway startup rejects an http:// peer endpoint.
• HTTPS peer routing fails closed when no explicitly scoped peer trust source is configured.
• Helm-generated and custom TLS configurations document the required peer CA, server identity, and client identity contract.
• The supported HA operation inventory includes sandbox create/watch, exec, file transfer, TCP or service forwarding, and relevant policy/provider configuration workflows.
• Automated HA tests exercise that inventory against a long-running sandbox across scale-up, scale-down, owner loss, and rolling pod replacement.
• Negative tests cover plaintext endpoints, missing peer CA material, and peer identity mismatch.
• Architecture and gateway-configuration documentation match the enforced transport and conformance behavior.

Alternatives Considered

Continuing to warn for plaintext preserves development configurations but permits an insecure production mistake. Platform trust roots are convenient for public endpoints, but internal gateway peers already have a chart-managed identity and should use narrowly scoped trust. Testing every RPC independently would be expensive and brittle, so the conformance suite should test the documented operation classes and shared routing paths instead.

Agent Investigation

This groups the remaining peer-security and coverage review discussions from PR #1868:

• Native-root fallback for peer TLS: feat(kubernetes): support HA gateway rebalancing #1868 (comment)
• Plaintext peer endpoint acceptance: feat(kubernetes): support HA gateway rebalancing #1868 (comment)
• Long-running sandbox and operation coverage: feat(kubernetes): support HA gateway rebalancing #1868 (comment)

Related umbrella issue: #1021.