Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,19 @@ mise run helm:k3s:create
Creates a k3d cluster and merges its kubeconfig into the worktree-local `kubeconfig` file.
When the named cluster already exists, the task starts any stopped containers and refreshes
same-named kubeconfig entries so a recreated load balancer's current API port takes effect.
Also applies the upstream agent-sandbox CRDs/controller (pinned via `AGENT_SANDBOX_VERSION`
Also applies the upstream agent-sandbox CRDs/controller and, for v0.5 and later,
the `SandboxClaim`, `SandboxTemplate`, and `SandboxWarmPool` extension APIs
(pinned via `AGENT_SANDBOX_VERSION`
in `tasks/scripts/helm-k3s-local.sh`, fetched from `github.com/kubernetes-sigs/agent-sandbox`
releases), enables its OTLP tracing on v0.5 and later, installs an OTLP trace
collector and UI in the `observability` namespace,
and preloads the default community sandbox image into k3d so the first sandbox create
does not wait on a large registry pull. Traefik is disabled at cluster creation time.

For a v0.4.x Agent Sandbox release, deploy OpenShell with
`server.warmPooling.enabled=false`. Verify a warm-capable installation with
`kubectl api-resources --api-group=extensions.agents.x-k8s.io`.

**Multi-worktree support:** the cluster name is derived from the last component of the
current git branch (e.g. branch `kube-support/local-dev/tmutch` → cluster
`openshell-dev-tmutch`). Each worktree therefore gets its own isolated cluster and its
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/branch-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -371,14 +371,17 @@ jobs:
agent_sandbox_version: v0.5.0
topology: combined
extra_helm_values: ""
e2e_task: e2e:kubernetes:warm-pool
- agent_sandbox_api: v1alpha1
agent_sandbox_version: v0.4.6
topology: combined
extra_helm_values: ""
e2e_task: e2e:kubernetes:v1alpha1
- agent_sandbox_api: v1beta1
agent_sandbox_version: v0.5.0
topology: sidecar
extra_helm_values: deploy/helm/openshell/ci/values-sidecar.yaml
e2e_task: e2e:kubernetes:warm-pool-sidecar
permissions:
actions: read
contents: read
Expand All @@ -389,6 +392,7 @@ jobs:
job-name: Kubernetes E2E (Rust smoke, ${{ matrix.topology }}, Agent Sandbox ${{ matrix.agent_sandbox_api }})
agent-sandbox-version: ${{ matrix.agent_sandbox_version }}
extra-helm-values: ${{ matrix.extra_helm_values }}
e2e-task: ${{ matrix.e2e_task }}
conformance-artifact-prefix: openshell-conformance

kubernetes-workspace-managed-e2e:
Expand Down
2 changes: 2 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

175 changes: 146 additions & 29 deletions architecture/compute-runtimes.md

Large diffs are not rendered by default.

57 changes: 44 additions & 13 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -219,8 +219,9 @@ token is reported as connected but unauthenticated.
Sandbox supervisor RPCs authenticate with explicit sandbox credentials; mTLS
does not grant sandbox identity. Kubernetes deployments use the
gateway-minted JWT bootstrap path: the supervisor starts with a projected
ServiceAccount token, exchanges it for a gateway-minted sandbox JWT, and uses
that JWT on subsequent gateway RPCs.
ServiceAccount token, registers the pod with the gateway, receives a
gateway-minted sandbox JWT after activation, and uses that JWT on subsequent
gateway RPCs.
User-facing RPCs are authorized by descriptor-declared role and scope policy
when OIDC or edge identity is enabled. The OIDC admin role grants platform-wide
access and bypasses workspace membership checks. Workspace Admin and Workspace
Expand All @@ -234,22 +235,43 @@ identity inspection without client-side token decoding.
Sandbox secrets are gateway-signed JWTs bound to a single sandbox ID. Docker,
Podman, and VM drivers deliver the initial token through supervisor-only
runtime material; Kubernetes supervisors exchange a projected ServiceAccount
token through `IssueSandboxToken`. The gateway delegates that opaque credential
to the selected compute driver's `AuthenticateSandbox` RPC. A capable driver is
trusted to return the authenticated sandbox ID, while the gateway still requires
a matching durable sandbox record before minting a JWT. The Kubernetes driver
uses its own named configuration to run TokenReview and verify the live pod and
controlling Sandbox CR. The bootstrap path accepts
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
the sandbox record still exists. Older tokens are not server-revoked; shared
deployments bound replay exposure with short `gateway_jwt.ttl_secs` lifetimes.
token through `RegisterSupervisor`. The gateway delegates that opaque
credential to the selected compute driver's `AuthenticateSandbox` RPC. The
driver verifies the credential and returns either a bound sandbox ID or a
warm-pending supervisor instance with an opaque activation guard. For
already-bound pods, the gateway still requires a matching durable sandbox record
before sending an activation message with the gateway JWT. For warm-pooled
Kubernetes sandboxes, the same check is re-anchored through the claim that
adopted the warm pod: the live pod must still be controlled by a live
`Sandbox`, that `Sandbox` must be associated with the live `SandboxClaim`, and
the sandbox-id metadata on that claim must identify the same OpenShell sandbox
record before activation can complete. Activation must present the same opaque
guard returned during registration. Both paths depend on the same Kubernetes
RBAC boundary: sandbox workloads and untrusted users must not be able to create
or mutate trusted Agent Sandbox objects, pod metadata, or the configured
sandbox ServiceAccount in the gateway-managed namespace. `IssueSandboxToken`
remains as a compatibility shim for older supervisor images and mints through
the same driver authentication path, but only for already-bound responses.
Supervisors renew gateway JWTs in
memory before expiry only while the sandbox record still exists. Older tokens
are not server-revoked; shared deployments bound replay exposure with short
`gateway_jwt.ttl_secs` lifetimes.
The config default is
`gateway_jwt.ttl_secs = 0` for local single-player Docker, Podman, and VM
gateways; those tokens carry `exp = 0` and do not expire. Kubernetes and other
shared deployments should set a positive TTL.

Warm supervisors begin capturing logs before activation but do not open the log
stream until registration supplies the authoritative sandbox ID and installs
the gateway JWT. A bounded pre-activation queue is rebound to that identity,
which avoids both missing startup records and a second bootstrap race. In
sidecar topology, the registering network supervisor also sends the activated
sandbox ID and name together with the main-process command, TTY, and attachment
state to the process supervisor over the authenticated local control socket.
The process supervisor applies that identity before initializing its OCSF
context and rejects conflicts with any identity already present in its
environment.

Gateway JWT signing-key rotation is currently an offline operator action. The
runtime loads one active signing key and one matching public verification key
from the configured secret at startup. To rotate that key material today,
Expand Down Expand Up @@ -373,6 +395,15 @@ created from a template resolves that resource once and persists an ordinary
name, labels, annotations, provider attachments, and policy. The sandbox stores
template provenance as the template name and resource version used for the
snapshot, so later template edits or deletes do not mutate existing sandboxes.
For drivers that opt into template reconciliation, a background worker reads
the complete template set and sends it as authoritative desired state through
the optional `SandboxTemplateReconciler` driver service. The required
`ComputeDriver` service remains limited to common sandbox lifecycle operations.
The worker runs at startup, after template mutations, and periodically, so a
failed driver call or missed notification is retried from the source of truth.
The driver applies the supplied set idempotently and prunes backend resources it
owns for absent templates. Template create and delete therefore need no hidden
delivery rows or tombstones.

OAuth refresh failures retain a gateway-owned recovery classification alongside
the refresh state. The gateway reads only a bounded error response and maps
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/ensure_providers_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -636,6 +636,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::SupervisorActivationMessage, Status>,
>;

async fn register_supervisor(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/mtls_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -499,6 +499,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::SupervisorActivationMessage, Status>,
>;

async fn register_supervisor(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/provider_commands_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1066,6 +1066,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::SupervisorActivationMessage, Status>,
>;

async fn register_supervisor(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -895,6 +895,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::SupervisorActivationMessage, Status>,
>;

async fn register_supervisor(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
11 changes: 11 additions & 0 deletions crates/openshell-cli/tests/sandbox_name_fallback_integration.rs
Original file line number Diff line number Diff line change
Expand Up @@ -587,6 +587,17 @@ impl OpenShell for TestOpenShell {
Err(Status::unimplemented("not implemented in test"))
}

type RegisterSupervisorStream = tokio_stream::wrappers::ReceiverStream<
Result<openshell_core::proto::SupervisorActivationMessage, Status>,
>;

async fn register_supervisor(
&self,
_request: tonic::Request<openshell_core::proto::RegisterSupervisorRequest>,
) -> Result<Response<Self::RegisterSupervisorStream>, Status> {
Err(Status::unimplemented("not implemented in test"))
}

async fn refresh_sandbox_token(
&self,
_request: tonic::Request<openshell_core::proto::RefreshSandboxTokenRequest>,
Expand Down
42 changes: 41 additions & 1 deletion crates/openshell-core/src/driver_utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

use std::path::{Path, PathBuf};

use crate::proto::compute::v1::DriverSandbox;
use crate::proto::compute::v1::{DriverSandbox, GetCapabilitiesResponse};

// ---------------------------------------------------------------------------
// Sandbox container/pod label keys (openshell.ai/ namespace)
Expand Down Expand Up @@ -698,6 +698,46 @@ pub fn sandbox_token_path(
Ok(path.join(sandbox_id).join("sandbox.jwt"))
}

/// Build a [`GetCapabilitiesResponse`] from the common driver capability fields.
///
/// Every compute driver constructs this response with the same fields. Shared
/// here to avoid repeating the struct literal in each driver crate.
pub fn build_capabilities_response(
driver_name: &str,
driver_version: impl Into<String>,
default_image: impl Into<String>,
) -> GetCapabilitiesResponse {
build_capabilities_response_with_template_reconciliation(
driver_name,
driver_version,
default_image,
false,
)
}

/// Build a [`GetCapabilitiesResponse`] and configure authoritative
/// sandbox-template reconciliation.
pub fn build_capabilities_response_with_template_reconciliation(
driver_name: &str,
driver_version: impl Into<String>,
default_image: impl Into<String>,
supports_sandbox_template_reconciliation: bool,
) -> GetCapabilitiesResponse {
GetCapabilitiesResponse {
driver_name: driver_name.to_string(),
driver_version: driver_version.into(),
default_image: default_image.into(),
gateway_manages_lifecycle: false,
supports_sandbox_authentication: false,
driver_reports_runtime_readiness: false,
supports_sandbox_template_reconciliation,
supports_warm_supervisor_bootstrap: false,
resource_capabilities: None,
rootfs_tar_staging_dir: String::new(),
rootfs_tar_max_bytes: 0,
}
}

/// Return the effective log level for a sandbox.
///
/// Uses the level from the sandbox spec when non-empty, falling back to
Expand Down
Loading
Loading