feat(gateway): separate collector endpoint for relayed agent traces #4648

User Story

I use OpenShell to run Claude Code and Python agents in sandboxes, and I send their traces through the supervisor OTLP relay (#4276) to an OpenTelemetry Collector with Tempo and Grafana. I need to send agent traces to a different collector than the gateway's and supervisors' infrastructure spans, so I can scale an agent collector pool on its own and re-export agent traces to a different sink.

Problem Statement

The relay forwards agent traces to the same collector as infrastructure traces. [openshell.gateway.otlp] already resolves separate endpoints for traces ([otlp.traces]) and OCSF events ([otlp.ocsf]), but agent traces have no table of their own. An operator who wants agent traces elsewhere has to move all traces or none.

Adding the agent table also brings up a naming question for the existing configuration. The proposal below organizes tables by signal (traces, later logs and metrics) with per-source sub-tables. [otlp.ocsf] already breaks that pattern: OCSF events are log records, yet the table sits at the top level. See Open Question: Where [otlp.ocsf] Belongs below. Maintainer input on it would help before the layout is extended further.

Impact / Why This Matters

Agent trace volume grows with the number of sandboxes and with how chatty each agent is. Infrastructure trace volume doesn't. Today both share one collector, so operators can't size them separately, apply different retention, or route agent traces to another backend such as an LGTM stack or a SIEM. The only way around it is to put routing rules into the shared collector itself, which pushes OpenShell-specific logic into operator infrastructure.

Proposed Design

Add an optional [openshell.gateway.otlp.traces.agent] table. It nests under [otlp.traces] because the agent lane is a trace signal, and collector backends are usually split by signal.

• endpoint is optional. The relay uses traces.agent.endpoint, falls back to traces.endpoint, and then to otlp.endpoint. Without the new table, behavior is the same as today.
• enabled defaults to true. With enabled = false, the relay stays off and infrastructure traces keep flowing.
• A configuration with only [otlp.traces.agent] and no base endpoint is valid. The relay runs, and infrastructure spans are not exported.
• Every compute driver delivers the resolved endpoint to the supervisor. The workload still exports to the reserved relay address and never sees a collector address.
• Helm deployments set the table through gatewayConfig, the same way as traces and ocsf.

Later agent lanes for logs and metrics would follow the same pattern: [otlp.logs.agent] and [otlp.metrics.agent].

Open Question: Where [otlp.ocsf] Belongs

The signal-first layout exposes an inconsistency in the existing configuration. OCSF events are exported as OTLP log records (#4500), but [otlp.ocsf] sits at the top level next to [otlp.traces], not under a logs table. Once agent logs are relayed, the full picture would look like this:

[openshell.gateway.otlp]            # default for every signal
[openshell.gateway.otlp.traces]     # infrastructure traces
[openshell.gateway.otlp.traces.agent]
[openshell.gateway.otlp.logs]       # future: default for log records
[openshell.gateway.otlp.logs.agent] # future: relayed agent logs
[openshell.gateway.otlp.ocsf]       # OCSF log records, outside the logs table

This issue leaves [otlp.ocsf] unchanged. When the logs lane lands, the options are:

• Keep [otlp.ocsf] as a separate audit stream and document why it sits outside the signal tables.
• Let [otlp.ocsf] fall back to [otlp.logs] before otlp.endpoint, so it behaves like a logs sub-table without moving.
• Add [otlp.logs.ocsf] as the canonical name and keep [otlp.ocsf] as an alias.

Feedback on which direction maintainers prefer is welcome now, so the new traces.agent table doesn't lock in a layout that has to change later.

Suggested UX (if applicable)

[openshell.gateway.otlp]
endpoint = "http://otel-collector.observability.svc:4317"

[openshell.gateway.otlp.traces]
endpoint = "http://tempo.observability.svc:4317"

# Relayed agent traces go to their own collector pool.
[openshell.gateway.otlp.traces.agent]
endpoint = "http://agent-collector.observability.svc:4317"

# Helm
gatewayConfig:
  openshell.gateway.otlp.traces.agent:
    endpoint: http://agent-collector.observability.svc:4317

# Infrastructure traces only, no agent relay.
[openshell.gateway.otlp.traces.agent]
enabled = false

Acceptance Criteria

• With [otlp.traces.agent] endpoint set, relayed agent spans reach that collector, and gateway and supervisor spans reach the [otlp.traces] collector.
• Without [otlp.traces.agent], agent spans go to the [otlp.traces] endpoint, or to [otlp.endpoint] when no traces table is set.
• [otlp.traces.agent] enabled = false leaves the relay inactive and logs the reason, while infrastructure trace export continues.
• A configuration with only [otlp.traces.agent] endpoint starts the gateway, and the relay runs.
• Unknown keys in the new table are rejected at startup, like in the other [otlp] tables.
• The behavior is the same on the Kubernetes, Docker, Podman, and VM compute drivers.
• docs/how-it-works/gateways/configuration.mdx documents the table, the fallback order, and the Helm example.

Alternatives Considered

• A source-first [otlp.agent] table covering everything the agent emits. One endpoint for agent traces, logs, and metrics only works if it points at a collector that fans out again, because backends are split by signal. The fallback to the traces endpoint would also be a special rule instead of following from the nesting.
• [otlp.traces.infra] next to [otlp.traces.agent]. [otlp.traces] already covers infrastructure traces once the agent lane has its own table, so every configuration works without it. It can be added later without breaking anything.
• Routing in the collector. Operators can split on openshell.telemetry.source=agent with a routing connector. That works, but it gives up the separate scaling this issue is about and pushes OpenShell-specific routing into every deployment.

Agent Investigation

• Endpoint resolution lives in crates/openshell-server/src/config_file.rs (RawOtlpConfig and OtlpConfig). Each enabled signal resolves its endpoint at parse time.
• feat(supervisor): export OCSF events as OTLP log records #4545 delivers the OCSF endpoint to supervisors through each driver. The agent endpoint can take the same path as a new supervisor variable read only by the relay in crates/openshell-supervisor/src/otlp_relay/mod.rs.
• Transport settings (TLS, headers, compression) are out of scope here and tracked separately under #3899.

Follow-up to #4275 and #4276, requested in this review comment.

Related: #2641, #4500, #4545