feat(observability): supervisor OTLP telemetry relay - #3196
Open
rhuss wants to merge 9 commits into
Open
Conversation
rhuss
requested review from
a team,
derekwaynecarr,
mrunalp and
sjenning
as code owners
September 5, 2026 04:26
rhuss
marked this pull request as draft
September 5, 2026 05:44
Add an OTLP relay to the sandbox supervisor that accepts agent-emitted traces on 127.0.0.1:4318, enriches spans with sandbox resource attributes, and forwards them to the gateway over the existing session protocol. The relay enables zero-config observability for OTel-instrumented agents inside network-isolated sandboxes. Agents export to the standard OTLP endpoint; the supervisor handles enrichment, buffering, and forwarding without requiring egress policy exceptions. Key components: - OTLP HTTP receiver (protobuf + JSON) with per-driver binding (netns for Docker/Podman, direct for K8s/VM) - Span enrichment with sandbox identity (sandbox_id, workspace_id, policy, user, image, driver) and telemetry source marker - Bounded buffer (4096 slots) with non-blocking try_send forwarding - Gateway-side TelemetryRelayExporter with dedicated gRPC client that preserves supervisor-enriched resource attributes - Capability negotiation: supervisor advertises telemetry_relay, gateway confirms only when OTLP export is configured - Graceful shutdown ordering: relay drains before network teardown Proto: TelemetryData message, capabilities on SupervisorHello and SessionAccepted. Closes: NVIDIA#2641 Signed-off-by: Roland Huß <rhuss@redhat.com>
Fix Critical and Important findings from multi-agent code review: - Fix shutdown deadlock: drop telemetry sender before awaiting forwarder to allow the mpsc channel to close - Limit OTLP request body to 4 MiB (http_body_util::Limited), return 413 Payload Too Large when exceeded - Replace expect() with ConnectError on malformed OTLP endpoint URI to prevent gateway panic on startup - Add 10s timeout on telemetry export gRPC calls to prevent unbounded task accumulation during collector brownout - Validate OCSF events as JSON before logging and emit as structured field to prevent log injection via embedded newlines - Cap concurrent OTLP receiver connections at 64 via semaphore - Strip existing trusted keys before enrichment to prevent agent-supplied attribute spoofing - Fix rate limiter TOCTOU race with fetch_update CAS loop - Log session drop counter during shutdown for observability - Log on supervisor session try_send failure instead of silent discard Signed-off-by: Roland Huß <rhuss@redhat.com>
Clarify in architecture/sandbox.md that the OTLP receiver only binds port 4318 when the relay is active (gateway has OTLP configured and confirms the telemetry_relay capability). Document topology constraints: all current topologies keep the process supervisor co-located with the agent, so 127.0.0.1 is correct. Note that future topologies moving the process supervisor out of the workload pod would need a different address. Remove the netns_fd gate on OTEL env var injection in process.rs since all topologies keep the supervisor co-located and the env vars are harmless when the relay is not running (OTel SDKs handle unreachable endpoints gracefully). Signed-off-by: Roland Huß <rhuss@redhat.com>
The cherry-pick conflict resolution for podman container.rs used --theirs which in a cherry-pick context takes the spike's version, not main's. This accidentally included unrelated spike changes (TLS secret prefix removals, SPIFFE mount changes). Revert to upstream/main's version since our PR has no podman changes. Also revert mise.lock changes that were auto-generated when the pre-commit hook installed tools. Signed-off-by: Roland Huß <rhuss@redhat.com>
rhuss
force-pushed
the
feat/supervisor-otlp-relay
branch
from
September 5, 2026 06:51
2038d0d to
e6cada9
Compare
Replace hardcoded "127.0.0.1:4318" with constants in sandbox_env.rs (OTLP_RECEIVER_ADDR and OTLP_RECEIVER_ENDPOINT) so the bind address and agent env var are defined in one place. Future topologies that move the process supervisor out of the agent's network namespace can update these constants or derive the address from the topology. Signed-off-by: Roland Huß <rhuss@redhat.com>
…, and forwarder Add 13 new tests bringing the relay test suite from 4 to 17 tests: Buffer (4 new): - depth_tracks_send_and_recv: queue depth accuracy across operations - drop_count_increments_on_each_overflow: counter accuracy under sustained overflow - metrics_shared_across_clones: shared metrics between sender clones - recv_returns_none_when_all_senders_dropped: channel close behavior Enrichment (4 new): - enrichment_strips_agent_supplied_trusted_keys: dedup prevents spoofing - enrichment_preserves_non_trusted_agent_attributes: custom attrs kept - enrichment_handles_json_content_type: JSON input, protobuf output - enrichment_rejects_invalid_protobuf: error handling for bad input Rate limiter (3 new): - rate_limiter_acquires_initial_tokens: exact token count - rate_limiter_drops_when_exhausted: OcsfRelaySink drop counting - rate_limiter_refills_after_time: time-based token refill Forwarder (2 new): - forwarder_constructs_telemetry_data_messages: correct message construction - forwarder_increments_session_drop_counter: session channel backpressure Signed-off-by: Roland Huß <rhuss@redhat.com>
The OTLP relay was silently disabled when the gateway config lacked an [openshell.gateway.otlp] section, with no log output at any decision point. This made it impossible to diagnose why telemetry data was not flowing from supervisor to gateway. Add info-level logging when the relay exporter connects and when capabilities are confirmed, and debug-level logging for the normal inactive paths. Add a commented-out OTLP section to the Docker gateway config as a setup reference. Fix pre-existing rustfmt and clippy issues in OTLP relay code. Signed-off-by: Roland Huß <rhuss@redhat.com>
rhuss
marked this pull request as ready for review
September 5, 2026 13:38
Contributor
Author
|
@krishicks @drew this is a PR for introducing an OTEL relay to the supervisor, forwarding OTLP traces to a collector via the gateway. This is the hardened OTEL-relay spike that I did for creating this report in #2641 (comment) It would be great if you could put your review agent army on this PR (I did some self-review, but we all know the more reviews the merrier :) See also https://github.com/rhuss/OpenShell/blob/48e9ece979889aa49b1df146c3d0d08e7118496a/architecture/sandbox.md#telemetry-relay for a recap of the overall architecture. |
2 tasks
Rename the proto message and all related types to clarify that this carries OpenTelemetry signal data destined for an OTLP collector, not product analytics telemetry. Add a `oneof signal` wrapper around `trace_data` so future metrics and logs signals can be added without breaking the wire format. Renames: - Proto: TelemetryData -> OtelExportData, oneof payload field telemetry -> otel_export - Capability: "telemetry_relay" -> "otel_export" - Gateway: TelemetryRelayExporter -> OtelRelayExporter, telemetry_relay.rs -> otel_relay.rs - Supervisor: TelemetryRelay -> OtelRelay - All related variables, functions, and log messages Signed-off-by: Roland Huß <rhuss@redhat.com>
2000krysztof
suggested changes
Sep 9, 2026
Comment on lines
+958
to
+1014
| // OTEL relay: bind OTLP receiver for all Linux topologies. | ||
| // All current topologies keep the process supervisor co-located with | ||
| // the agent, so 127.0.0.1 is reachable from agent processes. Future | ||
| // topologies that move the supervisor out of the workload pod would | ||
| // need to derive the bind address from the topology (e.g., pod IP | ||
| // via downward API) and update OTEL_EXPORTER_OTLP_ENDPOINT to match. | ||
| let otel_rx = { | ||
| #[cfg(target_os = "linux")] | ||
| { | ||
| let otlp_addr = openshell_core::sandbox_env::OTLP_RECEIVER_ADDR; | ||
|
|
||
| let (otel_session_tx, otel_session_rx) = | ||
| tokio::sync::mpsc::channel::<openshell_core::proto::SupervisorMessage>(64); | ||
|
|
||
| let relay_config = openshell_supervisor_network::otlp::RelayConfig::default(); | ||
| let metadata = openshell_supervisor_network::otlp::SandboxMetadata { | ||
| sandbox_id: sandbox_id.clone().unwrap_or_default(), | ||
| workspace_id: workspace_rx.borrow().clone(), | ||
| policy: sandbox_name_for_agg.clone().unwrap_or_default(), | ||
| user: resolved_process_identity | ||
| .uid() | ||
| .map_or_else(String::new, |uid| uid.to_string()), | ||
| image: std::env::var("OPENSHELL_CONTAINER_IMAGE").unwrap_or_default(), | ||
| driver: std::env::var(openshell_core::sandbox_env::SUPERVISOR_TOPOLOGY) | ||
| .unwrap_or_else(|_| "container".to_string()), | ||
| }; | ||
| let relay = openshell_supervisor_network::otlp::OtelRelay::new( | ||
| relay_config, | ||
| metadata, | ||
| otel_session_tx, | ||
| ); | ||
|
|
||
| let bind_addr: std::net::SocketAddr = otlp_addr.parse().unwrap(); | ||
|
|
||
| if let Some(ns) = netns.as_ref() { | ||
| match ns.bind_tcp_in_netns(otlp_addr).await { | ||
| Ok(listener) => { | ||
| let handle = relay.start_with_listener(listener); | ||
| tracing::info!(bind = %bind_addr, "OTEL relay started (netns)"); | ||
| otel_relay_handle = Some(handle); | ||
| } | ||
| Err(e) => { | ||
| tracing::warn!(error = %e, "OTEL relay failed to bind in netns; continuing without relay"); | ||
| } | ||
| } | ||
| } else { | ||
| match relay.start(bind_addr).await { | ||
| Ok(handle) => { | ||
| tracing::info!(bind = %bind_addr, "OTEL relay started"); | ||
| otel_relay_handle = Some(handle); | ||
| } | ||
| Err(e) => { | ||
| tracing::warn!(error = %e, "OTEL relay failed to start; continuing without relay"); | ||
| } | ||
| } | ||
| } | ||
| Some(otel_session_rx) |
Contributor
There was a problem hiding this comment.
The OTLP listener appears to bind before the gateway confirms the otel_export capability, and it remains active when the gateway declines it. This reserves the standard 127.0.0.1:4318 port even when OTLP export is not configured. The receiver can also return 200 OK for telemetry that will not be forwarded because the session does not drain otel_rx unless the capability is confirmed. Could listener startup be gated on the negotiated capability, or could the listener be shut down when the capability is declined?
2000krysztof
suggested changes
Sep 9, 2026
Comment on lines
+85
to
+92
| tokio::spawn(async move { | ||
| let svc = service_fn(move |req| { | ||
| let buf_tx = buf_tx.clone(); | ||
| let metadata = metadata.clone(); | ||
| async move { | ||
| handle_request(req, &buf_tx, &metadata, enrichment_enabled).await | ||
| } | ||
| }); |
Contributor
There was a problem hiding this comment.
I think an accepted keep-alive connection can block shutdown. Each detached serve_connection task retains a TelemetrySender, but receiver_handle tracks only the accept loop. RelayHandle::shutdown() drops only its own sender and then waits for the forwarder. The forwarder cannot observe channel closure until every connection task exits, so this wait is unbounded. Should connection tasks receive shutdown cancellation or be tracked and explicitly drained or aborted?
P.S sorry for the two reviews I selected the wrong lines earlier.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
127.0.0.1:4318, enriches spans with sandbox resource attributes, and forwards them to the gateway over the existing session protocolRelated Issue
Closes #2641
Changes
Proto (
proto/openshell.proto)TelemetryDatamessage withsandbox_id,trace_data(serialized OTLP), andocsf_eventscapabilitiesfield onSupervisorHelloandSessionAcceptedfor feature negotiationTelemetryvariant onSupervisorMessageSupervisor OTLP Module (
crates/openshell-supervisor-network/src/otlp/)POST /v1/traces(protobuf + JSON), TCP_NODELAY, dual spawn variants for netns vs direct bindopenshell.sandbox.id,.workspace_id,.policy,.user,.image,.driver) andopenshell.telemetry.source: "agent"routing markerTelemetryRelayorchestrator,RelayHandlefor lifecycle,RateLimitedOcsfSinkfor OCSF events, forwarder with non-blockingtry_sendSupervisor Process Integration
child_env.rs: OTEL env var injection helperprocess.rs/ssh.rs: SetOTEL_EXPORTER_OTLP_ENDPOINTandOTEL_EXPORTER_OTLP_PROTOCOLfor agent processessupervisor_session.rs: Advertisetelemetry_relaycapability, gate forwarding on gateway confirmation, log capability negotiation outcomeSandbox Lifecycle (
crates/openshell-sandbox/src/lib.rs)Gateway (
crates/openshell-server/)telemetry_relay.rs: DedicatedTelemetryRelayExporterwith separate gRPC client to OTLP collector (preserves supervisor-enriched resource attributes), info-level logging on connectsupervisor_session.rs: Capability negotiation with logging (confirmtelemetry_relayonly when OTLP is configured, log why when not confirmed)Configuration
deploy/docker/gateway.toml: Commented-out[openshell.gateway.otlp]section as setup referenceSupporting
openshell-otel/propagation.rs:HeaderMapInjectorandinject_traceparent_if_missing()for W3C context propagationopenshell-ocsf:OcsfRelaySinktrait andOcsfRelayLayer(implemented but not wired into subscriber, follow-up)architecture/sandbox.md: Telemetry relay section with data flow diagramTesting
cargo checkpasses for all relay cratesChecklist
architecture/sandbox.md)