feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders

User Story

As a security or compliance reviewer investigating a policy violation, I want each OCSF event to carry the ID of the trace that produced it, so that I can jump from the event in my log aggregator straight to the supervisor's trace instead of matching sandbox IDs and timestamps by hand.

Problem Statement

Open Cybersecurity Schema Framework (OCSF) security events and OpenTelemetry (OTel) traces exist in separate systems with no connection between them. A security reviewer filters the OCSF log aggregator (Loki, Splunk) for DENY events in the last hour and finds a network deny for api.suspicious.com from sandbox sb-abc123. To see how the supervisor handled that connection (which policy matched, what L7 enforcement and middleware ran), they search Jaeger for the sandbox ID and hope the timestamps line up. Nothing links the security event to its trace.

Today only the gateway emits OTel spans. Once #3977 lands, the supervisor does too, and most OCSF events and OTel spans then come from the same code paths. The OCSF builders have no way to record the active trace context.

Impact / Why This Matters

Every deny investigation needs manual correlation across two systems. The workaround is searching the trace backend by sandbox ID within a time window. That breaks down quickly: with #3977 every egress connection starts its own trace, so a busy sandbox produces many traces per second and the time window rarely identifies one. SIEM pipelines also have no join key to automate the correlation.

Proposed Design

Record the active trace context in OCSF events using the OCSF 1.8 Trace profile. When a sampled OTel span is active at emission time, the event carries trace.uid and lists trace in metadata.profiles. Without one, the trace object is omitted. The reviewer pastes trace.uid into Jaeger or Tempo and lands on the supervisor trace that produced the event.

Representation

OCSF 1.8 has no top-level trace_id or span_id attributes. It defines a Trace profile whose trace object carries the W3C trace ID in trace.uid, plus an optional span object:

{"class_uid": 4001, "activity_id": 1, "metadata": {"profiles": ["security_control", "network_proxy", "container", "host", "trace"], ...}, "trace": {"uid": "0af7651916cd43dd8448eb211c80319c"}, ...}

The first version populates trace.uid only. The 1.8 span object requires start_time and end_time, which are unknown while the span is still open at emission time. Whether and how to fill trace.span is settled during implementation against the vendored-schema validation tests. The Trace profile schema files get vendored next to the existing ai_operation profile.

Layering

openshell-ocsf stays independent of OpenTelemetry:

openshell-ocsf adds a plain-data trace correlation type and a builder setter that renders the trace object and adds the profile.
openshell-otel provides a function that extracts the active OTel context and returns it only when the span context is valid and sampled. An unsampled trace ID points to nothing in the trace backend.
Binaries that emit OCSF events register this extractor with openshell-ocsf at startup. ocsf_emit! fills in the trace context when the builder has not set one, so individual call sites need no OTel dependency. An explicit setter call overrides the automatic value.

Egress deny spans

The supervisor egress spans from #3977 are DEBUG, and the supervisor's OTLP filter resolves to openshell=info at the default warn log level. Without a change, network denies get no trace ID at default settings. The proposal is an INFO-level span for deny decisions only. Denies are rare and are what reviewers investigate, while allowed connections stay at DEBUG to keep span volume down.

The decision is known only after authorization returns, and every deny OCSF event is emitted after the authorization scope has ended. So the deny span is opened in the deny branch and entered around the OCSF emission at every deny site: L4 policy denials, mapping and SSRF denials, and L7 HTTP denials in the relay and middleware. It carries the decision attributes (policy, reason, destination). A span that wraps only the policy evaluation would leave the emission hook without a trace context.

At the default log level the DEBUG connect span is disabled, so the deny span has no exported parent and becomes a one-span root trace. trace.uid still resolves, but the full connect, authorize, resolve and dial tree appears only at debug level, where the deny span nests under the connect span.

Out of scope: agent trace context

Each egress connection starts its own root trace, so the linked trace shows the supervisor's handling of the connection, not the agent's activity. Linking a deny to the agent's own trace, through the workload's traceparent header relayed via #3196, is a follow-up. The workload controls that header, so it goes in a separate, clearly labelled field and never in the trace object.

Scope

crates/openshell-ocsf/: trace correlation type, builder setter, trace serialization, metadata.profiles entry, vendored Trace profile schema, emission hook, and downgrade stripping of the trace object and Trace profile for OCSF 1.3 and 1.1 (the profile first appears in 1.4)
crates/openshell-otel/: sampled trace context extraction
crates/openshell-supervisor-network/: INFO-level deny span, entered around the OCSF emission at every deny site
Supervisor and gateway binaries: register the extractor at startup
No breaking changes (the trace object is optional)

Dependencies

feat(supervisor): export OTLP traces from sandbox supervisors #3977 (supervisor OTLP span export) must land first.

Acceptance Criteria

OCSF events emitted inside a sampled OTel span include trace.uid (32 lowercase hex characters) and list trace in metadata.profiles.
OCSF events emitted without an active sampled span omit the trace object and do not list the profile.
At the default supervisor log level, OCSF network and HTTP deny events carry a trace.uid that resolves to an exported trace containing the deny span.
Events downgraded to OCSF 1.3 or 1.1 contain neither the trace object nor trace in metadata.profiles, covered by regression tests next to the existing downgrade tests.
openshell-ocsf has no OpenTelemetry dependency.
The Trace profile schema is vendored, and the schema validation tests cover events with and without trace.
The published observability docs describe the trace field and how to follow it to the trace backend.

Alternatives Considered

Top-level trace_id / span_id fields: Easy to query, but outside the OCSF 1.8 schema. Consumers that validate against OCSF treat them as unknown attributes. The standard Trace profile covers the same need.

The unmapped mechanism: Schema-valid, but unmapped is meant for source data that has no OCSF attribute. Trace context has one, and downstream tooling won't look for it in unmapped.

Per-site .trace_context() calls: Explicit, but every emitting crate would need an OTel dependency, and adoption would drift across the call sites. The emission hook covers every event, and the explicit setter stays available where the current span is the wrong source.

Enrichment in the JSONL layer: Works, but puts the trace context into one output format instead of the event itself, so the shorthand format and any future sink would miss it.

Raising all egress spans to INFO: Gives every OCSF network event a trace ID, but exports one trace per outbound connection at default settings. Scoping the INFO span to denies covers the investigation case at a fraction of the volume.

Agent Investigation

OCSF builders live in crates/openshell-ocsf/src/builders/. Each builder declares its profiles through ctx.metadata(&[...]), and api_activity.rs already adds ai_operation conditionally.
The vendored schemas cover OCSF 1.8.0 but include only the ai_operation profile. The Trace profile is available from the OCSF schema server.
OCSF events are emitted via ocsf_emit!(), which stores the event in a thread-local and emits via tracing::info!(). The shorthand and JSONL layers extract it from there.
openshell-otel owns OTel context handling via tracing_opentelemetry::OpenTelemetrySpanExt. The supervisor crates have no OTel dependency on main.
ocsf_emit! call sites exist in openshell-sandbox, openshell-server, openshell-supervisor, openshell-supervisor-network, and openshell-supervisor-process.
feat(supervisor): export OTLP traces from sandbox supervisors #3977 (closes feat(observability): OpenTelemetry span emission from the sandbox supervisor #2508) adds supervisor span export. Its egress spans (supervisor.egress.connect, authorize, resolve, dial) are DEBUG, and the OTLP filter is info,openshell=<level> with a floor of INFO.

Related: #1055 (Enterprise Observability), #2508 (Supervisor OTel span emission), #3977 (supervisor OTLP span export), #3196 (OTLP relay), #2507 (Gateway OTel export surface)

I checked this against the current crate boundary and the vendored OCSF schema.

One wrinkle: the OCSF 1.8 base schema does not define top-level trace_id / span_id fields for these events. It does provide the schema-bounded unmapped escape hatch.

There is also a dependency-boundary question:

openshell-ocsf currently owns OCSF event construction and depends on tracing;
openshell-otel already owns OpenTelemetry context extraction through tracing_opentelemetry::OpenTelemetrySpanExt.

I would prefer not to make the OCSF crate itself responsible for discovering the active OpenTelemetry context.

A narrow design could be:

let the OCSF builders accept optional trace/span correlation values;
represent them through the schema-supported extension/unmapped path;
have the OTEL layer extract the active context and pass those values into the event builder.

That keeps OCSF serialization independent of the OpenTelemetry implementation.

If top-level trace_id / span_id fields are intended as an OpenShell-specific extension instead, I think that should be explicit in the contract because it is outside the vendored base schema.

Which representation do you prefer?

@kvnloo

Thanks for checking this against the schema, @kvnloo. You're right on both counts. The dependency point also exposes a bug in my original proposal: tracing::Span::current() only gives you process-local tracing IDs, not W3C trace and span IDs.

Representation. I'd go with neither top-level fields nor unmapped. OCSF 1.8 defines a Trace profile for this case, with a trace object that holds the W3C trace ID in trace.uid. We only vendor the ai_operation profile today, so it's easy to miss. An event would look like this:

{
  "metadata": { "profiles": ["security_control", "network_proxy", "container", "host", "trace"] },
  "trace": { "uid": "0af7651916cd43dd8448eb211c80319c" }
}

One catch: the 1.8 span object requires start_time and end_time, and we emit the OCSF event while the span is still open. So I'd ship trace.uid first, since that's what a security reviewer pastes into Tempo or Jaeger anyway, and settle trace.span in the PR against the vendored-schema validation tests.

Layering. Agreed, openshell-ocsf shouldn't know about OpenTelemetry. What I have in mind:

openshell-ocsf gets a plain-data correlation type and a builder setter that renders the trace object and adds the profile.
openshell-otel extracts the active context and returns it only when the span context is valid and sampled. An unsampled trace ID sends the reviewer to an empty search result, which is worse than no ID.
Binaries that emit OCSF events register that extractor with openshell-ocsf at startup, and ocsf_emit! fills in the context when the builder hasn't set one. The emitting crates don't each need an OTel dependency, and an explicit setter still wins where the current span is the wrong source.

Prerequisite. I also got something wrong in the issue: on main, the supervisor crates have no OpenTelemetry instrumentation yet. #3977 adds it, so this should build on that PR. Even with #3977 there are two gaps for the network-deny case:

The egress spans are DEBUG, and the supervisor's OTLP filter resolves to openshell=info at the default warn log level, so at default settings a deny would still carry no exported trace ID. I'd lean toward an INFO span around deny decisions only. Denies are rare and they're what a security reviewer investigates, while allowed connections stay at DEBUG to keep the span volume down.
Each egress connection starts its own root trace. That trace shows how the supervisor handled the connection, not what the agent was doing. The agent's side lives in the workload's own traceparent header (relayed via feat(observability): supervisor OTLP telemetry relay #3196). The workload controls that header, so it needs a separate, clearly labelled field and must not end up in the trace object. I'd handle that as a follow-up.

I'll update the issue body to reflect this. Are you planning to pick up the implementation? If so, building on top of #3977 makes sense.

I checked this against main at 5601d71b0 now that #3977 has landed, and ran a couple of focused tests around the OCSF/OTel path you described.

The correlation gap is still present as expected. With an active sampled OpenTelemetry span, current_trace_context_carrier() gives me a valid W3C traceparent, but an OCSF NetworkActivity emitted in that context still serializes without a trace object / trace.uid.

I also noticed two implementation wrinkles / edge cases that may be worth accounting for when this is implemented.

1. OCSF downgrade path

OpenShell emits OCSF 1.8 internally but supports downgrading JSON output to 1.3 and 1.1. The OCSF Trace profile is not present in those older schemas, while the current downgrade code only strips the newer fields/profiles it already knows about (ai_model, container, ai_operation, etc.).

I tried a 1.8 event containing:

{
  "metadata": {
    "profiles": ["security_control", "trace"]
  },
  "trace": {
    "uid": "0af7651916cd43dd8448eb211c80319c"
  }
}

and downgraded it to 1.3. The trace object survived the downgrade.

So it looks like adding the Trace profile here will also require the downgrade path to strip both the trace object and "trace" from metadata.profiles for 1.3/1.1, with a regression test alongside the existing downgrade tests.

2. Span lifetime around automatic enrichment

In the current egress path the connect span is made current while authorization runs, but some deny OCSF events are emitted afterwards.

I checked the same lifecycle in a focused test:

inside connect span = Some(traceparent ...)
at later OCSF emission = None

So if ocsf_emit! obtains correlation only from the current active context, the new INFO level deny span needs to remain current through the actual OCSF emission.

Wrapping only the policy evaluation would still leave the emission hook with no trace context. The other option would be to capture the correlation while the span is active and carry it explicitly to the event.

A narrow way to implement and test this seems to be:

add the plain data Trace correlation and sampled context extractor as proposed;
make the INFO deny span cover the OCSF emission point, rather than just the authorization call;
add downgrade coverage so trace and the Trace profile are removed for OCSF 1.3/1.1;
keep an acceptance test showing that a deny event at the default log level contains a trace.uid that resolves to the exported supervisor trace.

Does that match the intended implementation boundary for the deny span and the older schema downgrade path?

@HarryMoss

Thanks @HarryMoss, both points check out and they sharpen the plan.

Downgrade. Confirmed: the Trace profile and the trace object first appear in OCSF 1.4, and ocsf_schema_version only accepts 1.1 and 1.3 as downgrade targets. So the existing <= 1.3 branch in format/downgrade.rs is the right place: strip the trace object and the trace profile there, with a regression test next to the existing ones.

Deny span lifetime. Also confirmed, on all the L4 egress paths: authorization runs inside in_scope(...) and the deny events are emitted after it returns. There's a second reason the span can't wrap only authorization: the decision isn't known until authorization finishes, so a span that exists only for denies has to be opened in the deny branch anyway. The plan is a small helper that every deny site enters around its OCSF emission, carrying the decision attributes (policy, reason, destination). That includes the L7 deny sites in the relay and middleware. The automatic hook then picks up the context without call-site changes, and the explicit setter stays available for emissions outside such a span.

One consequence worth stating: at the default log level the DEBUG connect span is disabled, so the deny span has no exported parent and becomes a one-span trace of its own. trace.uid still resolves and the span carries the decision attributes, but the full connect, authorize, resolve and dial tree only shows up at debug level. I think that's the right trade-off for span volume. The acceptance test should check exactly that: at the default level, a deny event's trace.uid resolves to an exported trace containing the deny span.

So yes, your four steps match the boundary I have in mind. I'll fold both points into the issue body.

On implementation: I have nothing in progress on this. If you'd like to take it on, @HarryMoss (or @kvnloo, since you looked at it first), go ahead. I'm happy to collaborate, answer design questions as they come up, and review the PR. If neither of you wants to pick it up, just let me know and I'll start on it myself.

@rhuss

Thanks @rhuss, I’m happy to take it on and I’ll start looking at it tonight and tomorrow.

@kvnloo, if you’ve already started implementation please let me know, I’m happy to leave it with you rather than duplicate.

@HarryMoss

Thanks @HarryMoss, please go ahead and take the lead! I’d started an implementation, but it never got pushed and that workspace is currently inaccessible, so I don’t have a usable branch to hand over. No need to wait on me. Happy to help review/test your PR. Thanks for catching the downgrade and span-lifetime cases, and @rhuss for clarifying the implementation boundary.

@kvnloo

Thanks @kvnloo, really appreciate that. I’d definitely welcome your review and testing, and if you have any thoughts on the approach below I’d be keen to hear them too.

@rhuss, I’ve got a local implementation of #2640 ready and I’m doing the final code review before committing. It covers:

sampled trace.uid correlation, explicit builder overrides, and automatic enrichment without adding OpenTelemetry dependencies to openshell-ocsf;
a shared INFO deny span around the actual L4 and L7 OCSF emissions, including a root trace at the default log level and a parented trace at debug level;
trace object and profile removal for 1.1 and 1.3, plus the documentation updates.

The focused regressions and affected crate suites pass, including the loopback OTLP tests. Linux ARM64 compilation also passes, I haven’t run a deployed sandbox E2E.

I also noticed #4288 now overlaps the downgrade path and several builder and schema changes. I’m keeping those broader fixes separate from #2640.

I’d be interested in both your thoughts on the implementation. @rhuss, from a merge order point of view, would you prefer I wait for #4288 to land and then rebase and adapt #2640, or prepare #2640 for review in parallel?

@HarryMoss

@HarryMoss great, thanks for moving so fast, and thanks @kvnloo for handing it over.

On merge order, I'd prepare #2640 for review in parallel rather than wait. #4288 is a much larger change and still in review. The overlap with #2640 is small and mechanical, mainly format/downgrade.rs, which #4288 rewrites, so whichever lands second rebases. The final order is the maintainers' call, of course. cc @zanetworker

Three things should make the parallel path smooth:

Keep the downgrade change in its own commit. fix(ocsf): emit schema-conformant OCSF events and honest downgrades #4288 replaces the strip lists with a schema-driven downgrade. Its generated 1.1 and 1.3 definitions have no trace, so the object moves to unmapped.downgraded_attributes and the trace profile is dropped automatically. If fix(ocsf): emit schema-conformant OCSF events and honest downgrades #4288 lands first, that commit simply goes away. trace.uid then survives a downgrade under unmapped, which is arguably better for correlation in older SIEMs and still meets the acceptance criterion. So please have the downgrade tests assert "no top-level trace, no trace profile" rather than "no trace ID anywhere".
Vendor the Trace profile attribute on every class you touch. The vendored 1.8 http_activity.json already has the profile's trace attribute, but network_activity.json and base_event.json don't, and the trace and span objects aren't vendored at all. Today's validator only checks top-level non-profile attributes, so it won't notice. fix(ocsf): emit schema-conformant OCSF events and honest downgrades #4288's validator rejects undefined attributes, so a Network Activity event with trace would fail there. Running your events through fix(ocsf): emit schema-conformant OCSF events and honest downgrades #4288's validator once, for example by cherry-picking onto its branch locally, would catch this early. But I guess this also work just when rebasing later on, so probably not that urgent.
Deployed check. I have a local Podman setup with an OTel Collector and Tempo from the relay work. Once you push, I'm happy to trigger a deny at the default log level and confirm the trace.uid resolves in Tempo, on top of reviewing the code.

@rhuss

Thanks @rhuss, that sounds good. #2640 is close to being ready for review in parallel, and I’ll keep the downgrade change in its own commit.

I’ve updated the vendored Trace schema coverage and checked fixtures from all nine builders, with and without trace correlation, against #4288’s validator in a separate local checkout. The downgrade tests check that successfully converted 1.1 and 1.3 events have no top-level trace or Trace profile, without requiring the trace ID to disappear from unmapped.

I’m finishing the rebase and final review before pushing the branch. Once it’s up, I’ll share the branch link and exact commit for your Podman and Tempo check. Thanks for offering to run that, and @kvnloo for offering to review and test.

One production question for later is how we interpret a trace.uid that doesn’t resolve in Tempo. A failed lookup alone doesn’t tell us whether the trace is delayed, has been lost, or cannot currently be queried. I’ve added a brief note to the documentation in my local #2640 changes that correlation does not guarantee successful export or retention.

Would it be useful to capture that as a separate investigation, particularly around supervisor restarts and collector backpressure? I’d keep it outside #2640 and wouldn’t add requirements to the deployed check you’ve offered.

cc @zanetworker

@HarryMoss

Thanks @HarryMoss! The parallel review approach looks good, and validating all nine builders against #4288's stricter schema checks was a great catch.

I agree with keeping the downgrade change isolated and preserving trace correlation under "unmapped" where applicable.

Once your branch is pushed, I'm happy to review the sampled-context extraction, explicit overrides, and L4/L7 deny-span lifecycle.

On unresolved Tempo lookups, I think a separate investigation makes sense. We should distinguish delayed ingestion, dropped exports during supervisor restarts or collector backpressure, and retention/query failures. That shouldn't block #2640.

Thanks again for driving the implementation, and @rhuss for clarifying the merge and validation strategy!

@kvnloo

Thanks @kvnloo and @rhuss. The branch is now pushed at revision 3fae097a5d4355eeff77746f3ea90e89228455e3.

Both commits are signed off, with the downgrade change kept separate.

@kvnloo, this is ready for your review of sampled-context extraction, explicit overrides and the L4/L7 deny-span lifecycle. @rhuss, it’s available for the deployed default-level deny lookup in Collector/Tempo; that check is still pending.

I’ll keep the broader Tempo reliability investigation separate from #2640.

@HarryMoss

Thanks @HarryMoss — reviewed 3fae097a5d4355eeff77746f3ea90e89228455e3 on Linux x86_64 / Rust 1.95.0. No blocking finding in this bounded review. The head remains unchanged.

Executed on your exact branch:

cargo +1.95.0 test --locked -p openshell-ocsf -p openshell-otel: 220 passed, two doc examples ignored. Covers sampled/unsampled/invalid context, explicit overrides, automatic/routed enrichment, all nine builders, and both downgrade contracts.
cargo +1.95.0 test --locked -p openshell-supervisor-network --lib deny -- --test-threads=2: 59 passed, including all eight new correlation checks. Real emission helpers and the loopback OTLP receiver confirm default-level L4/L7 deny correlation and allowed/operational controls. telemetry.rs:63–86 keeps the INFO span current through actual synchronous OCSF dispatch.

Two integration boundaries to retain:

DEBUG parenting is conditional. proxy.rs:3026 drops the connect span before L7 relay. Later L7 denies can therefore be separate roots even at DEBUG. Your docs already qualify the live-parent condition; retain that qualification in the PR summary. The DEBUG regression proves staged L4 parenting, not a full CONNECT-to-L7 journey.
If fix(ocsf): emit schema-conformant OCSF events and honest downgrades #4288 lands first, retain your Trace schemas, let its schema-driven downgrade replace the separate strip-list implementation, and adapt the two assertions to DowngradeOutcome::Downgraded. Successful 1.1/1.3 output must lack top-level trace and its profile, while correlation under unmapped may remain. KeptNative must retain the whole 1.8 event, including trace/profile.

The separate schema review pinned @zanetworker's #4288 at 78621e8bb1fb9af4c91f9cf8dd5c3ebffc62b963: 3 isolated downgrade checks passed, plus one 54-case matrix using Harry's schema files over #4288's unchanged production functions. Results: 18 native no-ops, 24 successful downgrades, 12 correct kept-native outcomes; every result validates against its declared version. Commands were cargo test --locked -p openshell-ocsf --features test-support --test review_trace_downgrade -- --nocapture and cargo test --locked -p openshell-ocsf --lib review_schema_overlay -- --nocapture. Pinned receipt, source locations, replay fixtures and logs. This independently confirms your earlier nine-builder check; it is a schema overlay, not a compiled full implementation rebase. Suite counts overlap and should not be summed as unique regressions.

Credit to Harry for the implementation, @rhuss for the schema/lifetime design and planned deployed check, and @zanetworker for #4288. These local checks do not establish deployed Collector/Tempo delivery; that validation remains with @rhuss. No competing implementation or PR created.

Navigation Menu

Repository navigation

feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640

User Story

Problem Statement

Impact / Why This Matters

Proposed Design

Representation

Layering

Egress deny spans

Out of scope: agent trace context

Scope

Dependencies

Acceptance Criteria

Alternatives Considered

Agent Investigation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

feat(ocsf): add trace_id/span_id correlation fields to OCSF event builders #2640

Description

User Story

Problem Statement

Impact / Why This Matters

Proposed Design

Representation

Layering

Egress deny spans

Out of scope: agent trace context

Scope

Dependencies

Acceptance Criteria

Alternatives Considered

Agent Investigation

Activity

kvnloo commented on Sep 27, 2026

rhuss commented on Oct 4, 2026

HarryMoss commented on Oct 6, 2026

rhuss commented on Oct 7, 2026

HarryMoss commented on Oct 7, 2026

kvnloo commented on Oct 7, 2026

HarryMoss commented on Oct 7, 2026

rhuss commented on Oct 8, 2026

HarryMoss commented on Oct 8, 2026

kvnloo commented on Oct 8, 2026

HarryMoss commented on Oct 9, 2026

kvnloo commented on Oct 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions