feat(observability): add OTEL traces and logs to harnesses by supreme-gg-gg · Pull Request #2805 · kagent-dev/kagent

supreme-gg-gg · 2026-09-11T21:57:03Z

Adds correlated OpenTelemetry tracing and logging across the Kagent, Codex, and Claude harnesses. Refactors shared tracing configuration and setup code. Builds on top of previous V2 observability works like #2663 and #2786. Also closes #2760 and partially addresses #2762.

Key Changes

Resolve and validate OTLP trace and log settings consistently across all harness compilers.
Propagate telemetry configuration, service identity, and collector egress into runtime revisions.
Configure Codex and Claude native trace and log exporters with content capture enabled.
Propagate W3C trace context through Codex JSON-RPC requests and Claude process environment.
Initialize shared Go tracing in the native harness wrappers and move common tracing code out of the ADK package.
Flush request spans before terminal and input-required events cause an Actor to pause.

Notes

Traces and logs are independently enabled through OTEL_TRACING_ENABLED and OTEL_LOGGING_ENABLED
Invalid or missing enabled-signal endpoints and unsupported protocols now fail harness compilation.
Span attributes for Claude Code is documented here and available events / logs is documented here
Events / logs emitted by Codex is documented here. There is no official docs for Codex's span hierarchy and attributes
Codex and Claude Code force flushes traces when their process exists before the gateway Suspends the instance. The adapter process will also flush its own traces before it responds with a terminal event as per the changes in go/adk/pkg/a2a/server/server.go. However, during a pending HITL, since we do not quit the Codex or Claude process, it is not guaranteed that all traces are delivered immediately.
For both Codex and Claude, native spans after HITL completes remain in the original request trace because the in-flight process carries the same trace context (through env var or JSON-RPC). The resume A2A trace may contain only controller and Go wrapper spans.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

supreme-gg-gg · 2026-09-11T23:21:49Z

This recording shows the traces and logs for ADK, Claude, and Codex harness in Grafana

input.mp4

krisztianfekete

Thanks @supreme-gg-gg, this is looking really good, added some comments/questions!

cc. @dhaifley w/r/t the flushing problems you were working on recently.

krisztianfekete · 2026-09-14T15:19:48Z

+		return nil
+	}
+	span := trace.SpanFromContext(ctx)
+	span.End()


Since this ends otelhttp's server span, so some attributes are likely lost. You can start your own child span in Before and end that one instead, or capture the status yourself before ending.

krisztianfekete · 2026-09-14T15:23:55Z

+			corev1.EnvVar{Name: "OTEL_LOG_USER_PROMPTS", Value: "1"},
+			corev1.EnvVar{Name: "OTEL_LOG_TOOL_DETAILS", Value: "1"},


These are all off by default, so maybe we should wire these up to Helm, so it's opt-in? We should do this for all similar stuff as well (outside of this PR).

For this and the following comments, I think we can have two helm values:

one for including details like user prompt, tool, assistant response etc. which will apply to relevant settings for both traces and logs, defaulted to false (an alternative could be to have them configurable for traces and logs separately, but I assume most user would want both since traces would not contain all the details and would result in very granular settings)

the other one for logging API bodies, since that one could be noisy and rarely useful compared to the first one

krisztianfekete · 2026-09-14T15:25:08Z

+				corev1.EnvVar{Name: "OTEL_LOG_ASSISTANT_RESPONSES", Value: "1"},
+				corev1.EnvVar{Name: "OTEL_LOG_RAW_API_BODIES", Value: "1"},


These should potentially be opt-in as well, controlled separately from prompt logging as I'd expect this to be quite noisy/expensive?

krisztianfekete · 2026-09-14T15:26:12Z

+
+	parsed, err := url.Parse(endpoint)
+	if err != nil || (parsed.Scheme != "http" && parsed.Scheme != "https") || parsed.Hostname() == "" || parsed.User != nil || parsed.RawQuery != "" || parsed.Fragment != "" {
+		return signalConfig{}, fmt.Errorf("OTLP %s endpoint must be an absolute HTTP(S) URL without credentials, query, or fragment", signal)


Can we check this at startup, and log a warning if it's wrong instead?

krisztianfekete · 2026-09-14T15:28:31Z

+	if cfg.Telemetry != nil {
+		native.Otel = &nativeOtel{
+			LogUserPrompt: cfg.Telemetry.CaptureContent,
+			Environment:   "kagent",


What's the Codex default here?

The default value for environment should be "dev" here: https://learn.chatgpt.com/docs/config-file/config-advanced#observability-and-telemetry; I'll remove this kagent setting

krisztianfekete · 2026-09-14T15:30:06Z

+	environment = append(environment, traceConfig.Environment()...)
+	environment = append(environment, logConfig.Environment()...)


I think we have to set OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES so we are fully consistent with our wrappers, don't we? Likely in both Codex and Claude.

I don't see an option to configure service name in the docs unfortunately. We're already propagating traceparent and tracestate so that would give us trace correlation. Maybe we can just let each native CLI own their service identity?

Most SDKs should honor OTEL_RESOURCE_ATTRIBUTES by defaukt, can you please check if that's the case here?

I tried setting this for both harness but I don't seem to see the custom attributes. For Codex I think this is a known issue openai/codex#30987, not too sure why it's not working for Claude

krisztianfekete · 2026-09-14T15:31:37Z

 	for _, variable := range input.Harness.Spec.Env {
-		if _, reserved := ownedEnvironment[variable.Name]; reserved || strings.HasPrefix(variable.Name, mcpCredentialPrefix) {
+		_, reserved := ownedEnvironment[variable.Name]
+		if reserved || strings.HasPrefix(variable.Name, mcpCredentialPrefix) || strings.HasPrefix(variable.Name, otelEnvironmentPrefix) {


We have to pass some OTEL_ env vars to not break functionality (I am thinking mainly about the export intervals, but maybe even the endpoints?)

krisztianfekete · 2026-09-14T15:34:02Z

+// TraceConfig is the controller-owned trace export configuration compiled into
+// each runtime revision.
+type TraceConfig struct {
+	Enabled  bool
+	Endpoint string
+	Protocol string
+	hostname string
+}
+
+// LogConfig is the controller-owned log export configuration compiled into
+// each runtime revision.
+type LogConfig struct {
+	Enabled  bool
+	Endpoint string
+	Protocol string
+	hostname string
+}


Maybe this could be a SignalConfig instead?

EItanya

🤖 AI-generated review. Validated the HTTP response attribute loss discussed below.

supreme-gg-gg · 2026-09-14T22:34:48Z

@krisztianfekete I've addressed your comments in ad7cf7f, thanks for the review!

krisztianfekete

Looks good to me! I am a bit paranoid about potential flushing problems still, can we maybe have an e2e test to ensure that part os robust as well?

supreme-gg-gg · 2026-09-15T18:14:21Z

@krisztianfekete I've added a flush-traces E2E test in 57cf9a7, can you take a look?

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

supreme-gg-gg requested a review from a team as a code owner September 11, 2026 21:57

github-actions Bot added the enhancement New feature or request label Sep 11, 2026

supreme-gg-gg requested a lite review from Copilot and removed request for a team September 11, 2026 22:02

Copilot AI reviewed Sep 11, 2026

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Sep 11, 2026

mesutoezdil mentioned this pull request Sep 14, 2026

fix: flush buffered OTEL logs #2772

Open

mesutoezdil reviewed Sep 14, 2026

View reviewed changes

Comment thread go/pkg/tracing/tracing.go

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Sep 14, 2026

krisztianfekete reviewed Sep 14, 2026

View reviewed changes

EItanya reviewed Sep 14, 2026

View reviewed changes

Comment thread go/adk/pkg/a2a/server/tracing.go

krisztianfekete reviewed Sep 15, 2026

View reviewed changes

supreme-gg-gg requested a review from Charlesthebird as a code owner September 15, 2026 17:34

EItanya reviewed Sep 16, 2026

View reviewed changes

Comment thread .github/workflows/ci.yaml Outdated

EItanya approved these changes Sep 16, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

supreme-gg-gg added 5 commits September 16, 2026 10:43

propagate telemtetry settings in compiler

fb213fe

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

tracing setup and propagation in harness

50e1f7c

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

refactor tracing from adk to shared pkg

f5bad5e

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

flush traces in adk server, this affects other harness too

26dcb3f

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

enable logging

0453807

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

supreme-gg-gg added 6 commits September 16, 2026 10:43

add helm values for logging exporter protocol like tracing

0042ca2

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

codex self review

b938116

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

address review comments

74733f4

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

tracing e2e

59d7f3f

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

merge ci steps

60473b8

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

enable tracing in e2e for all agents

e35ffda

Signed-off-by: Jet Chiang <pokyuen.jetchiang-ext@solo.io>

supreme-gg-gg force-pushed the jetc/feat/harness-tracing branch from 6959402 to e35ffda Compare September 16, 2026 14:43

EItanya added this pull request to the merge queue Sep 16, 2026

Merged via the queue into main with commit ff60768 Sep 16, 2026
27 checks passed

EItanya deleted the jetc/feat/harness-tracing branch September 16, 2026 15:26

dhaifley mentioned this pull request Sep 22, 2026

feat(tracing): add harness invocation identity, lifecycle, and bounded capture #2863

Merged

krisztianfekete mentioned this pull request Sep 24, 2026

[BUG] Claude harness: Claude Code exports no spans against the real Anthropic API #2922

Open

		corev1.EnvVar{Name: "OTEL_LOG_USER_PROMPTS", Value: "1"},
		corev1.EnvVar{Name: "OTEL_LOG_TOOL_DETAILS", Value: "1"},

		corev1.EnvVar{Name: "OTEL_LOG_ASSISTANT_RESPONSES", Value: "1"},
		corev1.EnvVar{Name: "OTEL_LOG_RAW_API_BODIES", Value: "1"},

		environment = append(environment, traceConfig.Environment()...)
		environment = append(environment, logConfig.Environment()...)

Conversation

supreme-gg-gg commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Changes

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

supreme-gg-gg commented Sep 11, 2026

Uh oh!

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

supreme-gg-gg Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

supreme-gg-gg Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

supreme-gg-gg Sep 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EItanya left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

supreme-gg-gg commented Sep 14, 2026

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

supreme-gg-gg commented Sep 15, 2026

Uh oh!

Uh oh!

This comment has been minimized.

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

supreme-gg-gg commented Sep 11, 2026 •

edited

Loading

supreme-gg-gg Sep 14, 2026 •

edited

Loading

supreme-gg-gg Sep 14, 2026 •

edited

Loading

supreme-gg-gg Sep 14, 2026 •

edited

Loading