Skip to content

fix(harness): hold the Claude prompt until Claude Code can trace the turn - #2943

Open
dhaifley wants to merge 1 commit into
kagent-dev:mainfrom
dhaifley:dhaifley/claude-trace-init-race
Open

dhaifley wants to merge 1 commit into
kagent-dev:mainfrom
dhaifley:dhaifley/claude-trace-init-race

Conversation

@dhaifley

Copy link
Copy Markdown
Contributor

A Claude Harness turn against the real Anthropic API exports no claude_code.* spans under the turn's trace, so the trace holds only the harness runtime's spans (#2922). This holds each prompt until Claude Code can trace it, which makes Claude's spans arrive on every turn.

Closes #2922

Why

Claude Code starts a print-mode turn without waiting for its own OpenTelemetry initialization. With an Anthropic API key and no custom base URL, it first waits on a fetch of the organization's managed settings from api.anthropic.com/api/claude_code/settings. Until the tracer provider and W3C propagator register, every span is a no-op span and TRACEPARENT is ignored. In an actor the turn starts first, so claude_code.interaction is lost and later llm_request and tool spans become roots of new traces. With Claude Code's debug log routed out of the actor, the first turn logged [engine] turn 1 start 16 ms before isTelemetryEnabled, and the second 110 ms before the settings fetch settled. The e2e passes because its mock sets ANTHROPIC_BASE_URL, which skips the fetch. It is not a flush problem. The driver already waits for the process to exit, and Claude Code flushes what it recorded on exit. Claude Code 2.1.260 and 2.1.282 behave the same, and neither has a setting that makes print mode wait.

What changes

  • The driver passes the prompt on stdin with --input-format stream-json instead of as an argument, and closes stdin on the terminal result so Claude exits as before.
  • When traces are exported, the driver adds Claude Code's prometheus metrics reader on a free loopback port. It writes the prompt once claude_code.session.count appears there. Claude increments that counter only after initializeTelemetry has registered the tracer provider and propagator, so the counter proves the turn will be traced. Both are documented Claude Code telemetry surfaces.
  • The wait is bounded at 10 seconds, which caps the delay a stalled settings fetch can add. Claude Code gives each settings request 10 seconds and retries it, so during such a stall the turn runs without its Claude spans and logs a warning, as it also does if the port cannot be reserved.
  • OTEL_EXPORTER_PROMETHEUS_HOST and OTEL_EXPORTER_PROMETHEUS_PORT join the Claude-owned environment, so a Harness cannot redirect the reader.

The gate costs about 100 ms on a turn whose settings fetch is still in flight, and next to nothing when it has settled. Bedrock, Vertex and custom base URLs skip the fetch, so their gate opens as soon as initialization finishes, which also closes the smaller race they had.

Verification

  • On kind with Agent Substrate 0.2.0-beta5, this change applied to v1.0.0-alpha2 and the real Anthropic API, 10 of 10 turns exported Claude's spans under the turn's trace. They were cold and warm unary turns, a Bash tool turn (chat, execute_tool, chat) and a SendStreamingMessage turn. The published claude-harness image on the same cluster exported none in 7 of 7 turns, 3 of them run alongside.
  • Without Substrate, the pinned image in Docker behind a proxy that adds 1.5 s to api.anthropic.com exported nothing in 2 of 2 warm runs with stream-json input alone. With the gate it exported both runs' spans, and the turn started 47 to 50 ms after initialization.
  • TestProcessDriverAwaitsTracing runs the test binary as a Claude Code that registers tracing 200 ms after start, and asserts the prompt arrives afterwards. It fails with the gate off. TestTracingGateWait, TestTracingGateReadyReportsUnreadableScrape, TestTracingGateEnvironment and the updated TestProcessDriverArgumentsAndStream cover the bound, early exit, a scrape that cannot be read, the metrics environment and the stdin message.
  • go test ./harness/... ./core/internal/translator/..., go vet, gofmt and go fix are clean, and so is golangci-lint over ./harness/claude/... with the repository config minus its kube-api-linter plugin, which does not load on macOS and checks only API types.

…turn

Signed-off-by: David Haifley <david.haifley@solo.io>
@github-actions github-actions Bot added the bug Something isn't working label Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Claude harness: Claude Code exports no spans against the real Anthropic API

1 participant