Skip to content

caude hook trace propagation - #330

Open
davidabram wants to merge 16 commits into
mainfrom
claude-hook-trace-propagation
Open

caude hook trace propagation#330
davidabram wants to merge 16 commits into
mainfrom
claude-hook-trace-propagation

Conversation

@davidabram

@davidabram davidabram commented Oct 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Validation evidence: Linux CI fixes (a4ee2a3, 126221b)

Linux Nix CI on 93808955 failed 3 of 666 tests (run 39295526499). The macOS-specific issues are handled separately.

1. Auth guidance tests (a4ee2a36)

login_device_flow_storage_failure_uses_noncircular_guidance and new_session_invalidates_previous_telemetry_credential_before_saving failed because the Linux and Windows login guidance wraps Running login again does / not help until across a line break. The assertion was brittle; the message was correct. assert_login_storage_guidance now collapses whitespace before asserting. It also asserts that credential-store login guidance never says Run \sce auth login` in an interactive terminal., which keeps the noncircular check. The messages, the auth.storage_unavailablekey, and exit code 4 are unchanged.services::auth_command`: 17/17 pass on Linux.

2. Turso multiprocess WAL panic (126221bd)

concurrent_legacy_and_new_writers_never_lose_committed_rows failed with a new-writer child panic during turso::Builder::build: shared WAL frame index length changed while publishing an entry.

Root cause: an upstream Turso 0.8.1 defect, triggered by a legacy-to-new handoff.

  • A legacy-mode process writes auth.db-wal without updating the auth.db-tshm authority.
  • The next new-mode cold open sees its disk scan disagree with that authority. It takes repair_or_reseed_authority_from_local_disk_scan's rebuild branch (wal.rs:1680, from seed_or_sync_authority), which discards and republishes the shared frame index.
  • That branch assumes exclusivity, but its only guard is !writer_or_checkpoint_lock_active(), and the publish mutex is process-local. Concurrent cold openers therefore rebuild together.

The backtrace confirms this path.

  • Panics are not the only outcome. Successful opens also read stale snapshots: the credential was missing, or the schema looked outdated (failed to run auth DB migrations).
  • This affects production hook processes (read-only path): the first burst of concurrent hooks after any legacy-binary write could crash or silently miss the credential.
  • It is not an SCE migration-ordering bug, and not a test orchestration bug.

Fix: an exclusive std::fs::File::lock on the sidecar auth.db-open.lock, held only around Builder::build() and connect(). It is taken after the compatibility lock, so a legacy holder is still rejected first. With the compatibility lock in place, the authority can only be stale while no new process holds the DB, so the first serialized opener rebuilds alone.

  • Queries, writes, and migrations are not serialized.
  • The Linux F_OFD_SETLK and macOS flock locks are unchanged.
  • The wait runs on a blocking thread, so the 2 s hook credential deadline still applies.

New regression: concurrent_cold_opens_after_legacy_write_all_see_committed_state runs 20 rounds of one legacy write followed by 8 concurrent new readers. Every reader must resolve the credential, and every legacy row must persist with integrity_check ok.

Evidence (local Linux x86_64, isolated sandboxes)

Check Result
Mixed test, baseline binary 2 failures / 220 runs (1/60 @4-par, 1/160 @8-par), same panic as CI
Probe: 8 concurrent readers after a legacy write, ×20 rounds, baseline 7–14 failed opens per 160, across 6 runs (panics + stale reads)
Same probe after a new-mode write (control) 0 / 160
New regression without the cold-open lock (mutant) 20/20 runs fail
New regression with the fix 40/40 pass (@4-par, alongside mixed-test load)
Mixed test with the fix 600/600 pass (400 @8-par + 200 @4-par)
Full services::auth_db::multiprocess_tests module with the fix 12/12 runs pass @4-par (21 tests each)
Compatibility lock removed (mutant), mixed test 5/5 runs fail: committed legacy rows lost
32-reader cold-open burst ~55 ms → ~160 ms wall time (≈3 ms per serialized open)
nix flake check -L green: 669 passed, plus clippy and fmt
Linux Nix CI on 126221bd (run 39313063523) green: 669 passed, plus cli-clippy and cli-fmt

Remaining risks

  • Turso 0.8.1 still has the defect; SCE avoids triggering it for auth.db only. The Agent Trace DB (TursoDb) does not take the cold-open lock.
  • A process that hangs inside Builder::build() delays other cold opens. Hook lookups still give up at their deadline, and the kernel releases the lock when that process exits.
  • The earlier documented limits on concurrent creation and migration remain (see context/sce/auth-db.md).

🤖 Generated with Claude Code

davidabram and others added 6 commits October 17, 2026 13:28
@davidabram @shared-context-engineering
context: Define Claude hook trace propagation plan
811f91e 
Capture the staged implementation plan for connecting Claude Code hook execution to managed OTLP tracing, including propagation probing, hook span admission, child spans, privacy and parity constraints, and live verification.

Plan: claude-hook-trace-propagation
Tasks: T01, T02, T03, T04, T05

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
context: Record Claude hook trace propagation evidence
738a129 
Capture Claude Code 2.1.301 probe results for PreToolUse and PostToolUse hooks,
including the interaction-span parent context, tool_use_id correlation, caller
environment markers, and telemetry gating behavior. Update the context map and
complete T01 with the qualified parent_context_available verdict, while
documenting that future SCE hook spans must be siblings of Claude tool spans
under claude_code.interaction.

Plan: claude-hook-trace-propagation (T01)

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
runtime: Admit Claude hooks to managed trace export
80cbfad 
Enable managed OTLP telemetry for Claude-invoked hook processes while keeping non-Claude hooks and the telemetry-header helper on Noop. Classify admitted invocations before credential lookup, emit one privacy-filtered `sce.hook` span with Claude correlation IDs, and preserve fail-open hook behavior under credential or exporter failures. Add admission, payload-redaction, and parity coverage while updating the observability contract and decision record.

Plan: claude-hook-trace-propagation (T02)

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
telemetry: Parent Claude hook spans on propagated trace context
5545eec 
Join Claude hook telemetry to Claude Code's W3C trace when a valid parent is available, while preserving independent roots for missing or invalid context. Extract TRACEPARENT/TRACESTATE only for Claude hooks, attach the parent to the root span, and verify child-span inheritance across async polls and worker migration.

Plan: claude-hook-trace-propagation (T03)

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
observability: Export Claude hook operation child spans
eae7be8 
Expose the mutation-scope, Git snapshot, and persistence phases of admitted Claude hooks as low-cardinality OTLP child spans so hook traces show which operations actually ran without changing command behavior. Add reusable span instrumentation at the Claude adapter, mutation-trace coordinator, and scope runtime boundaries, and filter child spans out of ordinary command exports.

Ref: context/plans/claude-hook-trace-propagation.md (T04)

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
context: Record Claude hook telemetry verification
1f10987 
Complete T05 for the claude-hook-trace-propagation plan by recording Dash0 evidence for `sce doctor` and a Claude Code session, documenting the verified and unverified span hierarchy, and adding an opt-in live smoke procedure. Sync the current admission, session-boundary behavior, credential, and context-map documentation.

The live evidence confirms standalone and session-boundary exports while leaving parented tool-hook delivery unverified; local validation and smoke results are recorded without claiming collector delivery.

Plan: claude-hook-trace-propagation, task T05

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram
davidabram force-pushed the claude-hook-trace-propagation branch from b3d9d9f to 1f10987 Compare October 17, 2026 17:34
davidabram and others added 10 commits October 17, 2026 20:05
@davidabram
docs: verify live Claude-to-SCE trace hierarchy in Dash0
9d4f548
@davidabram @shared-context-engineering
runtime: Harden concurrent hook telemetry and diagnose fallbacks
8e47d2b 
Prevent concurrent Claude hook processes from silently losing managed telemetry when encrypted auth.db opens contend. Enable multiprocess WAL and an auth DB busy timeout, avoid migration writes on current schemas, serialize and retry pending migrations, and skip database creation for missing-credential reads.

Classify and log Noop fallbacks, identify hook entry points with `sce.hook.command`, and add multiprocess coverage for readers, migrations, and correlated hook spans. Document rollout risks and validation evidence in the `claude-hook-trace-propagation` plan.

Ref: #330

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
runtime: Harden auth DB compatibility and Claude hook telemetry
84847c8 
Prevent mixed-version auth DB writers from overlapping by holding a shared compatibility lock, and make telemetry credential lookups open existing databases without creating files or encryption keys. Record each Claude hook command before reading stdin so malformed or unreadable payloads still identify the hook span. Update regression coverage and the related auth DB and observability documentation.

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
runtime: Keep hooks and setup from prompting for auth DB credentials
7befb20 
Claude hooks, managed telemetry startup, and `sce setup` could open or create the encrypted auth DB and trigger OS credential-store prompts (such as the macOS Keychain dialog) in unauthenticated or non-interactive contexts. Credential access is now explicit: `CredentialAccess::Interactive` is limited to `sce auth login`, `logout`, and `whoami`, while hooks, telemetry startup, `sce auth telemetry-headers`, and `sce sync` use `NonInteractive`, which only reads existing keys and, on macOS, disables Keychain user interaction. Inaccessible keys surface as a typed `CredentialAccessUnavailable` error. Credential-store access is serialized per process, and only a missing keyring entry may generate a new key.

Add a non-secret telemetry readiness hint (`telemetry-ready.json`) that `sce auth login` writes after persisting a telemetry credential and that logout clears, so managed telemetry returns Noop (`not_configured`) without opening `auth.db` when no hint exists. `sce setup` now only creates the auth DB parent directory, no longer enrolls repositories, and is excluded from managed telemetry. Non-login paths open only an existing auth DB, and token storage caches only successful opens. Update regression coverage and the auth DB, telemetry credential, and observability documentation.

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
runtime: Use flock for the macOS auth DB compatibility lock
444ebb2 
On macOS the shared compatibility lock used `fcntl(F_SETLK, F_RDLCK)`, a process-scoped record lock that POSIX releases whenever the process closes any descriptor for `auth.db`. The guard could silently drop while the `EncryptedTursoDb` was still open, letting a legacy writer through.

macOS now takes `flock(LOCK_SH | LOCK_NB)` on the SCE-owned descriptor. The lock belongs to the open file description, so it lasts until the `EncryptedTursoDb` is dropped or the process exits, and it still conflicts with the legacy whole-file `F_WRLCK` in both directions. Linux and other Unix locking is unchanged; `EWOULDBLOCK` is now also classified as a lock conflict.

Add four macOS-only multiprocess tests covering mutual exclusion with the legacy lock, survival across an unrelated descriptor close, and concurrent new holders. Record macOS validation evidence and residual risks in the auth DB documentation and the claude-hook-trace-propagation plan.

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
auth-db: Coordinate concurrent writer lifetimes
833948f 
Prevent compatibility regressions from masking insert failures when Turso WAL connection startup and teardown overlap concurrent writes. Keep all writers open through coordinated inserts, report row-specific errors, and document the lifecycle-race isolation while preserving the legacy reopen checks.

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
doctor: Report managed telemetry health and auth recovery steps
9380895 
Expose read-only managed telemetry health in doctor text and JSON output,
using readiness hints and the bounded non-interactive credential lookup.
Keep unconfigured installs warning-free and report disabled telemetry as
manual-only warnings without changing credentials or blocking hooks.

Provide platform-specific auth storage recovery guidance and separate login
failures to avoid circular instructions while preserving error identifiers
and exit codes. Cover health classification, output, auth side effects, and
real database states; correct the macOS multiprocess sandbox state path.

Plan: claude-hook-trace-propagation (auth storage guidance and telemetry health follow-up).

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
auth-db: Classify auth storage failures with typed causes
afc7029 
Auth storage failures from auth commands and `sce sync` always rendered the
macOS Keychain guidance, even when the real cause was a locked or unreadable
`auth.db`. The cause was lost because every downstream mapper collapsed into
one variant and matched on rendered text.

Introduce a typed `AuthStorageFailure` cause (`CredentialAccess`,
`DatabaseLocked`, `Database`) and thread it from the sources to
`UserError::AuthStorageUnavailable`/`LoginAuthStorageUnavailable`. Add typed
markers so classification never depends on diagnostic text:
`CredentialAccessDenied` for keyring authorization failures (tagged by
`NoStorageAccess` or the macOS `OSStatus` codes), `db::DatabaseLocked` for the
last encrypted-open attempt failing on the compatibility-lock conflict or a
Turso `Busy`/`BusySnapshot`, and `open_lock::SharedLockConflict` with unchanged
text. `ControlPlaneError::Storage` now carries the cause, and
`is_storage_failure()` became `storage_failure() -> Option<AuthStorageFailure>`
across the control-plane, stream, and sync error types.

Guidance now matches the cause: DB causes say the DB is locked or unreadable
and never mention the Keychain, "Always Allow", or `sce auth login`; the macOS
command guidance gained "or run `sce doctor` to verify telemetry access".
Managed telemetry still reports `database_error` for a locked DB. The key
`auth.storage_unavailable`, the runtime class, exit code 4, and the rendered
compatibility-lock rejection text are unchanged.

Plan: context/plans/claude-hook-trace-propagation.md

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
auth: Normalize whitespace in login storage guidance assertions
a4ee2a3 
The Linux and Windows login guidance wraps "Running login again does not
help until" across lines, so the exact-substring assertion in
`assert_login_storage_guidance` failed on Linux CI. Collapse whitespace
before asserting, and also assert that credential-store login guidance never
tells the user to run `sce auth login` again. The messages, the
`auth.storage_unavailable` key, and exit code 4 are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-authored-by: SCE <sce@crocoder.dev>
@davidabram @shared-context-engineering
auth-db: Serialize Turso cold opens after legacy WAL writes
126221b 
A legacy-mode SCE process writes auth.db-wal without updating the
multiprocess authority in auth.db-tshm. The next new-mode cold opens then
take Turso 0.8.1's exclusive rebuild branch in
`repair_or_reseed_authority_from_local_disk_scan`. That branch is guarded
only by "no writer active" and a process-local publish mutex. When several
processes open concurrently, they rebuild the shared frame index at the same
time. The result is a `shared WAL frame index length changed` panic, which
is the Linux CI failure, or a successful open that reads a stale snapshot,
such as a missing credential or an outdated schema.

Take an exclusive `File::lock` on a sidecar `auth.db-open.lock` around
`Builder::build()`/`connect()` only, after the compatibility lock succeeds.
Queries, writes, and migrations are not serialized, and the `F_OFD_SETLK`
and macOS `flock` compatibility locks are unchanged. Add a regression that
runs 8 concurrent cold opens after each of 20 legacy writes. It failed
20 of 20 runs without the lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-authored-by: SCE <sce@crocoder.dev>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@davidabram