feat(evaluations): preserve the tool trajectory for judges

[donei003]

Now carries two commits: the trajectory capture, and #98 (merged in here) which made every judge path share one message_history.

Problem

A judge can only grade what it is shown. Handler packages record tool traffic onto OpenTelemetry spans and return only {output, usage}, so by the time a judge ran, the calls made on the way to that output were gone. "Did the agent call the right tool, in the right order, with the right arguments?" was an unaskable question of an SDK that had just run the agent that answered it.

Underneath that, the three judge paths had already drifted. Each built message_history with its own inline join:

Path	message_history was
Offline evals	row input + output + format block
Online inline (run_judges)	user input + output + format block
Online deferred (run_judge)	output + format block — no input at all

A deferred judge was grading a response with no request beside it. That is a pre-existing bug, fixed here as a consequence of unifying.

Approach

Capture by wrapping the tool map, which both paths already own — so every handler package is covered without changing any of them, and a custom caller-supplied handler is covered too. A handler still resolves a tool by the key the model named and calls it.

• Offline: once per row, in evaluations/runner.py.
• Online: once per invocation, in tracking.execute_and_track / execute_and_stream, which now return the rendered trajectory alongside response and track_data.

judge_scoring.build_message_history is the only place a history is built — in the module that already owns the {score, reasoning} contract, for exactly the same reason. It orders the conversation the way it happened: input → trajectory → output → format block, skipping empty parts. All three paths call it.

What a judge now sees:

Where is order A1?

Tools available: lookup_order, issue_refund
Tool calls made while producing the response, in order:
1. lookup_order
   arguments: {"id":"A1"}
   result: order A1 shipped 2026-08-02
2. issue_refund
   arguments: {"id":"A1","amount":19.99}
   error: refund window closed

That order shipped on Aug 2 and is outside the refund window.

Your response MUST be in valid JSON format with the following structure:
...

Verified live against a real LaunchDarkly judge: the judge's reasoning cited the call, its arguments, and its result — all three only visible via the trajectory.

Properties pinned by tests

• The recorder observes; it never intervenes. A wrapped tool returns exactly what the original returned and raises exactly what the original raised. Calls past the 50-call cap still execute and are only counted — truncation drops the record, never the work, because a harness that changed the agent's behaviour would no longer be evaluating the agent.
• A recorder belongs to one invocation. Rows generate concurrently against one shared tool map, so a shared recorder would splice one row's calls into another's and hand the judge a conversation that never happened. Tested with two rows whose calls interleave.
• Order is call-start order, not completion order. A judge asked whether the agent searched before it refunded is reading a sequence.
• Inline and deferred produce byte-identical history for the same row. This is the property that makes a rubric portable between a production sample and a dataset replay.
• A tool result stays literal in the judge prompt. A tool result is a new injection surface alongside generated output, closed by the existing rule: the judge config is passed unrendered and the handler makes exactly one template pass. Tested with a tool returning the literal text {{expected_output}}.
• Back-compatible. A run with no observable tools adds no block, so a judge authored before this reads exactly the message_history it read before.
• Nothing is added to any event payload. The trajectory reaches LaunchDarkly only inside the prompt a judge was shown, never as a wire field the backend has not specified.

Design notes

• No standalone trajectory variable. An earlier revision exposed {{tool_trajectory}} too; it overlapped message_history and bought nothing, while inviting a rubric to interpolate both and pay for the trajectory twice. Confirmed live — a real judge config's own scaffolding already interpolates {{message_history}}. A test pins its absence. Since message_history is what judges cloned from the AI Library's default templates read, an existing judge becomes a trajectory judge by editing its rubric text alone.
• Native provider tools are skipped in both paths, and left out of "Tools available". Online, wrap_tool_handlers substitutes a callable tracking stub, so a native call is locally observable — but the stub returns nothing, so recording it would show a judge a call with an empty result while the provider's real result stayed invisible. Recording is therefore composed inside that wrapper, on the original map. Tests assert both the exclusion and that $ld:ai:tool_call still fires underneath.
• JudgeTask gains user_input and trajectory as plain strings — every field on it has to survive pickling to a worker thread; a test pins that.
• Graph-level judges get no trajectory, deliberately: graph_judge grades an answer produced across several nodes, and splicing their trajectories would describe a conversation that never happened. Per-node judges get their own node's.
• Bounds: 50 recorded calls per invocation, 2000 characters per rendered argument bag or result. A trajectory goes into a judge prompt, so an agent looping over a large result set would otherwise spend the judge's context window — and budget — on a tail no judge reads.
• trajectory.py sits at the package root, not under evaluations/, since it is no longer evaluations-specific.

Deliberately out of scope

Scorers cannot see the trajectory. Scorer.fn(row, output) is the contract, and the trajectory is not dataset-owned so it does not belong on DatasetRow. A deterministic check like "called lookup_order exactly once" is a natural follow-up but needs a contract change, not a quiet signature widening.

Validation

• uv run pytest -q — 1282 passed, 11 skipped
• uv run mypy packages/client/src/launchdarkly_ai_server — clean; ruff check / format --check — clean
• Exercised end-to-end in ai-sdk-evaluations-example (launchdarkly-labs/ai-sdk-evaluations-example#5), including a live run against a real judge

Language-agnostic spec: launchdarkly/ai-sdks-monorepo#13 — being restructured to describe the shared flow rather than the offline phase alone, now that this shape is settled. Submodule pointer: launchdarkly/ai-sdks-monorepo#14.

🤖 Generated with Claude Code

---

Note

Overview
Judges can now grade tool use as well as final answers by recording each invocation’s tool-call trajectory and folding it into {{message_history}} (input → trajectory → output → JSON format block), with no new template variable or event payload fields.

A new TrajectoryRecorder wraps tool implementations before handlers run (online in execute_and_track / streaming, offline once per concurrent eval row) without changing tool behavior; it renders ordered calls with limits (50 calls, 2k chars per arg/result), skips native/handoff/unexposed tools, and keeps trajectories out of generation wire events.

judge_scoring.build_message_history replaces per-path inline joins so inline, deferred (JudgeTask now carries user_input and picklable trajectory), offline evals, and per-node graph judges share the same transcript; deferred judges previously saw output only.

^{Reviewed by Cursor Bugbot for commit a137de8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 5517990. Configure here.}

[cursor]

Wrapper changes sync tool calls

Medium Severity

_record always replaces a tool with an async def wrapper, and _run_rows now hands that map to every offline handler. A custom handler that invoked a sync tool without await (the natural call for a sync function) now gets a coroutine object as the result and can finish the row with garbage output instead of raising.

Additional Locations (1)

• packages/client/src/launchdarkly_ai_server/evaluations/runner.py#L555-L562

 

^{Reviewed by Cursor Bugbot for commit 5517990. Configure here.}

[devin-ai-integration]

Note

This report is out of date. Scroll down for Devin Review's latest report on this PR.

Devin Review found 2 new potential issues.

[devin-ai-integration]

🟡 Mutated tool arguments mislead judges

When a tool mutates its argument mapping, _call_arguments retains the same object and records its post-call contents. Judges then grade different arguments from those the model supplied.

Learn more

The recorder reserves a call before invoking the tool, but _call_arguments returns mutable positional arguments by reference. A tool can mutate that object before render_trajectory serializes it. The recorded trajectory then describes the mutated state, not the call boundary.

Example: The model calls lookup_order with {"id": "A1"}. The implementation runs order_id = args.pop("id") and returns successfully. The trajectory renders arguments: {} although the model supplied A1.

Recommended fix: Capture a stable snapshot in _reserve without changing the object passed to the tool. Handle values that cannot be copied or serialized so trajectory capture remains observational and never fails the invocation.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 Trajectory values exceed their cap

_truncate appends its suffix after retaining 2,000 characters, so each oversized value exceeds the documented cap. Large trajectories can exceed their promised prompt budget.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Devin Review found 1 new potential issue.

[devin-ai-integration]

🟡 Unrenderable tool values fail invocations

When a tool value’s string conversion raises, _render_value lets that exception escape after generation succeeds. invoke() then fails, and streaming omits its final event despite successful provider output.

Learn more

Trajectory rendering runs after the handler has returned successfully. json.dumps(..., default=str) can propagate any exception raised by a value's __str__, but this block catches only TypeError and ValueError. The fallback str(value) can also raise, so recording introduces a new failure after provider work and tool side effects have completed.

Example: A tool returns an object whose __str__ raises RuntimeError("closed"). A custom handler consumes that object and returns valid output, but execute_and_track raises while rendering the trajectory instead of returning the output.

Recommended fix: Make _render_value total: catch exceptions from both JSON encoding and fallback conversion, then emit a fixed placeholder or safe type name. Add invoke and streaming tests proving malformed representations cannot suppress a successful response.

---

Was this helpful? React with 👍 or 👎 to provide feedback.