Repository navigation
docs(results): why model-picked follow-up injections were not built - #43
Merged
Merged
Conversation
…cost The 2026-09-30 completeness replay chose by rule which names' lists the follow-up hook would inject. Before building a picker that sees the follow-up question, we need its ceiling. bench/followup_picks.py replays the v13 and v14 laya-codex sessions against the benchmark checkouts. It bounds what any picker could remove: an exhaustive oracle, a perfect picker of at most k names, and six rules without a model, all under character budgets. It also prices a removed lookup call from the transcripts' usage (each token once) against the injection's cost at the measured 0.455 tokens per char. It keeps followup_completeness.py's coverage rule: a call is removable only if every tool use in it is covered. It adds three things: `search`'s name semantics (ported from mcp.rs), lookup scope (path, glob, type), and code snippets for Reads anchored at a candidate's use. It fixes three ways the old replay parsed Grep names. Tests show each one, red against the old extraction and green after: - `\bparse` was read as the name `bparse`; - `content-type` and `middleware/etag` were split into words; - a `fn test_` or `#\[test\]` Grep counted `test_` as a name. followup_completeness.parse_session now also records each API call's usage and the prompt's result usage. Its committed v13 output is unchanged, byte for byte. Tested by bench/test_followup_picks.py and bench/test_followup_completeness.py (python3 -m pytest bench). author: Tin Dang
The follow-up's lookups are the largest remaining lever on lookup round trips. The question was whether a picker that sees the follow-up question could inject the right callers and tests ahead of time. The pass bar was set before the replay: at least 50% of follow-up lookup calls removed, at no added cost (injection below calls removed x $0.005). bench/followup_picks.py on the v13 and v14 sessions: - An oracle that knows Claude's lookups removes 53% [44, 62] of them at 4k chars on all tasks, but only 40% [28, 52] on the held-out tasks. The planned shape (lists + at most 1 snippet) reaches 48%. - About 23% of follow-up tool uses ask for names Claude first thinks of on the follow-up. No injection made before it can hold them. Without that limit the same lists would cover 85%. - Cost was not the limit. A removed call saves $0.0093, so break-even is about 4.9k injected chars per call. The owner parked the work on 2026-10-07; don't retry without new evidence that the follow-up's names can be known in advance. The JSON holds every table in the section. author: Tin Dang
# Conflicts: # docs/RESULTS.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Records why model-picked follow-up injections were not built.
The idea: before Claude's follow-up lookup ("which tests cover this / who calls this"), inject the callers and tests that Claude would otherwise search for. Names come from Claude's first answer.
Result: an oracle that knows which lookups Claude will make covers 40% of held-out follow-up lookup calls at 4k chars (46% at 8k), against a 50% bar.
Contents:
bench/followup_picks.pywith tests. It also fixes three name-parsing bugs in the shared parser:\bparseread asbparse, hyphenated literals split into words, and test-only Greps counted as names.The 2026-09-30 replay still reproduces byte for byte. This PR will need
mainmerged in after the measurement PR, because both edit the top of RESULTS.md.