Repository navigation
ADD 4.1.0: measured skill changes, --effort medium, and the full SWE-bench Lite trade-off - #232
Merged
Merged
Conversation
The meter hard-coded `--effort medium` for every arm, so one question could not be asked: does ADD's ceremony at low effort hold the quality raw Claude Code needs medium for, on the same model? Effort now resolves like model — the arm's `effort`, else `run-all --effort`, else medium — is validated at load time (low|medium|high|xhigh|max) and stamped into artifacts.effort. Two new arms: add-4-low (shipped ADD 4.0 at low) and add-4-audited-low (the prompt-audited SKILL.md variant at low: numeric turn counts out, stub only what the checks import, personas picked by grep at their installed `.add/` path). Control arms keep the medium default. Tests: benchmark/tests/test_arm_effort.py (red first, 9 cases); suite 540 pass. author: Tin Dang
Round 7 put ~175 s of each ADD run's ~200 s of test-command wall time outside pytest itself (pytest reported 19-31 s per run): Verify started the app with `python3 -m app &` to probe it by hand, and an unkilled or crashed server held the Bash call open until it was stopped or timed out at 120 s. add-4-probe-low = the audited variant plus one Verify line: probes are test files in `.add/tasks/<slug>.probes/` run by the test command; an app the agent must start logs to a file and is killed in the same command. Low effort. Tests: test_arm_effort.py +1 (red first); suite 541 pass. author: Tin Dang
…sions
A live stall watch on round 8 caught the operator's `security-guidance`
plugin: its PostToolUse hook runs an Opus security review on every
`git commit`, holding the agent's Bash call for 100-185 s and billing
outside the run record. ADD commits 2-4 times per run (freeze, build,
verify); vanilla never commits — so the operator's plugin was measured as
ADD's wall time in every round since 4.0. The user CLAUDE.md also reached
every arm ("interview me until 95% confidence" in a headless run).
`--setting-sources project,local` drops user plugins, hooks and
~/.claude/CLAUDE.md (verified: plugins 17 -> 4 org-managed, context
35k -> 24k tokens, the user rule no longer quotable). Organisation-managed
plugins still load, equally for every arm.
Tests: test_arm_effort.py +1 (red first); suite 542 pass.
author: Tin Dang
The runner still installed ADD 2.0 (`.add/tooling/add.py`, `--force`), drove the retired engine, pinned `--effort medium` for both arms and loaded the operator's user settings. Now: - add arm: the ADD 4.0 installer only; the prompt drives `.claude/skills/add/SKILL.md` (freeze/verify commits, a reproducing check, host tests as the regression floor, headless — no human) - ARM_EFFORT: add=low, vanilla=medium on claude-sonnet-5-5, via the wm bench's own argv builder (so --setting-sources project,local applies) - collect_patch: `git add -A` + `diff --cached <base>`, so a NEW file the fix creates is no longer dropped from the prediction - --sample N --seed S: a fixed slice of all 300 Lite ids; --workers N - each instance keeps transcript.jsonl (the trajectory a submission needs) - evaluation documented on Modal (x86) instead of local docker Tests: test_swe_smoke.py rewritten for 4.0 (red first, 7 new failing); suite 544 pass. author: Tin Dang
The per-id datasets-server /filter endpoint returned HTTP 500 through every retry; the paged /rows fetch that lists the 300 ids already carries each row, so it now fills instances.json and /filter is never reached. Tests: test_swe_smoke.py +2 (red first); suite pass. author: Tin Dang
…ator-config finding Round 8 (isolated, sequential) supersedes round 7: the operator's security-guidance plugin reviewed every git commit (ADD-only cost) and the user CLAUDE.md made vanilla halt on amb1. Verdicts: ADD at low holds and pays on test strength (amb1 mutation 0.83 vs 0.19) at 1.3-1.9x cost; low effort is the cheaper default; ship only the audit's path fix; the probe rule is not needed. author: Tin Dang
…vanilla-medium 21/30 ADD resolves a strict superset (+2: scikit-learn-14087, sympy-14817) at 2.7x cost ($0.33 vs $0.13 per resolved); McNemar p=0.5, directional. Scored with the official harness locally (swebench 5.0.2, x86 images under Rosetta) since every swebench release's Modal path is broken. author: Tin Dang
…s only where the spec is silent, contradictions in the caller's favour, honest README price author: Tin Dang
…nd the benchmark variant tests Both pinned their checks to artifacts this task moves on purpose (the 2026-09 results page; the live SKILL.md). They now read the cited results page and a 4.0.0 base snapshot — same assertions, stable sources. author: Tin Dang
… silences, contradictions in the caller's favour - floor = a change of shape to a consumed surface, not a touch; a fix that restores intended behavior without changing its shape is not floor work - a Quick commit body records lane + red->green evidence (an artifact) - ASSUMPTIONS: settled dimensions get no line, '- none — <why>' is valid; contradictory requirements -> the caller-in-control reading, first - personas index path is the installed .add/personas-index/use-when.md - READMEs quote round 8's isolated price, recommend --effort low, name the security-guidance contamination; new results page backs every number - SWE runner: the ADD prompt lets the skill size the work - compensations to stay at 200 lines: two persona/report glosses dropped (still in references/personas.md), 'performance' out of the risk examples list - guard tests read the cited results page / a 4.0.0 base snapshot Tests: add-method 160 pass, benchmark 547 pass. author: Tin Dang
…utation claim
Round 9: the caller-in-control reading ('reject') left amb1's waitlist
requirements 4-6 with nothing to act on — ambiguities handled 5.3 -> 4.0,
one oracle 0.88. Round 9 vanilla scored 0.67 on amb1 mutation, so round
8's 0.83 vs 0.19 overstated the gap; the READMEs quote rounds 8-9 pooled.
author: Tin Dang
…utation gap Round 9 measured the caller-in-control reading as net harmful on amb1 (ambiguities 5.3 -> 4.0, one oracle 0.88). The READMEs and results page now quote rounds 8-9 pooled (amb1 0.81 vs 0.43), disclosing that round 9's ADD arm ran the reverted draft. Tests: add-method 159 pass, benchmark 547 pass. author: Tin Dang
Seal intact since refreeze 0a7fa83; check 5+22 pass, regression 159+547 pass, three refute probes pass on a fresh install. Quick lane on bounded fixes: SWE Lite cost -35% (2.7x -> 1.8x vanilla), 30/30 patches with tests; resolved 23 -> 21 of 30 (within noise). Open risk owned by Tin Dang: settle with the full 300-instance run. The commit-body artifact (M2) did not transfer; the contradiction rule (M4) was reverted as harmful. author: Tin Dang
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| from benchmark.check_isolation import find_leaks | ||
| from benchmark.check_isolation import main as check_isolation_main | ||
| from benchmark.runner.agent import PINNED_MODEL, build_argv, resolve_model | ||
| from benchmark.runner.agent import PINNED_MODEL, build_argv, resolve_effort, resolve_model |
… the model never does author: Tin Dang
…d found: step; refute for security work Yield audit of rounds 8-9 (24 task files) and the SWE-bench pilots: falsifier-per-check followed 24/24 and carries the mutation gain; found: 0/24; refute probes 1/24 with 24/24 PASS; Quick tests 1.3 asserts/fix and the 3 lost SWE issues were wrong fixes. READMEs: a measured row for tests run (30/30 vs 7/30) and shipped (30/30 vs 2/30). Tests: add-method 163 pass, benchmark 547 pass. author: Tin Dang
…2x2 effort sweep author: Tin Dang
Round 10 runs the final value-final skill and raw Claude Code at --effort low and medium on claude-sonnet-5-5, isolated with --setting-sources project,local. wm1 and amb1 use n = 3 per cell. SWE-bench Lite uses the same 30 instances (seed 0), scored locally by the official harness (swebench 5.0.2). SWE resolved: vanilla-low 21, vanilla-medium 20, ADD-low 24, ADD-medium 22. ADD-low's set contains every vanilla-medium resolve plus 4 more. Cost per resolved issue is $0.23 for ADD-low against $0.14 for vanilla-medium. Medium effort bought neither tool anything. The results page, the SWE pilot doc, both READMEs and the CHANGELOG now carry the grid. author: Tin Dang
Seal intact since freeze 6fc12fe. Check: 4 tests pass. Regression: 163 + 549 pass. Round 10 re-measure, on the same 30 SWE-bench Lite instances: ADD-low resolves 24/30, a superset of vanilla-medium's 20, at $0.23 per resolved issue. That recovers the lean skill's 21 at a lower cost than 4.0.0. On wm1 and amb1, ADD-low leads on ambiguities (5.3 vs 4.3) and on mutation, and correctness ties. author: Tin Dang
…rded fallback when the suite cannot run author: Tin Dang
… no suite runs Quick now greps every other caller or emitter of what it changes and records each site as fixed or unaffected. With no runnable suite it runs the issue's repro as written, or imports the touched files and traces the inputs, and records suite: unavailable. Verify never passes a check it could not run. Compensating cuts keep SKILL.md at 200 lines. author: Tin Dang
…cial eval image A bare clone left both arms unable to run tests on 9 of 30 instances (unbuilt matplotlib, Django and sklearn setups), so fixes shipped unexecuted. The agent now edits a host copy of the image's /testbed. Shims for python, pytest and pip route calls inside it to the container, mounted at /testbed and at its host path. ZDOTDIR keeps the shims first on PATH, because the agent's shell otherwise sources the operator's profile. The patch is diffed against the image HEAD, and caches are filtered out. A live smoke resolved under the official harness. author: Tin Dang
…la 21/30 With the eval image's env available to both arms, ADD at low resolves 23 of 30 against vanilla-medium's 21. It costs $0.16 against $0.12 per resolved issue, down from $0.23 in round 10, and 27 of 30 ADD runs verified on a real green (round 10: 15). The sites: record transferred in only 2 of 30 runs. Losses against round 10 are re-readings with 0 existing tests broken. author: Tin Dang
Seal intact since 68e0f39; check 4 pass; regression 167 + 556 pass. Round 11: ADD-low 23/30 vs vanilla-medium 21/30 at $0.16 per resolved issue. The sites: record transferred in 2/30 runs and did not flip django-13265; owner Tin Dang cuts or reworks it if yield stays <= 2/30. author: Tin Dang
… and end the reply with the record author: Tin Dang
…s its reply with the record Quick now reads the nearest tests and sibling code before writing its test. The fix keeps their conventions: inputs not mutated, parallel paths consistent, the neighbours' pattern. The fix the request literally suggests is treated as a candidate wrong fix. The Quick record (lane, intent, sites, red->green) now ends the reply, which is always read, rather than living only in a commit body. Non-code Quick work uses the check that fits. Direction grounds in the nearest tests too. Compensating cuts keep SKILL.md at 200 lines. author: Tin Dang
Three cells of the same arm running one instance at the same time collided on swe-<arm>-<iid>, and the second docker run failed with 'testenv start failed'. author: Tin Dang (cherry picked from commit 200a480)
Three cells of the same arm running one instance at the same time collided on swe-<arm>-<iid>, and the second docker run failed with 'testenv start failed'. author: Tin Dang
The seal is intact since 1f8c624. Regression: 173 + 557 pass. On the 22 issues only vanilla resolved, the intent skill at low recovered 13, against 11 for the same-skill rerun, at +5% cost with 10/10 controls kept and no tests broken. Directional at n = 22; owner Tin Dang confirms on a full rerun. author: Tin Dang
…ade-off, loss included author: Tin Dang
…uded All 300 issues on Sonnet 5.5 in the official eval environment. Vanilla Claude Code at medium resolved 215; ADD at low resolved 201 (McNemar p = 0.016). ADD cost 1.4x the dollars and 2.0x the seconds per issue. ADD saw a passing test run before shipping in 255/300 against 42, and shipped a test with 298 against 14. The results page leads with the 300: per-repo split, AI effort, review proxies, the 4-way diagnostic and a Not measured list (human review time, contamination-free issues, seeds). The READMEs replace the 30-issue claim and drop the plugin-contaminated 4-5x speed figure. author: Tin Dang
The seal is intact since b83577a. Check: 8 pass. Regression: 177 + 557 pass. The front door states the 300-issue loss (201 vs 215, p = 0.016) beside what ADD buys, and names human review time as not measured. author: Tin Dang
…stem prompt At --effort low the ADD arm read SKILL.md in only 134 of 270 full-300 runs, so the method was half-applied. --inject-skill appends the installed skill with --append-system-prompt, and the prompt says the skill is already loaded. Without the flag, the argv and prompt are unchanged. author: Tin Dang
ADD at medium effort with the maintainer-intent skill resolves 83 of 101 dev issues, against 80 for a fresh vanilla-medium run (+7/-4, p = 0.55). At low effort it resolves 76. Skill injection at low (70), intent-lines-first (81) and fix-first (76) are dropped. The held-out confirming run is under way with the owner's OK. author: Tin Dang
… and recommends medium author: Tin Dang
… recommend medium Both arms at --effort medium, fresh and side by side, official eval env. Results: ADD resolves 226 of 300 against vanilla's 215 (24 vs 13 one-arm-only, McNemar p = 0.099). On the held-out 199 it is 143 vs 135. Price: 1.8x the dollars and 2.5x the seconds per issue. Test discipline: a passing test run before shipping 274 vs 53, tests shipped 299 vs 17. The READMEs now recommend --effort medium. At low effort the skill was read in only half the runs and resolved 201. The low-effort section stays on the results page as history. author: Tin Dang
The seal is intact since a75fcbd. Regression: 181 + 560 pass. The front door states the fresh 300 at medium (226 vs 215, p = 0.099) with its 1.8x price and recommends --effort medium. author: Tin Dang
…anilla Each row is a value the user gets: fixes that land, verified before it ships, ships with a test, tests that catch bugs, guesses you can review, claims you can trust and correctness, plus the price rows. Each has vanilla's number beside ADD's. The numbers are unchanged and trace to the results page. lane: quick — README table restructure, same numbers, owner request 2026-10-07 red→green: add-method/tests/test_value_side_by_side.py · suite: cd add-method && python3 -m pytest -q → 182 passed author: Tin Dang
…plain-words README opening author: Tin Dang
…rds README opening docs/add-results.html tells the 300-bug result for someone who has never seen a benchmark. It covers bugs fixed (226 vs 215), checking before handing back (274 vs 53, as 300-dot grids), shipping a test (299 vs 17), unclear requests, the price (1.8x money, 2.5x time) and the effort setting. It says in words what is not proven and what is not measured. The page is self-contained, respects reduced motion, works on a 390 px screen and has a dark theme. Every number comes from an embedded data block that traces to the results page. Both READMEs now open with an ADD in plain words table linking it, and the docs index links it. author: Tin Dang
The seal is intact since 35135e2. Regression: 187 + 560 pass. Rendered at 1200 px and 390 px with no overflow. The read-through for overclaims and jargon found two issues, both fixed. author: Tin Dang
…ts come first The page now opens with 'Why write the plan and the tests first?'. Two flows light up step by step: Request, Code, Done against Request, Plan, Failing test, Code, Passing test, Evidence. Four plain reasons follow, each with its measured line: gaps get filled either way (3 of 3 vs 0 of 3), a test written first proves the fix (274 vs 53), a saved check can't be quietly weakened (81% vs 43% of planted bugs caught), and the next session can read it (299 vs 17 shipped with a test). ADD's colour is now turquoise in both themes. lane: quick — one page plus its guard, owner request 2026-10-07 red→green: add-method/tests/test_results_why.py · suite: cd add-method && python3 -m pytest -q → 189 passed author: Tin Dang
…rite, keep the grep author: Tin Dang
The model wrote the sites: record in 2 of 30 measured runs and it did not flip its target, so the record is cut (SKILL.md, Learn: a zero-yield control is cut or fixed). Quick still greps every other site that calls or emits what it changes. author: Tin Dang
The seal is intact since 02c1c6d. Regression: 192 pass. This closes the open risk that blind-spots accepted. author: Tin Dang
Bump the eight version declarations (pyproject, __init__, package.json, package-lock x2, plugin.json and three SKILL.md trees) from 4.0.0 to 4.1.0, and cut the CHANGELOG entry. 4.1.0 carries the skill changes measured in rounds 8-11 and the full SWE-bench Lite runs: the shape floor, the Quick falsifier, the no-suite fallback, maintainer-intent, and a Quick record without sites:. It also carries the --effort medium recommendation, the fresh-300 result (ADD 226 vs vanilla 215) and the plain-language animated results page. Not tagged: the owner tags v4.1.0 after the review study and PR approval. author: Tin Dang
TinDang97
changed the title
The results page is now one insight headline and one chart per scene: the two flows, bugs fixed, checked before done, shipped with a test, the request gap, the price, and the effort setting. Explanatory paragraphs and the which-to-use cards are gone; the not-measured line stays. In the README, the price bullet says it once, the older-round paragraph becomes a link to the results history, and the fine print is three sentences. lane: quick — docs page and README wording, same numbers, owner request 2026-10-07 intent: guards pin the honest phrases (not proven, costs more, not measured) → kept in headlines and notes red→green: existing guards add-method/tests/test_plain_results.py, test_results_why.py · suite: cd add-method && python3 -m pytest -q → 192 passed author: Tin Dang
One reviewer (the project owner) judged six SWE-bench Lite issues blind, by interview and untimed, from the medium-effort runs. They preferred vanilla's output in 6 of 6 and found it easier to decide on in 6 of 6. 2 of the 5 vanilla patches approved as is fail the benchmark's tests; the reviewer's verdict agreed with the tests in 3 of 6 for vanilla and 5 of 6 for ADD. The result goes against the claim that ADD's artifacts save review effort, so it is published as such on the results page, both READMEs, the 4.1.0 changelog entry and the animated page (new scene 7). It is labelled anecdotal: one reviewer, six issues, no timing. Timed review stays listed as not measured. lane: quick — docs only, no shipped behavior red→green: tests/test_review_study.py · suite: pytest add-method/tests → 196 passed author: Tin Dang
…evidence proves the rest
Adds a transcript-level reading of the fresh 300 to the results page. Both arms shipped the identical source fix in 114 of 300 issues (60% of 1-2 line fixes). 17 of ADD's 24 one-arm wins wrote the fix before the test, and 73 of its 74 unresolved fixes shipped a passing test. The sequence figures come from shell-command heuristics and are labelled approximate; this is the evidence behind the important-tests-only change, not an ablation. lane: quick — docs only test: none — the results-page guards already cover the page suite: cd add-method && python3 -m pytest -q → 202 passed author: Tin Dang
The ADD arm's prompt told every run to "start from a test that reproduces the issue and fails before the fix". That instruction came from the harness, not the skill, so it produced part of the 299-of-300 "shipped a test" figure and would have masked the important-tests-only Quick lane in any rerun. The sentence is removed; the regression-floor and never-weaken instructions stay. lane: quick — one sentence in one prompt red→green: test_add_arm_lets_the_skill_decide_on_a_test · suite: pytest benchmark/tests/test_swe_smoke.py → 35 passed author: Tin Dang
…fixes keep theirs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ADD at
--effort lowagainst vanilla Claude Code at medium, both on Sonnet 5.5, measured with the operator's config isolated. Acting on that measurement, bounded fixes now take the Quick lane.effort, recorded in every run record.--setting-sources project,local, so the operator's~/.claudestays out of measured sessions.--sampleslice,--workers, kept transcripts, and patches that include new files.security-guidanceplugin ran an LLM review on everygit commit, 100–185 s each. Only ADD commits, so only ADD paid it.CLAUDE.md("interview me") made vanilla halt on the ambiguous spec.lean-bounded-fixes, verify RISK-ACCEPTED)--effort low, and the pooled mutation gap.Results
Round 10: the 2×2 effort sweep (final
value-finalskill, verify PASS)ADD-low resolves every issue vanilla-medium resolves, plus 4 more. Medium effort bought neither tool anything. Page:
benchmark/results/2026-10-add-4.0-low-effort-vs-vanilla.md.Round 11: official eval environment +
blind-spotsskill (verify RISK-ACCEPTED)--testenv dockergives both arms the SWE-bench image's runnable env. ADD-low 23/30 vs vanilla-medium 21/30, at $0.16 vs $0.12 per resolved issue (ADD was $0.23 in round 10). 27/30 ADD runs verified on a real green (was 15). Thesites:rule barely transferred (2/30); the decision on it is owned and recorded.All 300 SWE-bench Lite issues: vanilla fixes more (published, loss included)
The 30-issue lead (23 vs 21) did not hold. Diagnostic on the 22 vanilla-only issues: an ADD rerun recovers 11 (noise), the new
maintainer-intentskill 13, ADD at medium 14, and vanilla re-resolves 18. Tasks:maintainer-intent(RISK-ACCEPTED),publish-300(PASS). READMEs, the results page and the CHANGELOG now carry the full trade-off. Human review time is listed as not measured.Fresh 300 at medium effort: bar met (2026-10-07)
--effort mediumChosen on a random dev third (
benchmark/GAP-TRACK-2026-10-06.md), confirmed on the held-out 199. At low effort ADD read its skill in only half the runs (201/300). READMEs now recommend--effort medium. Taskpublish-medium: PASS. Still pending before merge: the owner's 6-issue review study.All n = 3 or n = 30: directional, not significant. Details:
benchmark/PILOT-4v3-2026-10-02-r7.md(rounds 7–8)benchmark/SWE-LITE-PILOT-2026-10-03.mdbenchmark/PROMPT-AUDIT-2026-10-02.mdbenchmark/results/2026-10-add-4.0-low-effort-vs-vanilla.mdOpen risk
The Quick lane cut SWE cost 35%, but resolved went 23 → 21 of 30, within noise. A full 300-instance run (~$70) settles it.
Test plan
cd add-method && python3 -m pytest -q: 159 passpython3 -m pytest -q benchmark/tests: 547 pass, 12 skippedSKILL.mdis 199 of 200 linespilotspace-add initresolves the persona path the skill names