Conversation
…le canary Refs #414. Adds adapters for pi and the Devin CLI that report per-turn usage, transcript, and loaded context, with host configuration kept out of the run (no context files or extensions for pi; an isolated HOME, no cross-tool rules, and denied MCP tools for Devin). Runs now fail a canary when instruction files exist above the workspace, the default output directory moves outside the repository, and results record whether network off was checked. Updates the matrix example and the benchmark contract.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #414. Makes the benchmark runnable with pi (GPT, Gemini) and the Devin CLI, and closes a context leak found while doing it.
Adapters (
benchmarks/agent/adapters/), sameBENCH_*contract as the Claude Code reference:pi.py:pi -p --mode json. Disables context files, extensions, prompt templates, themes, and skill discovery, then loads the guide with--skill. Reads per-turn usage frommessage_endevents, the context limit frompi --list-models, and loaded skills from the system message. Gemini needsBENCH_PI_EXTENSIONS(thepi-antigravityprovider is an extension);BENCH_PI_WEB_EXTENSIONS(for examplepi-web-access) is loaded only for knowledgeweb.devin.py:devin -pwith an isolatedHOMEthat holds only the Devin credentials, a config withread_config_fromoff, MCP tools denied, web tools denied unless knowledge isweb, and the guide installed as a workspace skill. Usage and transcript come from the ATIF export. Built-in skills are ignored; account-managed plugin skills are listed apart fromloaded.Context leak fixed. Devin loaded the user's global
CLAUDE.md,~/.agents/skills, and the repository'sAGENTS.md, because agents discover instruction files by walking up from the workspace, and the default output directory was inside this repo. Now:invalid(--no-ancestor-checkdisables it);--outis~/.cache/wright-agent-bench;networkEnforcement(canary-checkedordeclared-only), since pi and Devin shells can still reach the network;.agentsand.devinare ignored as unsafe edits, because skills are installed there through the agent's own mechanism.Docs:
docs/agent-benchmark.mdstates that the scenarioprompt.mdis the only text the harness gives an agent (no system prompt or hint added), lists the adapters, and explains how to set the model in--agent-cmd, absolute paths, and--env-pass HOME.matrix.example.jsonnow lists pi (GPT, Gemini) and Devin.Verification: 25 tests pass (5 new for adapter parsing, plus the ancestor canary), with the oracle installed. Real single trials, not results: pi with
openai-codex/gpt-6-lunaonunderstand-score-flow(usage, context share of the 272K window, and the guide were recorded); Devinswe-2-maxonrepair-runaway-loopwith and without the guide (both valid, loaded context[wright]and[], and Wright used only with the guide); pi withantigravity/gemini-3.6-flashanswered once with the extension loaded (adapter not run end to end for Gemini).Not verified: network
offenforcement (acurlfrom my session fails even without a sandbox, so I could not tell a real block from my own network); Devin--sandbox(available throughBENCH_DEVIN_SANDBOX=1, untested on scenarios); thewebcells for pi and Devin.