Add a skill behaviour check, and a lint for the cheap half - #20
Open
StefanoGuerrini wants to merge 9 commits into
Open
Add a skill behaviour check, and a lint for the cheap half#20StefanoGuerrini wants to merge 9 commits into
StefanoGuerrini wants to merge 9 commits into
Conversation
This was referenced Sep 9, 2026
StefanoGuerrini
changed the title
StefanoGuerrini
force-pushed
the
test/agent-discovery-eval
branch
from
September 9, 2026 15:14
fe53d66 to
9e01366
Compare
StefanoGuerrini
changed the base branch from
main
to
fix/protected-fetch-auto-mode-default
September 9, 2026 15:14
StefanoGuerrini
force-pushed
the
test/agent-discovery-eval
branch
2 times, most recently
from
September 9, 2026 15:22
9e01366 to
d09fc23
Compare
zenrows init sets up .zenrows/, but an agent only reads what its own harness loads. Nothing in the project tells it the CLI exists, so it reaches for a plain HTTP client instead. This eval measures that, in a container with no agent config, no plugins, no MCP servers and no zenrows binary, so anything the agent knows it learned from the arm under test. It scores two things per arm: whether the agent picks the CLI, and whether it cites a real cost rule, because an agent that finds the CLI and then enables js-render and premium-proxy together costs the caller 25 credits per request. The pass criterion is fixed here, before any implementation, so a candidate change either clears the bar or does not.
The first metric matched generic words like multiplier, so the untreated baseline scored 5/8 while every one of those answers opened with 'assuming a commercial scraping API, you didn't say which'. It measured whether the model can talk about scraping costs, not whether it knows ours. The marker set is now limited to things only this product has. Under it the baseline scores 0/8, and the gap between candidate wirings is visible.
A local tarball cannot be run through npx, so the candidate arm silently installed nothing and scored 0/8 on every metric, which is indistinguishable from the change not working. The CLI is now installed once after the cleanliness gate, and an arm whose init leaves no .zenrows aborts the run.
Two scoring fixes from the same run. The discovery pattern looked for 'zenrows <subcommand>' and missed 'Zenrows Protected Fetch, via the protected-fetch skill', which is how the agent phrases it, so a working build scored 0/8. The new costly-default metric counts discovery answers that reach for premium_proxy when nobody asked about cost. That pair with js_render is 25 credits per request, while mode=auto bills only for what succeeds. The failure survives a passing discovery score: the skills scored 8/8 on discovery and still recommended premium_proxy in 6 of 8 answers.
The costly-default metric matched any mention of premium proxies, so an answer saying 'no JS rendering or premium proxies on the first attempt' counted as reaching for them. It scored a correct run 4/8 when the true value was 0/8. An answer is now costly only when it names an escalation and never mentions auto mode. A metric that punishes the behaviour we want would eventually push someone to fix the wrong thing.
Discovery, the question that started this, needs no harness: run init and look for a SKILL.md outside .zenrows/. What does need one is skill content. Skills are prose shipped into other people's agents, no unit test can tell you the prose still steers the way you meant, and a version that scored 8/8 on being chosen still recommended a 25 credit configuration in 7 of 8 answers. Says plainly that it is not a CI gate, and what size of difference to believe.
Two deterministic rules over skills/*/SKILL.md, derived from a regression the agent-discovery harness found empirically: a runnable example that turns on both --js-render and --premium-proxy must state the cost on that line, and any skill showing an escalation flag must mention auto mode somewhere. The rules are narrow on purpose. A synopsis listing optional flags in brackets is documentation rather than a recommendation, and prose about escalating after a failure is the behaviour we want. This runs in milliseconds, needs no API key, and works on fork pull requests, so it belongs in CI in a way the harness itself does not.
Every warning that quoted a figure now names the direction and points at cost-control, which owns the multipliers. A price repeated in six files goes stale in six files. Also restores tests/skill-content.test.ts, which an earlier commit on this branch overwrote.
It was a candidate wiring we tested and rejected: a short note in CLAUDE.md naming the CLI. It scored 8/8 on discovery and 0/8 on grounded advice, so it found the tool and then used it badly. Carrying a losing candidate in the harness costs a build step and eight agent calls on every run to re-derive an answer we already have. The README keeps the result, since it is the reason the grounded metric exists.
StefanoGuerrini
force-pushed
the
test/agent-discovery-eval
branch
from
September 9, 2026 15:30
6774c30 to
5ef92d9
Compare
StefanoGuerrini
changed the title
StefanoGuerrini
marked this pull request as ready for review
September 9, 2026 15:31
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Skills are prose we ship into other people's agents. No unit test can tell you that prose still steers an agent the way you meant, and a skill that reads well can still cost customers money.
Measured: a version of
protected-fetchthat installed correctly, passed every test, and scored 8/8 on being chosen still told the agent to enable JS rendering and premium proxies in 7 of 8 answers to a question that never mentioned configuration. That is the most expensive configuration the API offers.Nothing else in the repo can catch that.
What this is
A manual pre-release check. Run it when you change something under
skills/, and compare against the build you shipped last.docs/contributing.mdgains a section saying so.It is not a CI gate. It needs an API key, a container, and about ten minutes. Eight runs of a language model is a smoke test with opinions, not a statistical result. The README says to treat a difference of one or two runs as noise, and 7/8 against 0/8 as real.
How it works
Three arms, built from a real
initin a container with no agent config, no plugins, no MCP servers and nozenrowsbinary, so anything the agent knows there it learned from the arm under test. Each arm is scored on what the agent picks, whether its advice is grounded in this product, and whether it reaches for an expensive configuration nobody asked for.It also answers a one-off question the arms make easy: whether a given wiring makes an agent aware of the CLI at all. That is what #22 uses it for.
Also here: a deterministic lint that does belong in CI
The harness cannot gate a merge. It needs an API key, so it fails on fork pull requests, it costs money and ten minutes, and eight runs of a language model will cross a threshold on sampling alone. A required check that goes red without a code change teaches everyone to bypass it.
So the lesson it found is also encoded as a plain unit test,
tests/skill-escalation-cost.test.ts, with two rules overskills/*/SKILL.md:--js-renderand--premium-proxymust state its cost on that line, or point atcost-controlIt runs in milliseconds, needs no key, works on forks, and it fails the build on the exact regression in #21. The rules are narrow on purpose: a synopsis listing optional flags in brackets is documentation rather than a recommendation, and prose about escalating after a failure is the behaviour we want. It matches any cost signal rather than a figure, so a price change does not break it.
The division is deliberate. The cheap deterministic rule blocks a merge. The expensive probabilistic one informs a human before a release.
Reviewing this
The scoring is the part worth reading, not the plumbing. Four of the nine commits fix a metric that produced a confident wrong answer:
The harness aborts rather than reporting a score when the agent cannot authenticate, when the CLI will not install, or when an arm's
initleaves no.zenrows/. Each otherwise scores 0 on every metric and reads exactly like a real negative result. All three happened.That history is the argument for trusting a number it prints, and for reading the raw answers it dumps under the table before you do.
Scope
evals/agent-discovery/, one unit test, and one section indocs/contributing.md. No CLI changes. Not wired intozenrows eval run: that runner executes API steps and needs no model.Stacking
Based on #21, because the lint fails against the skills as they are today, which is the point. Review #21 first, or read this branch's diff against it.