context-report

context-report is an open, signed report format for one question: does this agent context artifact actually work?

The problem

A plugin, an AGENTS.md, a skill, a hook, an MCP server — every catalog ships them, and none come with evidence attached. Nobody records whether the artifact reaches the agent at all, how it fails when it can't run, or what it costs in latency and context tokens, and whether the artifact's own instructions change what the agent does is rarely checked at all. context-report is a predicate an author's CI produces and a catalog verifies at submission — one row per fact, a basis declaring whether the row is recomputable or only claimed, and never a "pass"/"fail" for the artifact as a whole (the consumer sets its own thresholds).

30-second quickstart

pip install context-report            # 0.1.0 on PyPI; pip install -e ".[dev]" from a checkout to hack on it
context-report produce --subject ./my-plugin --kind plugin --target claude_code --n 20 \
  --out report.json                   # one statement: reachability, cost and fault rows for one target
context-report run run.json           # a whole manifest: subjects x models x tasks in one shot
context-report compare out --history  # every run of that manifest side by side

The optional context-report[efficacy] extra pulls in the anthropic client for the efficacy row (context-report efficacy --help). context-report run reads a JSON manifest matching spec/run/v0.1/schema.json — see spec/run/v0.1/examples/run.json for a worked one (two subjects, four models across the three providers, three tasks) — and supports --dry-run (rules and call budget, no model touched), --n (override arms.nPerArm for a smoke run), and --resume (continue the latest run, reusing every existing statement and matching transcript, calling only for the rest). Two providers have a backend: anthropic (the API) and claude-cli (the local claude CLI, so one manifest can compare opus/sonnet/fable); any other subject model gets an honest NotAvailable efficacy row instead of a guess.

What a report looks like

Trimmed from a committed statement over a real public plugin (paper/measurements/catalog-sample/official/ai-plugins.json):

{
  "subjectKind": "plugin",
  "target": {"name": "claude_code"},
  "attributes": [
    {
      "attribute": "cost.context_tokens",
      "basis": "re-derivable",
      "result": "PASSED",
      "inputHash": "sha256:4e85a09a2014600e...",
      "conditions": {"tokenizer": "approx-regex-v1", "files": 2},
      "measurement": {"unit": "tokens", "n": 1, "mean": 2289}
    }
  ]
}

v0.1 rows: conformance · reachability · decision · fault.scriptMissing · fault.interpreterMissing · fault.timeout · fault.malformedOutput · cost.latency_ms · cost.context_tokens · interference · efficacy (extensions use an x- prefix). A row that could not be measured says NotAvailable, Error or NotApplicable and why — never a silent pass. v0.1 draft: schema at spec/attestation/v0.1/schema.json, worked example at spec/attestation/v0.1/examples/plugin-copilot.json, predicate type https://open-coder-ai.github.io/context-report/attestation/v0.1, hosted at https://open-coder-ai.github.io/context-report/attestation/v0.1/.

Three measurements

The measurement paper ran the reference producer over chock's 88 bundles, a sample of 18 public Claude Code plugins, and seven third-party instruction files and skills. Three findings from that run:

Reachable is not the same as executable. Of 18 public plugins, three (carta-cap-table, carta-crm, carta-investors) share a dispatch script with no execute bit — reachability FAILED, 0 of 4, exit 126. Every hook that runs, across both samples, allows on malformed input. See §5.2.

Cost spans two orders of magnitude. Hooks that shell out to npx cost 916.8–941.5 ms p50; a local script costs 7.3–53.9 ms. Context weight varies about a hundredfold across the sample, roughly 1,500 to 147,000 tokens. See §5.2.

No efficacy row reaches PASSED. Three instruction files, ablated on opus, sonnet and fable (168 recorded transcripts, one judge model held fixed): with four observations per arm the 95% interval is about ±0.49 wide, and the row reports the interval instead of rounding it to a verdict. A naming-convention rule was the one consistent positive (+0.25 to +0.50 on every model); a prompt-injection rule moved nothing on any model. See §5.3.

Who it's for

• An artifact author wants a report their own CI can produce before anyone else asks for one.
• A catalog maintainer wants a submission format their existing verifier can check without adopting anyone else's test suite, and a re-derivable/claimed split to build a policy on.
• A researcher or reviewer wants a re-derivable record of what was actually measured, not a vendor's prose description of it.

Every model you can reach

Subject models come from the manifest, never from code: anthropic, claude-cli, or openai-compatible with a baseUrl, which is any server speaking the chat-completions shape, hosted (OpenAI, Gemini, Mistral, Groq) or local (Ollama, vLLM, LM Studio). One manifest lines up every model you can reach; the API-shaped ones answer without tools or a checkout, which the run spec states.

Two models, not one

Efficacy needs two roles, never one: the subject model runs a task with the rule prepended and without it; the judge model never performs the task, only reads the transcript and decides whether that arm met the rule's criterion, held fixed across every subject model so a comparison across models is fair. A machine-checkable criterion is graded by code instead, never guessed at.

Use as a library

Beyond the CLI, context_report exposes a small stable API for a catalog or CI job to import directly: validate, verify, produce_statement, load_manifest, run, resolve_run_dir, history_markdown, rule_history_markdown, render_table, render_history (see __all__ in context_report/__init__.py).

from context_report import validate, verify

errors = validate(stmt)  # schema errors, [] means well-formed
result = verify(stmt, subject_path="clone/")  # bound + schema check, never a verdict

See docs/library.md for a full catalog-verification and CI-production example.

Contributing

Bug reports, spec feedback, and PRs are welcome — see CONTRIBUTING.md for the development loop and the DCO sign-off every commit needs. Discussion, spec proposals, and reports of your own runs happen in GitHub Discussions. See SECURITY.md to report a vulnerability privately.

Scoped starting points, each naming the file it lives in, are listed under Good first contributions: another target agent's payload shape, codex_cli's documented fault behaviour, a producer that drives a live client for the fault rows, decision replay, interference measurement, a real tokenizer behind a new method value, leave-one-out arms, and another instruction-file sample for the paper's measurements. Comment on a good first issue to claim it, and keep the Co-Authored-By trailer if an agent helped — every diff is read in full before merge either way.

Part of the open-coder-ai family

Everything under open-coder-ai is built on one rule: a claim must match a mechanism. Where this repository sits among the others:

Repository	What it is
chock	The framework: write a policy once, enforce it on git hooks, CI, and every agent
chock-catalog	The policies, each graded by what it actually enforces
agentseam	The primitives layer under chock: one handler API over every agent's hooks, with a capability matrix that carries its provenance
chock-threat-intel	A weekly, human-reviewed threat digest scored against the catalog
chock-claude-plugins · copilot · cursor · codex	The catalog compiled into each client's native plugin format; generated only, rebuilt and diffed in CI
chock-quickstart · chock-example	Template repositories: exactly what chock init leaves behind, and a working adoption with one policy per layer

License

Apache-2.0.