Open-source research software. Licensed under the MIT License.
diff is a local evidence and completion-gating layer for coding-agent work.
Its thesis is deliberately narrower than "agent evaluation":
Evidence is only valid for the state, verifier, scope, and human review surface it actually observed. When any of those change, dependent conclusions become stale and should turn into explicit repair work.
diff does not assign a universal trust/productivity score. It records independent evidence surfaces, checks whether they still apply, and exports only findings that cite concrete observations.
Lead artifact for Pi / coding-agent runtime evidence, and the default correctness primitive in the closed minimal-tool portfolio. In the matched external SWE-smith ablation, +diff preserved baseline native success (8/10) while preventing all 8/8 applicable stale-verification false completions. Scope and Cage remain optional for different failure regimes; diff does not require them.
diff has one product job: decide whether recorded evidence still supports conclusions about the current artifact/workspace state.
evidence + current workspace
↓
DiffRuntime
fresh | stale | missing | scoped
↓
diff CLI
stable JSON output
/ \
/ \
Pi adapter Codex adapter
The product surface is intentionally small:
- library —
openDiff(repoPath)/DiffRuntime; - CLI —
diff workspace | run | status | gate | review | export; - adapters — privacy-minimized Pi/Codex trace normalization;
- research — case runners and formal-conformance experiments live under
research/, notsrc/.
The deep evidence modules remain internal implementation details. Callers should not assemble ledger, freshness, review, workspace, and assessment modules themselves.
Pi / Codex trace ─────────────┐
│
Git workspace fingerprint ────┤
│
verifier receipts ────────────┤
├─> typed, locked, fsynced journal
human review checkpoint ──────┤ │
│ ▼
read/search/edit evidence ────┘ evidence contract
│
fresh / stale / missing / scoped
│
repair instructions
│
supported findings
Current implementation includes:
- Pi adapter for structural Pi session JSONL observations;
- Codex adapter for
codex exec --jsonevent streams; - split workspace identity: checkout/branch instance identity plus content/artifact identity;
- strict compiled contracts with exact contract and verifier-spec identities;
- state-bound verifier receipts with command, expectation, scope, stdout/stderr digests, declared verifier-input digests, and before/after workspace identity;
- stale-evidence invalidation after later workspace changes;
- verifier-integrity checks when declared tests/check definitions change between baseline and final;
- scope checks distinguishing targeted and full verification;
- human review checkpoints bound to the exact workspace/patch reviewed;
- commit-message capture as a human statement, never as proof;
- context/read-before-edit evidence and explicit impact candidates;
- labeled retrieval metrics only when a gold relevant-file set exists;
- typed, hash-chained, serialized and fsynced local evidence journal under
.diff/; - single assessment authority for evidence admission; export only serializes admitted findings;
- small
DiffRuntimepackage facade so callers do not assemble core modules themselves; - actionable repair guidance for missing/stale evidence;
- evidence export where findings must cite observation IDs.
npm install
npm run build
# In a Git project you want to study:
node /path/to/diff/dist/bin/diff.js init diff.yaml
# Establish the expected broken state.
node /path/to/diff/dist/bin/diff.js run diff.yaml baseline
# Optional context evidence an agent/human can record.
node /path/to/diff/dist/bin/diff.js observe read src/foo.ts
node /path/to/diff/dist/bin/diff.js observe read test/foo.test.ts
node /path/to/diff/dist/bin/diff.js observe edit src/foo.ts
# After implementation:
node /path/to/diff/dist/bin/diff.js run diff.yaml final
node /path/to/diff/dist/bin/diff.js gate diff.yaml
# If review is required:
node /path/to/diff/dist/bin/diff.js review diff.yaml accepted "Reviewed exact final patch."
node /path/to/diff/dist/bin/diff.js gate diff.yaml
# Optional: preserve what the commit says as a statement.
node /path/to/diff/dist/bin/diff.js statement commit HEAD
# Export observation-backed findings.
node /path/to/diff/dist/bin/diff.js export diff.yamlThe package surface is intentionally small:
import { openDiff } from "diff-research";
const diff = openDiff(repoPath);
const contract = await diff.loadContract("diff.yaml");
await diff.run(contract, "final");
const { assessment } = await diff.assess(contract);
if (!assessment.complete) {
console.log(assessment.gaps);
}DiffRuntime owns composition of workspace observation, evidence journal access, assessment, review, and export. Callers do not need to assemble the internal modules.
A gap looks like this conceptually:
stale-verification
Final verifier "acceptance" covers an older workspace,
not the current workspace.
Repair:
Re-run npm test on the current workspace.
The point is not to stop the agent permanently. The point is to make the missing evidence legible enough that the agent or human can repair it.
See diff.example.yaml.
A contract can require:
- a failing baseline reproduction;
- a passing final verifier;
- full or targeted scope;
- stable declared verifier-input files;
- exact current human review;
- read-before-edit evidence;
- explicit required/impact context files.
Research runners live under research/cases/; measured fixtures/results remain under cases/.
Every case follows:
problem → method → measured observations → solution behavior → limitations
| Case | Result |
|---|---|
| 0001 — adapter fidelity | Pi/Codex structural normalization with content omission |
| 0002 — failed real task | failed unauthenticated Codex run preserved without false attribution |
| 0003 — authenticated agent fix | Codex produces independently verified one-line fix matching reference patch |
| 0004 — state-bound verification | later edit invalidates green verifier; rerun repairs |
| 0005 — verifier integrity/scope | changed verifier inputs or narrower scope block equivalence |
| 0006 — human review freshness | later edit invalidates review; re-review repairs |
| 0007 — retrieval/impact | recall only with labeled gold; live context uses coverage |
| 0008 — closed-loop gate | missing review blocks completion; review repairs; export succeeds |
| 0009 — code hardening | strict contracts, exact verifier identity, typed durable journal, correct receipt pairing, split workspace identity, deep runtime facade |
| 0010 — invariant mutation | focused deep-module tests kill 23/23 compile-valid invariant-weakening mutants |
| 0011 — evidence-validity matrix | 12/12 predefined multi-file evidence-validity scenarios receive the exact expected completion/gap classification |
| 0012 — formal conformance bridge | real fresh/stale/repaired verifier traces independently satisfy the shared Haskell freshness semantics |
| 0013 — freshness policy ablation | conservative hybrid scores 14/14 with 0 false accepts and 0 false invalidations on the labeled policy matrix |
Full index: docs/CASES.md.
Run the complete local research suite:
npm run research
# Focused deep-module mutation suite only
npm run mutateThe unit/integration suite currently covers:
- canonical deterministic hashing;
- evidence-envelope tamper detection;
- finding-to-observation referential integrity;
- Pi/Codex structural redaction behavior;
- Codex command-exit/file-change extraction;
- strict contract parsing and compiled verifier identity;
- hash-chain tamper detection;
- typed payload rejection and verifier-receipt self-consistency;
- concurrent journal append serialization;
- workspace/artifact identity changes and checkout-path separation;
- untracked symlink non-dereference;
- verifier result measurements;
- stale verification;
- stale human review;
- read-before-edit/context coverage;
- labeled recall;
- commit messages as statements.
Current suite: 44 tests, 44 passing.
Case 0008 exercises the live CLI/gate/export loop end to end. Case 0009 adversarially tests the hardened contract, journal, assessment, export, and workspace-identity boundaries. Case 0010 then mutation-tests those choke points: 23/23 compile-valid mutants killed, 0 survivors. Case 0011 adds a predefined multi-file evidence-validity population: 12/12 scenarios matched the exact expected completion status and gap-code set. Case 0012 projects real implementation freshness state into the shared formal vocabulary; fresh/stale/repaired traces are independently accepted by the Haskell conformance checker. Case 0013 closes the freshness-policy question on a labeled 14-scenario matrix: exact-state has no false accepts but four false invalidations, bare dependency-aware has four false accepts under incomplete coverage, and conservative-hybrid scores 14/14 with 0 false accepts / 0 false invalidations.
Observability answers:
What happened?
diff asks an additional question:
Does this evidence still support this conclusion about the current artifact?
A test that passed before a later edit is real history but stale proof. A review of an earlier patch is a real human action but stale approval. A full-suite requirement is not satisfied by a targeted run. A passing final test is not automatically equivalent to a failing baseline if the verifier itself changed.
See docs/PRIOR_ART.md and docs/SOURCES.md. Prior art explicitly includes Backcheck, grounded, no-vibes, Braintrust, Agent Retrieval Bench, SWE-Explore, CodeScaleBench, Hunk, OpenTelemetry, and OpenAI's published harness/auto-review work.
- Architecture
- Problem statement
- Method
- Quality model
- Measured observations
- Correctness and limitations
- Prior art
- Sources
- Invariant mutation testing
The Pi/Codex normalization adapters omit prompt text, model prose, command strings, command output, reasoning text, and filesystem paths where the structural study does not require them.
Live .diff/ledger.jsonl is local project data. It can contain:
- workspace fingerprints;
- verifier metadata;
- repo-relative discovery paths explicitly recorded through
diff observe; - human review notes;
- commit messages explicitly captured through
diff statement commit.
Raw case artifacts are only retained when a case needs them for reproducibility.
This repository is independent open-source research software. The Pi and Codex integrations are compatibility adapters; this is not an official project of either runtime.
Before tagging a release, review retained raw case artifacts for credentials/private data, re-check pinned upstream revisions where relevant, and add a citation/security policy if the repository begins accepting broader external use or contributions.
No novelty claim should be inferred from this repository. The goal is a precise, reproducible mechanism with explicit evidence boundaries and clear prior-art positioning.
The software is available under the MIT License; see LICENSE.
The current bounded diff thesis is closed. The default policy remains exact-state freshness unless dependency coverage is independently established complete, in which case the conservative-hybrid relation may safely reuse disjoint evidence. Future changes should be motivated by composition evidence rather than additional standalone mechanism scope.
MIT. See LICENSE.