Coding agents can produce plausible fixes without first proving that they reproduced the reported failure. ProofBeforePatch is an evidence-first agentic debugging prototype that adds a hard gate:
NO VERIFIED REPRODUCTION → NO PATCH
Bug report
→ behavioral contract and hypotheses
→ executable experiments and observations
→ candidate reproduction
→ buggy-vs-known-good deterministic verification
→ regression test
→ patch eligibility
The contribution is not a claim of reliable autonomous bug reproduction. It is an architecture that refuses to trust a patch until executable evidence survives strict, repeated, oracle-separated evaluation.
Developers and teams using coding agents for defect work. A confident patch is not enough when nobody has demonstrated that the agent triggered the reported bug, observed the right failure, and distinguished it from an unrelated error.
Bug reproduction is the trust bottleneck. Without a verified reproduction, reviewers cannot tell whether a patch fixes the reported mechanism, merely changes nearby code, or makes a generated test pass for the wrong reason. ProofBeforePatch turns that missing evidence into an explicit state-machine gate rather than another model confidence score.
The workflow produced verified reproductions in several recorded iterations, including a historical best 2/3 run. That exact producing source snapshot was not preserved, and the frozen submission source later audited at 0/3. A final recovery of the current code family reached 1/3. Model-driven experiment generation therefore remains unstable and is the main limitation.
| Evidence class | Baseline | Final | Meaning |
|---|---|---|---|
| Current reproducible recovery | 0/3 | 1/3 (33.3%) | Latest measured current-code-family comparison |
| Historical best recorded | 0/3 | 2/3 (66.7%) | Measured evidence; exact source snapshot unavailable |
| Frozen-source audit | — | 0/3 | Historical 2/3 was not independently reproduced |
| Five-case stress test | 0/5 | 0/5 | Preserved generalization regression |
| Groq compatibility gate | — | No benchmark | 1/6 schemas passed; no VRR exists |
These results are never averaged or combined. See the canonical results ledger for per-case metrics, runtimes, usage, provenance, and immutable paths.
The evidence-backed lesson is:
An agent failing safely is better than an agent succeeding falsely.
Baseline and experimental paths often produced plausible but invalid reproductions. The deterministic verifier rejected them rather than allowing a confident false success. Executable evidence was more trustworthy than model confidence.
Yes, at two levels:
- Deterministic replay: run tests and open the console using the stored JSON/JSONL evidence. No model or API key is required.
- Fresh agent run: use Ollama with
qwen2.5-coder:7b-instruct. The same evaluator, sandbox, model settings, and suites are documented, but task success may vary even with temperature 0 and seed 0.
ProofBeforePatch was created for this hackathon. No pre-existing proprietary application code was incorporated. Third-party runtimes, libraries, and model dependencies are disclosed separately.
The project source is provided under the MIT License. Its declared
Python dependencies are listed in pyproject.toml. Optional local inference
uses Ollama and qwen2.5-coder:7b-instruct; neither the Ollama runtime nor model
weights are bundled in this repository, and each remains subject to its
upstream license and terms.
This repository is a curated public export. See the public evidence sanitization notice for the precise, non-semantic removal of local-machine path prefixes from public evidence copies.
The UI is an engineering evidence console, not a chat clone. It displays:
- behavioral contract, hypotheses, experiments, and observations;
- strict buggy-fail / known-good-pass verification;
- generated regression tests and immutable trajectories;
- current, historical, audited, regression, and compatibility evidence in visibly separate provenance classes.
The successful demo trajectory is explicitly labelled Recorded historical verified run. It is not presented as output from the submitted source.
python scripts\build_ui_evidence.py
python -m http.server 4173 --bind 127.0.0.1 --directory apps\consoleOpen http://127.0.0.1:4173/.
The model sees only the bug report, visible buggy code, and public tests. It cannot access known-good/reference code, hidden evaluator files, oracle metadata, or ground-truth fixes.
The verifier requires all of the following:
- the generated reproduction fails on the buggy implementation;
- it passes on the evaluator-only known-good implementation;
- the failure class and public signature match;
- configured repeatability thresholds are met;
- oracle separation remains intact.
Implemented safeguards include strict Pydantic contracts, a workflow state machine, a human-approved patch gate, sandboxed fresh-workspace execution, append-only trajectories, AST assertion recognition, deterministic public failure-signature repair, safe test-harness completion, hidden-path rejection, and immutable machine-readable result storage.
Python 3.12+ and pydantic>=2.10,<3 are required.
python -m pip install -e .
python -m unittest discover -s tests -v
python scripts\clean_room_check.py
python scripts\build_ui_evidence.py
python -m http.server 4173 --bind 127.0.0.1 --directory apps\consoleThe practical free model is qwen2.5-coder:7b-instruct through any compatible
local endpoint; Ollama is the documented Windows runtime. API cost is $0
using local inference, while local RAM, compute, electricity, and roughly
4.7 GB of model storage are still required.
ollama pull qwen2.5-coder:7b-instruct
ollama serve
python scripts\check_model_server.py
python scripts\validate_structured_provider.py
python scripts\run_suite.py --workflow baseline --suite evaluation\suite_three_harness.json --iteration baseline
python scripts\run_suite.py --workflow final --suite evaluation\suite_three_harness.json --iteration finalFresh model inference is deliberately not required for judging the stored evidence. On this 15.7 GB RAM, integrated-graphics machine, a final three-case run can take roughly 18–29 minutes based on recorded iterations.
- Results ledger
- Evaluation checkpoint
- Improvement changelog
- Reproduction guide
- Video script
- Historical best comparison:
evaluation/results/history/20260829T183438Z_comparison_qwen2.5-coder-7b-instruct_29277affa2a9/ - Frozen-source audit:
evaluation/results/history/20260830T071547Z_final_qwen2.5-coder-7b-instruct_22ec9b/ - Final recovery comparison artifacts:
evaluation/results/history/20260830T094029Z_baseline_qwen2.5-coder-7b-instruct_4ae097/andevaluation/results/history/20260830T095829Z_final_qwen2.5-coder-7b-instruct_d1e90c/ - Five-case regression comparison:
evaluation/results/history/20260829T214442Z_comparison_qwen2.5-coder-7b-instruct_7f0b11f0b9ca/ - Groq schema validation only:
evaluation/results/history/20260830T102643Z_provider_validation_openai-gpt-oss-120b_b407ed/ - Final unit/integration validation (71/71):
evaluation/results/history/20260830T104703Z_test_suite_38a678/ - Final clean-room validation (71/71; zero secrets/runs copied):
evaluation/results/history/20260830T104807Z_clean_room_fbbdc8/
All failed and incomplete experiments remain preserved. Fixture-control and offline-smoke results test plumbing only and are never reported as model quality scores.