Issues

[Bug] Multi-turn tool transcript is spec-invalid (1 tool message for N tool_calls; reasoning_content never round-tripped) -> strict providers 400, scores silently drop
zenprocess/pawbench#18 · opened on Oct 2, 2026
[Bug] Default scenario loading crashes with KeyError: 'name' — scenarios/endpoint_contracts.json doesn't match the scenario schema (v1.1.4, still on master)
zenprocess/pawbench#17 · opened on Aug 29, 2026
spec 009 P7: ablation matrix (--ablate)
zenprocess/pawbench#13 · opened on Apr 7, 2026
spec 009 P6: manifest-only weekly Pawbench run
zenprocess/pawbench#12 · opened on Apr 7, 2026
spec 009 P5: orchestration axis + pawstyle-orchestration-matrix scenario
zenprocess/pawbench#11 · opened on Apr 7, 2026
spec 009 P4: artifact_quality analyzers (Python first)
zenprocess/pawbench#10 · opened on Apr 7, 2026
Benchmark mode: measure context compression vs task success tradeoff
zenprocess/pawbench#8 · opened on Apr 3, 2026
feat: public leaderboard
zenprocess/pawbench#7 · opened on Mar 26, 2026
docs: MkDocs documentation site
zenprocess/pawbench#6 · opened on Mar 26, 2026
feat: add ASCII art banner to README
zenprocess/pawbench#5 · opened on Mar 26, 2026
feat: mock mode for CI + Docker-based eval mode
zenprocess/pawbench#4 · opened on Mar 26, 2026
feat: expand scenario dataset (target 20-50 scenarios)
zenprocess/pawbench#3 · opened on Mar 26, 2026
feat: gold standard reference implementations for scenarios
zenprocess/pawbench#2 · opened on Mar 26, 2026
feat: execution-based correctness scoring (sandbox eval)
zenprocess/pawbench#1 · opened on Mar 26, 2026