docs(minimax): Phases 3-5 on png — null, wall-order leans negative

[daedalus]

This PR records the A/B results for the minimax Phase 3–5 arms added in #20, measured on png. It is docs-only.

Setup: bench_paired.py, target set container_signal (png_read_noasan.so), 20 seeds × 10k execs, 1 run per seed. Each arm is paired against its baseline from ARM_BASELINES.

arm (vs baseline)	W	L	McNemar	Wilcoxon	median Δ edges
elo-op-minimax vs elo (P4)	10	10	1.0	0.68	0.0
wall-order vs baseline (P3)	6	14	0.115	0.17	−4.5
minimax-select vs minimize-2k (P5)	10	10	1.0	0.98	−0.5

Result: none of the three arms helps on png. wall-order is the only one that leans either way, and it leans negative, though not significantly. All three flags are currently on under --hail-mary.

Caveats:

• The runs were not --lock-single-thread: three shards ran in parallel on 4 cores. EPS numbers are only roughly comparable.
• One minimax-select cell (seed 17) scored 348 edges. Re-running that cell gave 126, so it was a lucky single draw. The paired tests are rank-based and unaffected.
• For wall-order, cmplog is live on png (733 pairs), but per-operator pick counts aren't logged. How often the wall order actually decided a pick is unmeasured.
• Phase 2 (the risk matrix) needs several different targets. Only png builds in this environment (the jpeg and ffmpeg libraries are missing), so it is unmeasured.

Changes:

• Added docs/learnings/2026-09-27-minimax-phases-png.md with the full numbers.
• Updated the minimax handover §2 with the result.
• Replaced the "A/B the minimax arms" TODO with the next steps: replicate wall-order (log pick counts first), measure Phase 2, then decide whether each flag stays in --hail-mary.

🤖 Generated with Claude Code

https://claude.ai/code/session_016fZHdKXWjWeWR5XqZC5YCt

---

Generated by Claude Code

Summary by Sourcery

Document the png evaluation of minimax Phases 3–5 and define follow-up decisions for the currently enabled experimental arms.

Enhancements:

• Record png A/B results showing no evidence that the Phase 3–5 minimax arms improve outcomes, with wall-order showing a non-significant negative lean.
• Update minimax handover guidance and TODOs to prioritize replicating wall-order, measuring the Phase 2 risk matrix, and deciding --hail-mary membership.

Documentation:

• Add a detailed learning note documenting the minimax Phase 3–5 png benchmark results and caveats.

[sourcery-ai]

Reviewer's Guide

This docs-only PR records paired png benchmarks for minimax Phases 3–5, finding no significant improvement, updates the handover with the evidence, and replaces the completed benchmarking TODO with replication, instrumentation, Phase 2 coverage, and hail-mary cleanup steps.

File-Level Changes

Change	Details	Files
Documented paired A/B results for minimax Phases 3–5 on the png target, showing no statistically significant benefit and a negative, non-significant lean for wall-order.	Recorded the benchmark setup, per-arm outcomes, aggregate performance metrics, and caveats. Called out the non-reproducible minimax-select outlier and the lack of wall-order pick-exposure metrics. Concluded that none of the measured arms improves png performance.	docs/learnings/2026-09-27-minimax-phases-png.md
Updated the minimax implementation handover with the new png measurement results and Phase 2 status.	Added the W/L, delta, and significance results for Phases 3–5. Linked reviewers to the detailed learning document and noted that Phase 2 remains unmeasured.	docs/handover/handover_minimax_implementation_2026-09-01.md
Replaced the minimax benchmarking TODO with follow-up actions based on the observed results.	Requested a locked, replicated wall-order run after logging condstmt_solve pick counts. Identified heterogeneous-target measurement for Phase 2 and a decision on hail-mary membership or arm removal.	docs/TODO.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Sorry @daedalus, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 5 days and 11 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[Copilot]

Copilot review overview

🟡 Changes recommended

Resolve the target-registration gaps and update the noted documentation and statistical reporting issues.

Review effort: Lite
Findings: 2

Open (2)

• Update stale Game Theory phase status in documentation · New
• Clarify rerun substitution and McNemar test interpretation · New

What changed in this PR

This docs-only PR records PNG A/B results for Minimax Phases 3–5 and updates follow-up evaluation guidance.

Changes:

• Adds detailed benchmark results and caveats.
• Updates the Minimax handover.
• Replaces the outdated TODO with next steps.

File	Description
docs/​TODO.md	Records results and follow-up measurement tasks.
docs/​learnings/​2026-09-27-minimax-phases-png.md	Documents PNG benchmark data and conclusions.
docs/​handover/​handover_minimax_implementation_2026-09-01.md	Summarizes measured phase outcomes and remaining work.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.