docs: multi-version FFmpeg synergy A/B result — no synergy

[daedalus]

Records the completed tools/ab_synergy_multi_ffmpeg.py run (FFmpeg 9.0.2/8.1.3/8.0.3 ASAN, 10 seeds × 10 campaigns, 6k execs, --jobs 4).

MULTI minus	8.0.3	8.1.3	9.0.2	W/L	Holm p
SPLIT_UNION (equal compute)	−1,921	−1,992	−2,071	0/30	0.018
FULL_OWN	−2,448	−2,206	−2,438	0/30	0.018
SPLIT_OWN (1/3 compute)	+1,693	+1,463	+1,689	25/5	≤0.027

Control passes (p 0.23/0.13/0.43). Joint fuzzing costs ~15% coverage per version vs separate campaigns merged afterwards.

• docs/DEEP_DIVE.md: result table + caveats.
• docs/TODO.md: closes item (d); opens (e) per-cell logs / exec count, (f) Holm-adjust the control check.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YByHAm3GvtQ8X2U1APstxK

---

Generated by Claude Code

Summary by Sourcery

Document the multi-version FFmpeg A/B results and record the remaining validation follow-ups.

Enhancements:

• Document the completed multi-version FFmpeg synergy experiment, showing that joint fuzzing underperforms merged separate campaigns except when compute is allocated to one version.

Documentation:

• Add the experiment results, statistical caveats, control-check limitations, and coverage trade-offs to the deep-dive documentation.
• Update the TODO list with the completed synergy analysis and follow-up work for per-cell logging and Holm-corrected controls.

[sourcery-ai]

Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

This documentation-only change records a completed three-version FFmpeg ASAN synergy experiment, showing that joint fuzzing loses coverage versus separately merged campaigns despite a positive one-third-compute comparison, and updates the campaign TODO with reproducibility and statistical-validation follow-ups.

File-Level Changes

Change	Details	Files
Document the completed multi-version FFmpeg synergy A/B experiment and its negative result.	Record the experiment configuration: three ASAN FFmpeg versions, 10 seeds, 6k executions, 23 hand-made seeds, and four concurrent jobs. Add per-strategy, per-version median edge deltas, win/loss counts, and Holm-adjusted p-values. Document the control results, transient control failure, false-alarm caveat, run noise, and approximately 15% coverage loss from joint fuzzing. Clarify that joint fuzzing is useful primarily for cross-version crash diffing rather than coverage synergy.	docs/DEEP_DIVE.md
Update the multi-version FFmpeg campaign follow-up status and record remaining measurement work.	Close the synergy A/B task with the conclusion that joint fuzzing provides no coverage synergy. Add follow-ups to preserve per-cell logs and final execution counts, investigate anomalous runtime and execution totals, and Holm-adjust control checks in future runs. Retain the separate unresolved crash-signature, upstream patch, and corpus limitations.	docs/TODO.md

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Sorry @daedalus, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 1 day and 13 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[Copilot]

Copilot review overview

🟢 Approval recommended

The documentation accurately reflects the benchmark’s design, results, and limitations.

Review effort: Balanced
Findings: None

What changed in this PR

Documents the completed multi-version FFmpeg synergy experiment and its negative result.

Changes:

• Records coverage, significance, and control results.
• Tracks remaining harness and statistical follow-ups.

File	Description
docs/​DEEP_DIVE.md	Adds experiment results and caveats.
docs/​TODO.md	Marks analysis complete and adds follow-ups.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.