Skip to content

docs: multi-version FFmpeg synergy A/B result — no synergy #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
daedalus merged 1 commit into from
Oct 1, 2026
Merged

docs: multi-version FFmpeg synergy A/B result — no synergy #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/DEEP_DIVE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1076,6 +1076,20 @@ from SPLIT. Replay A/A on 9.0.2: 12 of 6,327 ids differ between two replays. `--
in parallel, starting one only while MemAvailable >= 4 GB (RSS ~3.1 GB at 6k execs). Resumable
(`rows.pkl`, edge ids as `array('I')`). Test: `tests/test_ab_synergy_multi_ffmpeg.py`.

Result (2026-10-01; FFmpeg 9.0.2/8.1.3/8.0.3 ASAN, 10 seeds, 6k execs, 23 hand-made seeds, `--jobs 4`):
**no synergy.** Median edges per version, MULTI minus:

| base | 8.0.3 | 8.1.3 | 9.0.2 | W/L | Holm p |
|---|---|---|---|---|---|
| SPLIT_UNION (equal total compute) | −1,921 | −1,992 | −2,071 | 0/30 | 0.018 |
| FULL_OWN (one version, all compute) | −2,448 | −2,206 | −2,438 | 0/30 | 0.018 |
| SPLIT_OWN (one version, 1/3 compute) | +1,693 | +1,463 | +1,689 | 25/5 | ≤0.027 |

CONTROL passes (p 0.23 / 0.13 / 0.43; per-run noise ±1k edges, up to ±3k). It failed once mid-run
(8.1.3, 7 seeds, p=0.047) and recovered; the check is uncorrected across 3 versions (~14% false alarm).
Joint fuzzing costs ~15% coverage per version vs separate campaigns merged afterwards; it only pays
for cross-version crash diffing in one run.

### Vendored libsecp256k1 target (secp256k1_read)

`targets/secp256k1_read.so` wraps the vendored libsecp256k1 v0.8.0
Expand Down
2 changes: 1 addition & 1 deletion docs/TODO.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@
- [ ] **`FractalVoronoiMutator._boundary_cache` / `_root_hash_cache` are unbounded** (2026-09-25) — only `_plan_cache` is capped (now LRU, `core/lru.py`); the other two grow with every distinct cell/root across input lengths. Bound them with `LRUCache` (Hard Rule 54).

## Crash triage
- [ ] **Multi-version FFmpeg campaign** (2026-09-27) — `vendor_ffmpeg.sh --top=3` + `build_targets.sh --asan` build `ffmpeg_read_{9.0.2,8.1.3,8.0.3}_asan`. Open: (a) diff crash signatures per version (a crash in only one = regression or silent fix); (b) `patches/ffmpeg-vpk-divide-by-zero.patch` no longer applies to any of the three (WARN only) — check whether upstream fixed it; (c) FATE samples are proxy-blocked here, corpus is 23 hand-made seeds. (d) Run the synergy A/B: `tools/ab_synergy_multi_ffmpeg.py run --seed-corpus ~/fuzzing/ab_synergy/seeds --seeds 10 --budget 6000 --jobs 4` (~12 h; running since 2026-09-27 18:2x UTC), then `analyse`; a failing CONTROL invalidates it.
- [ ] **Multi-version FFmpeg campaign** (2026-09-27) — `vendor_ffmpeg.sh --top=3` + `build_targets.sh --asan` build `ffmpeg_read_{9.0.2,8.1.3,8.0.3}_asan`. Open: (a) diff crash signatures per version (a crash in only one = regression or silent fix); (b) `patches/ffmpeg-vpk-divide-by-zero.patch` no longer applies to any of the three (WARN only) — check whether upstream fixed it; (c) FATE samples are proxy-blocked here, corpus is 23 hand-made seeds. (d) Synergy A/B done (2026-10-01): no synergy; joint loses ~2k edges/version to merged separate campaigns, Holm p=0.018, see DEEP_DIVE. Open: (e) harness does not keep per-cell logs/final exec count — seed 3's 9.0.2 FULL ran 812 s vs ~1,900 s with 14.1k vs 15.5–16.5k edges, cause unknown; (f) CONTROL check is uncorrected across versions (~14% false alarm) — Holm-adjust it in the next run, decided before data.
- [ ] **`crash_hashes` grows per crashing exec, unbounded** (2026-09-24) — `adapters/filesystem.py::save_crash` adds every crashing input's hash, duplicates included; `_prune_crash_data` deliberately spares it. 109 B/entry; fuzzgoat crashes on ~29% of execs, so ~32 B/exec. Candidates: `core/bloom.py` or a bounded LRU; decide whether a false positive (skipped triage) is acceptable first.
- [ ] **Field-level causal search for crash explanation** (2026-09-20) — the crash sidecar names fields and marks which changed against the parent (`services/crash_explain.py`) but not which one triggers the crash. Remaining: (3) ddmin over *fields* rather than byte edits, reusing `core.root_cause.ddmin_edits` with edits grouped by `FieldSpan`, re-serializing CRC/length fix-ups per candidate or the checksum bytes pollute the causal set; (4) per causal field, bisect baseline→crash value for the threshold and classify it (zero, signed/unsigned limit, power of 2, length≠actual); (5) correlate field values with fault addr, access size, ASAN shadow info and cmplog operand pairs, tagging findings verified vs heuristic. Costs tens to hundreds of target execs, so run at idle or campaign end behind a budget flag plus a `fuzzer-tool explain <crash>` command, writing a write-once `<base>.explain.json` (sidecars are 0444) and adding that suffix to the sidecar tuples in `save_crash`, `sendmail.py` and `report.py`. Also open: field maps for the other ~35 formats with mutators (`mutations/`), and a baseline that is itself verified non-crashing (today only `crash_hashes` is consulted).

Expand Down
Loading