feat(zipf): power-law tail + Heaps fit; veto Chao2 saturation gate

[daedalus]

What

core/zipf.py: fits a discrete power law to seeds-per-edge counts (Clauset–Shalizi–Newman MLE), capped at m seeds, and fits Heaps' law to the coverage timeline.

• xmin scan (≤32) picks the fit with the lowest KS distance. The golden-section search runs over every xmin candidate at once, vectorized (1.5× faster than a scalar loop, identical results).
• A Vuong test against a geometric distribution, plus a tail-fraction guard, give a TailLaw verdict. The guard exists because lognormal data passes Vuong.
• Hurwitz zeta is implemented locally with Euler–Maclaurin; no scipy (Hard Rule 51).

Wiring

• EdgeTracker.zipf_estimate() (memoized on an owner-count version) and heaps_estimate().
• Run summary: Zipf tail: and Heaps: lines, next to the Chao2 lines.
• --report: new "Zipf Tail" section after the Chao2 section.
• --stats-file: new zipf entry.
• Behaviour change: SeedPicker._saturation_gate no longer switches on while the tail is POWER_LAW and Heaps β ≥ ZIPF_GROWTH_BETA (0.05, R² ≥ 0.9). Chao2 is only a lower bound when the tail is a power law, so its "saturated" reading can't be trusted there. The fit only runs when Chao2 already reads ≥ 0.99.
• Docs: DEEP_DIVE.md, TODO.md, architecture.dot and the png/svg rebuilt from it.

Tests

• tests/test_zipf.py: fixtures are deterministic, built from each model's PMF, so no RNG is involved.
  • Control tests: Basel value, direct sum, shift identity, and a fit on two interleaved halves agreeing.
  • Recovery of α ∈ {1.5, 2, 2.5, 3}, and a check that the cap reduces bias.
  • Falsification: geometric and lognormal data are rejected.
  • Adversarial: empty input, all ones, too few points, zeros, 10¹² values.
• tests/test_zipf_wiring.py: tracker fit and cache, including a prune plus new seed that leaves all totals unchanged; the gate veto with its falsification and adversarial cases; the report section and stats output.
• 44 new tests pass. The 135 existing suites that touch these files pass, except three TestFuzzerWiring constructor-flag-order tests (test_kruskal_count, test_novelty_confirm, test_seed_round_robin). Those three also fail on master and are unrelated to this change.
• ruff 0.9.0, strict mypy on zipf.py, and lizard CCN 15 are all clean. All pre-commit hooks pass, including impactguard.
• CI does not run on this PR: ci.yml triggers on main, and the base is master.

Cost

• Uncached fit: ~8 ms at 20k edges and 1k seeds. Cached: 27 µs. Heaps fit: 120 µs. For comparison, Chao2 takes 0.9 ms.

Calibration (fuzzgoat, clang, _noasan.so)

On this machine the ASAN builds fail for every target, not just fuzzgoat.

End-to-end at 20k execs:

• Chao2 saturation: 55.5%
• Fit: Zipf tail: s=4.08 (alpha=1.25, power_law)
• Heaps: beta=0.21, 2x execs -> +16.0% edges

The veto never fired: the gate needs Chao2 ≥ 0.99, which short runs don't reach.

Speed, master vs branch, same seeds, 20k execs, 4 runs in parallel on 4 cores:

run	peak eps	avg eps	crashes	final edges	saturation
master s1	122.1	27.7	3	338	55%
branch s1	124.0	110.3	0	278	78%
master s2	127.5	54.7	2	312	56%
branch s2	117.6	25.4	5	301	73%

Peak eps is the same within noise. Average eps and edges follow each run's own path: the runs diverge early (crash replays, cmplog load up to 4.1M comparisons), so these numbers say nothing about the change. That is expected: below saturation the branch adds no per-exec work, only a version-counter increment in record_edges. This is a speed check, not a coverage A/B. Measuring the veto's effect on coverage needs long campaigns and a CLI toggle; that is tracked in docs/TODO.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UbKUguEPpHswuuUbhCXSuT

[sourcery-ai]

Reviewer's Guide

This PR introduces a scipy-free, deterministic Zipf-tail and Heaps-growth analysis, wires memoized estimates into tracker, reporting, and stats paths, and uses the validated growth signal to veto Chao2 saturation gating only after Chao2 reaches saturation; extensive statistical, adversarial, integration, and output tests cover the implementation.

Sequence diagram for memoized tracker estimates

sequenceDiagram
    participant Caller
    participant EdgeTracker
    participant ZipfModule as ZipfModule

    Caller->>EdgeTracker: zipf_estimate()
    alt cached (_owner_version, seed_count)
        EdgeTracker-->>Caller: ZipfFit
    else cache miss
        EdgeTracker->>ZipfModule: fit_zipf(owner_counts, xmax=seed_count)
        ZipfModule-->>EdgeTracker: ZipfFit
        EdgeTracker-->>Caller: ZipfFit
    end

    Caller->>EdgeTracker: heaps_estimate()
    EdgeTracker->>ZipfModule: fit_heaps(coverage_execs, coverage_edges)
    ZipfModule-->>EdgeTracker: HeapsFit
    EdgeTracker-->>Caller: HeapsFit

Loading

Flow diagram for Zipf tail and Heaps fitting

flowchart LR
    A["Seeds per edge counts"] --> B["Aggregate positive counts"]
    B --> C["Scan xmin candidates <= 32"]
    C --> D["Fit alpha with vectorized MLE"]
    D --> E["Select minimum KS fit"]
    E --> F["Vuong test vs geometric"]
    F --> G["Tail fraction guard"]
    G --> H["TailLaw verdict"]
    I["Coverage timeline"] --> J["Fit D(N) = k * N^beta"]
    J --> K["HeapsFit"]

Loading

Flow diagram for the Zipf-aware saturation gate

flowchart TD
    A["Chao2 saturation >= 0.99"] -->|No| B["Do not engage saturation gate"]
    A -->|Yes| C["zipf_estimate().law == POWER_LAW"]
    C -->|No| D["Apply normal saturation gate"]
    C -->|Yes| E["heaps_estimate()"]
    E --> F["beta >= 0.05 and r2 >= 0.9"]
    F -->|Yes| G["Veto saturation gate"]
    F -->|No| D

Loading

File-Level Changes

Change	Details	Files
Add a self-contained statistical module for discrete power-law tails and Heaps’ growth fitting.	Implement local Euler–Maclaurin Hurwitz zeta evaluation without scipy. Fit truncated discrete power laws using a vectorized golden-section MLE across xmin candidates, selecting minimum KS distance. Classify tails with geometric Vuong comparison, alpha-boundary rejection, and minimum tail-fraction checks. Fit recent coverage-timeline points to Heaps’ law and expose projection and doubling-growth metrics.	src/fuzzer_tool/core/zipf.py
Expose Zipf and Heaps estimates through EdgeTracker with mutation-aware memoization.	Add zipf_estimate() and heaps_estimate() APIs. Cache Zipf fits by owner-count version and seed cap, invalidating on owner changes and pruning; clear cache on restore. Use owner counts capped by the current corpus size as the Zipf input.	src/fuzzer_tool/core/edge_tracker.py
Integrate the new estimates into runtime reporting and persisted statistics.	Print Zipf-tail and Heaps growth lines in the run summary. Add a Zipf Tail report section with fit diagnostics and Heaps metrics. Add a zipf object to stats-file output.	src/fuzzer_tool/services/report.py src/fuzzer_tool/services/stats.py
Prevent the Chao2 saturation gate from engaging when a validated power-law tail is still growing.	Run the additional fit only when Chao2 reaches the saturation threshold to avoid early-run overhead. Veto gating for POWER_LAW tails with Heaps beta ≥ 0.05 and R² ≥ 0.9; retain gating for insufficient, non-power-law, or plateau cases. Refresh the veto state on the existing saturation refresh cadence.	src/fuzzer_tool/services/seed_picker.py
Add deterministic coverage for fitting behavior, cache invalidation, gate decisions, and output wiring.	Test Hurwitz controls, exponent recovery, truncation correction, falsification against geometric/lognormal data, and adversarial inputs. Test Heaps fitting, projections, plateau behavior, and invalid timelines. Test EdgeTracker caching and restore behavior, saturation-gate veto conditions, report rendering, summary output, and stats serialization.	tests/test_zipf.py tests/test_zipf_wiring.py
Document the Zipf/Heaps capability and record the remaining saturation-veto validation work.	Describe the estimator, integrations, thresholds, and performance in the deep-dive documentation. Track the need for long-campaign A/B measurement of the veto and comparison with existing coverage projection. Regenerate architecture diagram sources and rendered assets.	docs/DEEP_DIVE.md docs/TODO.md docs/architecture.dot docs/images/architecture.svg docs/images/architecture.png

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Sorry @daedalus, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 5 days and 14 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate issues affect bounded-tail correctness, cache validity, fit cost, and Heaps reporting.

Review effort: Lite
Findings: 1

Open (1)

• Normalize bounded geometric likelihood over xmax · New

What changed in this PR

Adds Zipf tail and Heaps-law analysis, wiring results into tracking, reports, stats, and saturation gating.

Changes:

• Implements deterministic power-law and Heaps fitting.
• Adds caching and saturation-gate integration.
• Updates tests, reports, statistics, documentation, and architecture diagrams.

File	Review summary
tests/​test_zipf.py	Adds estimator and adversarial tests.
tests/​test_zipf_wiring.py	Adds integration and output tests.
src/​fuzzer_tool/​services/​stats.py	Adds stats output; two moderate issues remain (1 vote each) regarding cost gating and independent Heaps output.
src/​fuzzer_tool/​services/​seed_picker.py	Adds the Zipf-based saturation veto.
src/​fuzzer_tool/​services/​report.py	Adds the report section; one moderate issue remains (1 vote) regarding independent Heaps rendering.
src/​fuzzer_tool/​core/​zipf.py	Adds fitting algorithms; one critical issue (3 votes) concerns bounded geometric normalization, and one moderate issue (1 vote) concerns xmax filtering.
src/​fuzzer_tool/​core/​edge_tracker.py	Adds estimator APIs and caching; two moderate issues remain (1 vote each) regarding cost gating and resize cache invalidation.
docs/​TODO.md	Documents follow-up calibration work.
docs/​images/​architecture.svg	Regenerated architecture diagram.
docs/​DEEP_DIVE.md	Documents the new analysis and gating behavior.
docs/​architecture.dot	Updates architecture source.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.