Skip to content

types: evaluation-relative coverage time and mergeable summary families - #592

Draft
zzylol wants to merge 2 commits into
stack/509-w6-counter-prooffrom
stack/509-w3-relative-coverage
Draft

zzylol wants to merge 2 commits into
stack/509-w6-counter-prooffrom
stack/509-w3-relative-coverage

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Wave 1 chain: #594 → #593 → #592 → #591 → #595 → #596 → #597 → #598 → #599 (on #589)

Step 3 of the #580 plan (item A prerequisites). Part of #509.

Why

Stage 1 tumbling windows (W1) compose N SummaryAgg panes with one SummaryMerge. Two facts are missing from crates/types:

  • Where a pane sits in time. Panes are declared relative to the evaluation time (W2), but CoverageRegion.time_ms could only hold absolute timestamps.
  • Whether a family may merge. SummaryMerge accepted any state family with matching schemas, including Rate/Increase accumulators and CMS/CountSketch heaps. W6 says merged heaps are rejected until there is an accuracy model.

What

  • CoverageRegion.time_ms is now Option<CoverageTime>, where CoverageTime is Absolute(Range<i64>) or RelativeToEvaluation(Range<i64>). Pane i of width w covers -(i+1)w..-iw.
  • SummaryCoverage::validate/merge_disjoint treat the two anchors as incomparable. A coverage, or a merge, that mixes them fails with the new CoverageError::MixedTimeAnchors. Within one anchor nothing changes: adjacent regions coalesce, gaps are kept, and overlaps are PossibleOverlap.
  • FieldDataType::family_merges() returns true for exact Sum/Count/Min/Max, KLL, DDSketch, HLL, CMS, CountSketch and UnivMon. Everything else returns false, which fails closed: Rate/Increase/IRate, CmsWithHeap, CountSketchWithHeap, KMV/Theta, Sample/Wavelet/StatModel and plain values. SummaryMerge::validate_inputs rejects a family that does not merge.

How

  • Serde stays backward compatible. Absolute time keeps the plain {"start","end"} encoding, so existing JSON and fixtures are unchanged. Relative time is encoded as {"relative_to_evaluation": {"start","end"}}.
  • From<Range<i64>> for CoverageTime gives Absolute, so call sites only change Some(a..b) to Some((a..b).into()). Only test call sites in crates/types needed this.

Before this PR

5 × SummaryAgg(KLL), coverage time_ms: -60000..0, -120000..-60000, …   # no way to say "relative"
SummaryMerge over CmsWithHeap panes  -> accepted

After this PR

5 × SummaryAgg(KLL), time_ms: RelativeToEvaluation(-(i+1)·60000 .. -i·60000)
SummaryMerge  -> coverage RelativeToEvaluation(-300000..0)     (one region)
relative + absolute inputs -> CoverageError::MixedTimeAnchors
SummaryMerge over CmsWithHeap panes  -> "summary merge over Sketch(CmsWithHeap…) is unsupported: the family has no sound merge"

Tests

Tests were written first. With the validate_inputs check disabled, summary_merge_requires_a_mergeable_family fails.

  • summary_coverage.rs: adjacent relative panes coalesce to [-5m, 0), a relative gap is kept, overlapping relative regions are rejected, mixing anchors is rejected (in a merge and in validate), and serde round trip plus legacy JSON.
  • summary_merge_structure.rs: merging heap panes is rejected and merging KLL panes is accepted. There is also a table test of family_merges.

Design doc follow-up (#573, asap-primitive-schema; this PR does not edit that branch)

  • §4 (node fields / coverage): CoverageRegion.time_ms: Option<CoverageTime>, where CoverageTime = Absolute(Range<i64>) | RelativeToEvaluation(Range<i64>). Relative bounds are offsets from the evaluation time (pane i of width w is [-(i+1)w, -iw)). The two anchors are incomparable, so mixing them is MixedTimeAnchors. The legacy plain time_ms range means Absolute.
  • §5.6 (SummaryMerge): add one check, that the state family must satisfy FieldDataType::family_merges(). That holds for Sum/Count/Min/Max, KLL, DDSketch, HLL, CMS, CountSketch and UnivMon. Rate/Increase and heap top-k are rejected. The example can use relative panes: two [-60000, 0) / [-120000, -60000) KLL panes derive to [-120000, 0).

Open questions for review

  • KMV and Theta merge soundly in the literature but were not on the W6 list, so they currently fail closed.
  • family_merges looks only at the algorithm, not at GroupingStrategy. Hydra-shared states merge under the same rule until W7 defines Hydra merge.

Gate

  • cargo fmt --all --check: passes
  • cargo clippy --workspace --all-targets --all-features -- -D warnings: passes
  • cargo test --workspace: 1,524 passed, 10 ignored (feat(planner): whole-expression top-k heap sketch over raw samples #589: 1,517 / 10, plus 7 new tests)
  • stage_pipeline --example planner-layering-1 regenerates tools/dag-viewer/examples/planner-layering-example1.json byte-identically

🤖 Generated with Claude Code

zzylol and others added 2 commits October 5, 2026 04:48
CoverageRegion.time_ms is now Option<CoverageTime>: Absolute or
RelativeToEvaluation (tumbling pane i of width w covers -(i+1)w..-iw).
The two anchors are incomparable, so validate/merge_disjoint reject a
mix with MixedTimeAnchors; within one anchor, adjacent regions coalesce
and gaps are kept as before. Absolute time keeps the plain {start,end}
JSON encoding, so legacy coverage still deserializes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
FieldDataType::family_merges() is true for exact Sum/Count/Min/Max,
KLL, DDSketch, HLL, CMS, CountSketch and UnivMon; every other family
fails closed (Rate/Increase accumulators and heap top-k sketches
included, per W6). SummaryMerge::validate_inputs rejects the rest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-w6-counter-proof branch from ee6f881 to 761b2ce Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-w3-relative-coverage branch from 0f9db83 to 6b31d2d Compare October 5, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant