feat: expose kernel codec accessors and edge sampling - #495
Merged
Merged
Conversation
zzylol
force-pushed
the
feat/kernel-codec-accessors
branch
from
September 30, 2026 18:27
d718cad to
b945081
Compare
zzylol
force-pushed
the
feat/precompute-raw-sample-input
branch
from
September 30, 2026 18:27
4b6dffd to
f574c00
Compare
zzylol
force-pushed
the
feat/kernel-codec-accessors
branch
from
September 30, 2026 20:24
b945081 to
aebe2ea
Compare
zzylol
changed the base branch from
feat/precompute-raw-sample-input
to
feat/physical-compile-coverage-3
October 1, 2026 22:48
zzylol
force-pushed
the
feat/kernel-codec-accessors
branch
from
October 1, 2026 22:48
aebe2ea to
b8440f7
Compare
zzylol
force-pushed
the
feat/physical-compile-coverage-3
branch
from
October 1, 2026 22:58
148353e to
b2c0aa6
Compare
zzylol
force-pushed
the
feat/kernel-codec-accessors
branch
from
October 1, 2026 23:06
b8440f7 to
a9db8d7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #488.
Why
ASAPQuery-backend #805 stores Planner kernel states directly and keeps only storage codecs. Five gaps in the kernels forced workarounds there: a copy of the UnivMon accumulator, a msgpack round trip to reach the weighted-frequency kernel, a hand-written mirror of the exact state's serde shape, a clone of each cached sketch per ingest frame, and rejecting every edge-sampled frame (
0 < sample_p < 1) because kernels had nowhere to keepp.What
Each new public API exists for one backend need:
UnivMonAccumulator::from_sketch(UnivMon),sketch(),AggregateCore::estimatefor UnivMonasap_summary_state::univmonshim; encode/restore with sketchlib's codecWeightedFrequency::algorithm(),shape(),to_bytes(),from_bytes()ExactAccumulatordeserialization validates against its family (serde(try_from))decode_exact's shape mirror; no new methodAggregateCore::as_any_mutfrom_sketch(sketch, sample_p),sample_p(),merge_sample_p(p)on DDSketch, HLL, Count-Min, Count Sketchp, recordpof deltas applied in placeHow:
sample_ppis kept in a private field, so it is always in (0, 1].from_sketchrejects 0, values above 1, and NaN. The backend should map the proto3 default0to1before calling it.PointCount(total count)Cardinality, barePointCountquery_keyThese rules match the kernels at 76fbbf1 for DDSketch, HLL and Count-Min. Count Sketch had no
sample_pthere; it gets the same1/prule because it is linear.Merge rules:
pvalues merge and keep thatp.1merged with a sampledpgives the sampledp. This is the old "prefer sampled" rule, which exists because window-reset and Planner-created bases are unsampled.pvalues are rejected. The old code silently kept the left operand'sphere.Before this PR
The backend decodes
SketchEnvelope{CountMinState, sample_p: 0.25}and fails with "frame is edge-sampled … not supported by Planner kernels".After this PR
Validation
cargo fmt --all -- --check,cargo clippy --workspace --all-targets -- -D warnings, andcargo test --workspace --no-fail-fastall pass.as_any_mutupdates.pled tomerge_sample_p.Backend follow-ups
sample_pis private, so the backend's struct literals must switch to::neworfrom_sketch.NativeSummaryOutputmust implementas_any_mut.pnext toinner.to_msgpack(). Until they do, keep rejecting sampled frames.inner(e.g.inner.estimate(&str)or row totals) are unscaled. Usequery_key(&KeyByLabelValues::new_with_labels(vec![item]))or divide bysample_p().pis fixed per series. An adaptivepthat changes mid-series gets an error for two different sampled values.🤖 Generated with Claude Code