Skip to content

feat(executor): exact distinct, L2 and entropy, and typed UnivMon identities - #597

Draft
zzylol wants to merge 1 commit into
stack/509-w7a-univmon-accuracyfrom
stack/509-w7b-1-frequency-runtime
Draft

zzylol wants to merge 1 commit into
stack/509-w7a-univmon-accuracyfrom
stack/509-w7b-1-frequency-runtime

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Wave 1 chain: #594 → #593 → #592 → #591 → #595 → #596 → #597 → #598 → #599 (on #589)

Why

#509 Example 2 plans one UnivMon over src_ip for COUNT(DISTINCT src_ip), the entropy of the src_ip distribution and the L2 norm of per-src_ip counts. Pass 1 also keeps an exact candidate for each statistic. On the current stack none of these runs natively:

  • the executor has no native UnivMon state or readouts (summary family has no native DAG state implementation), and a summary build accepts only Float64 updates;
  • Cardinality, FrequencyL2 and FrequencyEntropy aggregates fail to bind (aggregate intent has no native implementation).

The parked draft PRs #557, #559 and #563 fixed this on the old stack (asap-physical-operators). This PR ports them onto crates/executor. Part of #509 / #580 (W8).

What

Before After
Operator::summary_build(flows(src_ip: Utf8), UnivMon, ..) → Err("summary numeric update requires Float64") one UnivMon state from Utf8/Int64/Bool identities; Cardinality, L2 and entropy readouts all read it
Aggregate [Cardinality / FrequencyL2 / FrequencyEntropy {accuracy: Exact}] → no native implementation exact reducers: distinct tuples, L2 of unit counts, entropy in bits
SELECT COUNT(DISTINCT src_ip) FROM flows WHERE keep cannot execute executes through a raw connector: 0 / 0 / 2 on the test inputs

How

Name and path changes only: asap_physical_operators → asap_executor, pre_asap/post_asap → types::ir::{operator, schema, properties}, PostAsapDAGNode → PhysicalASAPDAGNode. The test helper execute_raw_rows (from #563) uses compile_physical_asap_dag and roots[0]. The developer-doc notes from #557 and #559 are not ported.

Tests

Carried over from #557 (univmon_execution::typed_frequency_keys_preserve_identity, two kernel tests), #559 (exact_frequency_*, the memory-budget test) and #563 (exact_cardinality_*, sql_cardinality), plus 14c8ac8's one_univmon_state_answers_distinct_l2_and_entropy. precompute_raw_samples now expects UnivMon to compile and checks its three readouts (the old stack's d206f44), because UnivMon has native state. Before the port, both UnivMon tests fail at runtime and the reducer tests do not compile (no variant FrequencyL2/Cardinality).

Gate: cargo fmt --all --check and cargo clippy --workspace --all-targets --all-features -- -D warnings pass. cargo test --workspace: 1,527 passed, 10 ignored (#589: 1,517 / 10).

Ports #557, #559, #563. Next: SQL recognition (#598).

🤖 Generated with Claude Code

…ntities

Port of the parked runtime PRs for #509 Example 2 onto asap-executor:

- native UnivMon build and Cardinality / FrequencyL2 / FrequencyEntropy
  readouts (the old stack's 14c8ac8, a prerequisite missing here);
- #557: a UnivMon build accepts Utf8, Int64 and Bool identities as
  type-tagged keys; heap key bytes count toward state memory;
- #559: exact FrequencyL2 / FrequencyEntropy reducers (bits, NULL skipped,
  0 for an empty population, memory-accounted, cooperative);
- #563: exact Cardinality reducer over typed tuples.

The legacy raw-cost adapter lists the three intents as hash aggregates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-w7b-1-frequency-runtime branch from bceb13e to c4cd1d0 Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-w7a-univmon-accuracy branch from aab5e7e to a6e5cda Compare October 5, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant