refactor: consume Planner-owned physical DAG operators - #770
Merged
zzylol merged 0 commit intoSep 26, 2026
Merged
Conversation
This was referenced Sep 23, 2026
This was referenced Sep 24, 2026
zzylol
force-pushed
the
test/promql-exact-function-coverage
branch
from
September 26, 2026 04:42
3159bdf to
ef58ad2
Compare
zzylol
force-pushed
the
feat/shared-operator-foundation
branch
from
September 26, 2026 04:42
fef20a3 to
b8ef5d4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dependency stack: main → #768 → #737 → #749 → #771 → #770 → #763 → #765 → #761 → #728 → #742 → #759
Independent follow-ups to #765: #756 (diagnostics), #766 (runtime controls and overhead inspection).
Before this PR
Shared execution and mathematical kernels lived in the backend, while Planner's IR could change independently. Candidate pruning and grouped ranking still had dedicated backend representations.
After this PR
Consume
asap-physical-operatorsandasap_sketch_codecfrom Planner #462, pinned with Planner IR to revision645cacb04a451e25147f0d23aaddd05da16b4746. Remove the backend copies. The library depends on Planner types and has no backend dependency.The API follows Planner #461: phase belongs to node data state, candidate filtering uses a general semi-join, and grouped ranking composes Sort → grouped Limit. Old operator dispatch and wire compatibility paths are removed. Exact accumulator state is explicitly finalized before value consumers. Temporal exact TopK candidates also lower to grouped Sort → Limit, including integer Count scores.
The independent runtime owns shared producers, bounded delivery, cancellation and retained-output accounting. Native computation includes expressions, relations, window reductions and supported summary operations. The same implementation can run at ingestion time or query time. Custom summary values need not fit Arrow RecordBatch.
Backend #763 integrates ingestion and durable publication; #765 integrates query execution and completes relation/temporal/stored-state computation migration. Sources, storage and protocol conversion remain deployment responsibilities. Local raw Scan is deferred; blocking operators have no spill support.
Architecture and DataFusion comparison. Independent library tests run in Planner; backend tests exercise deployment bindings.
Validation
Planner: 246 shared-library unit tests, 18 library integration tests, one doc test, 432 mapping tests and the integration package passed. Both repositories pass strict workspace/all-target Clippy.
Backend stack: 118 type and 444 control-plane library tests passed. The data-plane run passed 925 tests; its remaining persistence synchronization test was fixed and passed separately. All 19 compatibility process tests and 59 other integration tests passed. Tests cover shared SQL sources, cancellation, stored-state coverage, temporal Sort/Limit, HLL/KLL confidence, and ingestion publication/recovery.
The mandatory #754 gate still fails on grouped-temporal-Sum producer shape and quantile-ratio local execution. Those assertions remain intact. Full #759 performance acceptance is not established. Local raw Scan remains explicitly deferred.
The publication process test verifies that a successor is cold before its own data arrives, rejects old-generation frames and returns the successor's new values; it does not assume cross-version payload reuse.
Planner #462 integration update
Pinned all Planner crates to
b58f24268270a4dd7b7caaf0076ea3db842f3e9bfrom ASAPPlanner #462 and sketchlib to5f03ccbd. Consumption usesphysical_planner, shared summary kernels and the current wire-state shapes. Unsupported counter-weighted heap frontiers are reported to Planner before selection; raw counter values are not silently substituted for rate values. Validation: 429 control-plane library tests and data-plane all-target compilation passed.Physical planning and deployment contract.
Physical planning integration status
Depends on ASAPPlanner #462 (
991ef5c3032507e8913bc6f3644a4008c61435b7). Planner owns physical candidate compilation and workload-cost winner selection; backend owns deployment binding and operation. The reordered stack puts operator and accuracy support before the Level 1 test PR so each test branch can be validated independently.Validation remains in progress. Level 1 passed on its own branch before the final counter-window dependency update. Level 2 currently passes six of seven live suites; the remaining sparse counter-window case is being repaired and rerun. Level 3 peak-memory acceptance cannot be established on the local Linux 5.15 host because
memory.peakis unavailable; CI evidence is required. General stored scalar/result frontiers remain integration work, and these results do not establish that support.