Skip to content

feat: separate physical planning from deployment and share DAG execution - #462

Draft
zzylol wants to merge 69 commits into
mainfrom
feat/shared-physical-operators
Draft

zzylol wants to merge 69 commits into
mainfrom
feat/shared-physical-operators

Conversation

@zzylol

@zzylol zzylol commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Before this PR

ASAPPlanner selected logical computations, while deployments implemented physical operators and execution separately. There was no shared library to keep Planner semantics, operator behavior and shared-producer execution consistent.

After this PR

Add asap-physical-operators: native relational and summary operators, a physical compiler, and a runtime for one DAG execution.

Logical Post-ASAP candidate + maintenance requirements
    → Planner physical compilation
    → precompute/query Physical DAGs + typed boundaries
    → deployment selection and source/state binding
    → shared executor + RunContext

Planner owns operators, dependencies, sharing and materialization frontiers. Backend owns deployment feasibility, runtime-statistics/resource/ERP costing, storage bindings and scheduling. Build, merge and readout are reusable operators, not fixed deployment phases. See the design.

The library includes:

  • Reader-independent physical compilation and versioned physical-program recovery; binding does not repeat logical lowering.
  • Bounded frontier enumeration and comparable workload-cost selection, including Rate → grouped Sum placement candidates.
  • Shared producers, isolated runs, bounded backpressure, cancellation and tracked memory budgets.
  • Relational/scalar execution, summary build/merge/readout, and automatic bounded temporal KLL pane compilation.
  • Float64 weighted CMS/CountSketch heaps and versioned native summary payloads; sketch algorithms remain in asap_sketchlib.
  • An explicit full-label source-row representation for open PromQL series. Rate/heap execution preserves labels not named in the query. Native current-series snapshots apply replacement, expiry and stale markers before heap updates, so historical gauge values do not accumulate.

Validation

Operator, physical-DAG and Planner-to-execution tests cover schemas, errors, resources, independent runs, shared dependencies and serialization. Latest physical-package run passed 320 tests including its documentation test; two additional identity/resource tests and the final weighted/current-series suites also pass. Strict all-target Clippy passes.

The weighted binding suite passes eight tests, including both grouped Rate/Sum placements and raw samples through Rate and both heap families. Coverage includes resets, zero scores, shifted windows and unreferenced labels. Current-series tests cover decreases, expiry, stale markers, conflicting timestamps and recovered physical programs. These are shared-library tests, not deployment E2E.

Integration limits

Backend #761 installs and costs native physical candidates for SQL, spatial TopK, Rate TopK and grouped Rate/Sum. Certified deployment tests execute query-time and precomputed CMS/CountSketch heaps and grouped Sum through durable SDS, HTTP and restart. Precomputed aggregates require finite complete-input closure; overlapping full windows preserve the query cadence. #728 now retains 10 selected plans and 34 compiled candidate plans, including both Sum placements, for human review. Other PromQL shapes retain documented Backend adapters; joint workload candidate search remains bounded rather than exhaustive.

General scalar-output persistence, distributed execution, sharding and spill remain outside the established deployment path. Ad-hoc SDS discovery is deferred. No new manual deployment verification or human plan approval is claimed. The previously recorded Level 3 performance gate remains failed.

@zzylol
zzylol marked this pull request as ready for review September 24, 2026 13:13
@zzylol
zzylol force-pushed the feat/shared-physical-operators branch from 50a3972 to 9ceeba7 Compare September 24, 2026 14:28
@zzylol
zzylol force-pushed the refactor/membership-subgraph branch from ad95c6d to 12ccae2 Compare September 24, 2026 14:28
@zzylol
zzylol force-pushed the refactor/membership-subgraph branch from 12ccae2 to 0afd38a Compare September 24, 2026 14:49
@zzylol
zzylol force-pushed the feat/shared-physical-operators branch 2 times, most recently from 2b6171a to 9463710 Compare September 24, 2026 15:00
@zzylol
zzylol force-pushed the refactor/membership-subgraph branch 2 times, most recently from 36e7da7 to da6ecbf Compare September 24, 2026 15:08
@zzylol
zzylol force-pushed the feat/shared-physical-operators branch 2 times, most recently from d782c4e to 0349523 Compare September 24, 2026 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants