Skip to content

Execute Planner physical operators through typed deployment bindings - #774

Merged
zzylol merged 234 commits into
mainfrom
feat/shared-operator-foundation
Sep 28, 2026
Merged

zzylol merged 234 commits into
mainfrom
feat/shared-operator-foundation

Conversation

@zzylol

@zzylol zzylol commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Deployments need to execute Planner-selected computation without maintaining a second interpretation of its operators. This PR consumes the physical-operator library from Planner #462 and connects typed query inputs, stored summaries, and protocol results to it.

Before this PR: Backend owned duplicate kernels and could lose semantics at the adapter boundary. For example, numeric semi-join keys were treated as absent labels, so 1 = 2 could match; renamed candidate labels could also rewrite the wrong authoritative selector. Physical resource errors could become capability misses and trigger another engine.

After this PR: Planner compiles selected Filter, Sort, Limit and semi-join fragments. Installed plans persist those physical fragments and their typed input contracts; runtime binds inputs and executes them. Label filters and numeric, renamed and multiple join keys retain their semantics. External Prometheus bindings fetch the selected authoritative subquery unchanged, and the native join compares its results. Invalid deployment input contracts fail before activation.

Physical errors preserve their original cause. Memory exhaustion and cancellation terminate instant and range execution without trying another engine or exact fallback. Fragments within one query evaluation share a run context. A concurrent revision change cannot mask terminal physical failures.

The Planner pin also fixes nested temporal aggregation schema derivation: avg_over_time((sum by (job)(m))[6h:]) preserves job and replaces the actual sample column. Its regression covers schema derivation through physical Sort/Limit execution. Exposed query candidates now finalize exact accumulator state before returning query values; internal shared and persisted edges retain their state types. A second regression executes grouped Rate candidates on both sides of a persisted boundary.

SDS follows #737/#749: Planner owns semantic definitions and logical dataset identity; deployment binds stored_output_id and definition_id; recovery stays within the installed plan version. #763 and #765 extend precompute/query DAG integration; #728, #742 and #775 validate candidate structure, synthetic-cost selection and installed data-plane execution respectively.

Scope: this foundation does not add arbitrary local raw Scan execution, ad-hoc SDS discovery, whole-process memory accounting, or HTTP-disconnect cancellation. Deployment source/storage availability remains distinct from operator support.

Validation: #774 passes 433 control-plane and 934 data-plane library tests and strict all-target Clippy. Planner passes 798 mapping/physical-operator tests, 219 type tests and 9 documentation tests. The downstream Level 1 inventory passes all four tests, synthetic Level 2 ranking passes, and the installed candidate sweep passes 23 executions with 66 comparisons against Prometheus. Logs remain local/CI artifacts, not repository files.

@zzylol
zzylol changed the base branch from refactor/backend-plan-split to main September 28, 2026 18:42
@zzylol zzylol changed the title Share Planner physical operators and candidate execution Execute Planner physical operators through typed deployment bindings Sep 28, 2026
@zzylol
zzylol merged commit dae148f into main Sep 28, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant