Skip to content

Execute query DAGs with consistent SDS snapshots and request limits - #765

Merged
zzylol merged 393 commits into
mainfrom
refactor/query-plan-dag-execution
Sep 29, 2026
Merged

zzylol merged 393 commits into
mainfrom
refactor/query-plan-dag-execution

Conversation

@zzylol

@zzylol zzylol commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Deployments need to execute Planner-selected query computations with the same physical operators used by precompute. Reusing individual kernels is insufficient if adapters reset resource budgets for every range step, combine unrelated input revisions, or turn cancellation into an exact fallback.

Before this PR: query execution retained duplicate summary/relational implementations. A range request could give each evaluation its own default budget. Concurrent publication could mask a native execution failure as a capability miss. Mixed local-state/external-exact plans did not establish a common input snapshot.

After this PR: query adapters execute shared Planner physical operators, retain typed storage bindings, and enforce request-wide resource and snapshot contracts.

Design

  • SQL relations execute a compiled physical DAG. Storage frontiers execute together, sharing producers, cancellation and resource control. PromQL values/windows bind to the same native operator library.
  • One request shares memory accounting and cancellation across input preparation, nested native runs, range evaluations and retained results. A dedicated worker owns Planner's non-Send context; request cancellation reaches remote waits and native polling.
  • Preserve feat: execute precompute DAGs with bounded corrections and consistent SDS reads #763's continuous revisions: local PromQL and SQL reads pin a common eligible snapshot in the selected catalog generation. A fresh complete r1 remains readable while r2 is partially published.
  • Reject candidates mixing local SDS/current-series state with external exact inputs without a common snapshot proof, including summary-derived candidate pruning. Installation validates this; serving checks defensively before source access. Whole-query external execution remains supported.
  • Native resource exhaustion and cancellation remain execution failures. They take precedence over revision races and never trigger exact fallback, including at the SQL HTTP boundary.

For example, ten individually small range evaluations must fail if their retained results exceed the single request budget. Likewise, local state at r1 / external result after a late correction cannot execute merely because both branches use the same evaluation timestamp.

Design and contracts.

Validation

Added regressions for shared input/result budgets, caller cancellation during remote waits and native polling, context isolation, revision-race error precedence, mixed-source rejection before I/O, external-only execution, and terminal SQL failures without fallback. The revision-race test was first run against the old ordering and reproduced the incorrect CapabilityMiss.

Local validation completed with a full workspace sweep and one corrective rerun:

  • 118 asap_types unit tests and 433 control-plane unit tests passed.
  • 920 data-plane unit tests passed in the sweep. The remaining new SQL test expected 10 instead of the existing Float64 encoding 10.0; its oracle was corrected and the test passed on rerun (921 unit cases verified).
  • All other workspace targets passed, including 20 production-process query E2E tests and 13 controller-to-backend integration tests. Continuous revisions, two-source consistency, restart, and new-version warm-up are included.
  • Strict cargo clippy --workspace --all-targets -- -D warnings, backend formatting, and diff checks passed.

Optional live ClickHouse tests were not exercised because CLICKHOUSE_URL was not configured. HTTP fixtures and the production-process suites did run. GitHub CI is rerunning on the final pushed head.

Scope

Memory accounting covers retained data and estimated workspace, not a hard process RSS bound. Some decoded values are measured after construction. Cancellation is cooperative at polling boundaries rather than preemption inside a synchronous kernel. No cross-system snapshot protocol, ad-hoc SDS discovery, or local raw Scan connector is introduced.

Based on main after #763. Planner remains pinned to 4b0839ce1733aab8231eb90944c876063a4551f9. #761 is the next dependent stack PR; #756/#766 remain diagnostics and runtime-control follow-ups.

Closes #764.

@zzylol
zzylol changed the base branch from refactor/backend-plan-split to fix/precompute-post-asap-dag September 22, 2026 20:45
@zzylol zzylol changed the title refactor: execute installed QueryPlan sub-DAGs refactor: share physical operator kernels and execute QueryPlan DAGs Sep 23, 2026
@zzylol zzylol changed the title refactor: share physical operator kernels and execute QueryPlan DAGs refactor: execute query DAGs with shared native physical operators Sep 23, 2026
@zzylol
zzylol force-pushed the fix/precompute-post-asap-dag branch from f695681 to e9dc0af Compare September 26, 2026 04:42
@zzylol
zzylol force-pushed the refactor/query-plan-dag-execution branch from 90154af to 6a9dc9c Compare September 26, 2026 04:42
@zzylol
zzylol changed the base branch from fix/precompute-post-asap-dag to main September 29, 2026 01:15
@zzylol zzylol changed the title refactor: execute query DAGs with shared native physical operators Execute query DAGs with consistent SDS snapshots and request limits Sep 29, 2026
@zzylol
zzylol merged commit 8a40569 into main Sep 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Execute installed QueryPlan sub-DAGs in the query engine

1 participant