Execute query DAGs with consistent SDS snapshots and request limits - #765
Merged
Merged
Conversation
This was referenced Sep 22, 2026
zzylol
changed the base branch from
refactor/backend-plan-split
to
fix/precompute-post-asap-dag
September 22, 2026 20:45
This was referenced Sep 22, 2026
This was referenced Sep 23, 2026
Merged
zzylol
force-pushed
the
fix/precompute-post-asap-dag
branch
from
September 26, 2026 04:42
f695681 to
e9dc0af
Compare
zzylol
force-pushed
the
refactor/query-plan-dag-execution
branch
from
September 26, 2026 04:42
90154af to
6a9dc9c
Compare
This was referenced Sep 26, 2026
…to impl/sds-stack-765
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Deployments need to execute Planner-selected query computations with the same physical operators used by precompute. Reusing individual kernels is insufficient if adapters reset resource budgets for every range step, combine unrelated input revisions, or turn cancellation into an exact fallback.
Before this PR: query execution retained duplicate summary/relational implementations. A range request could give each evaluation its own default budget. Concurrent publication could mask a native execution failure as a capability miss. Mixed local-state/external-exact plans did not establish a common input snapshot.
After this PR: query adapters execute shared Planner physical operators, retain typed storage bindings, and enforce request-wide resource and snapshot contracts.
Design
For example, ten individually small range evaluations must fail if their retained results exceed the single request budget. Likewise,
local state at r1 / external result after a late correctioncannot execute merely because both branches use the same evaluation timestamp.Design and contracts.
Validation
Added regressions for shared input/result budgets, caller cancellation during remote waits and native polling, context isolation, revision-race error precedence, mixed-source rejection before I/O, external-only execution, and terminal SQL failures without fallback. The revision-race test was first run against the old ordering and reproduced the incorrect CapabilityMiss.
Local validation completed with a full workspace sweep and one corrective rerun:
asap_typesunit tests and 433 control-plane unit tests passed.10instead of the existing Float64 encoding10.0; its oracle was corrected and the test passed on rerun (921 unit cases verified).cargo clippy --workspace --all-targets -- -D warnings, backend formatting, and diff checks passed.Optional live ClickHouse tests were not exercised because
CLICKHOUSE_URLwas not configured. HTTP fixtures and the production-process suites did run. GitHub CI is rerunning on the final pushed head.Scope
Memory accounting covers retained data and estimated workspace, not a hard process RSS bound. Some decoded values are measured after construction. Cancellation is cooperative at polling boundaries rather than preemption inside a synchronous kernel. No cross-system snapshot protocol, ad-hoc SDS discovery, or local raw Scan connector is introduced.
Based on main after #763. Planner remains pinned to
4b0839ce1733aab8231eb90944c876063a4551f9. #761 is the next dependent stack PR; #756/#766 remain diagnostics and runtime-control follow-ups.Closes #764.