feat: execute precompute DAGs with bounded corrections and consistent SDS reads - #763
Merged
Merged
Conversation
zzylol
changed the base branch from
docs/physical-plan-design
to
refactor/backend-plan-split
September 22, 2026 19:35
This was referenced Sep 22, 2026
zzylol
changed the base branch from
refactor/backend-plan-split
to
test/promql-exact-function-coverage
September 22, 2026 20:45
# Conflicts: # data_plane/tests/support/univmon_erp_process.rs
# Conflicts: # docs/design_docs/README.md
zzylol
changed the base branch from
test/promql-exact-function-coverage
to
feat/shared-operator-foundation
September 23, 2026 19:27
zzylol
changed the base branch from
feat/shared-operator-foundation
to
main
September 28, 2026 22:52
This was referenced Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Deployments need to execute Planner-provided precompute DAGs and publish the exact SDS outputs referenced by their QueryPlans. Continuous Remote Write also needs a defined correction and recovery contract: late input must replace affected results, and a query must not combine incompatible revisions from independently published outputs.
Before this PR
Precompute could follow aggregation configuration independently of the selected DAG. Completion depended on closing finite input. There was no durable continuous-input snapshot path for rebuilding affected windows, retaining an older readable revision during partial publication, or recovering those revisions after restart.
For example, after publishing
Sum([2, 3]) = 5, admitting a late4requires a replacement result of9. Appending the full replacement to the previous result would incorrectly produce14.After this PR
--remote-write-revision-dir. Capture accepted input durably, rebuild affected results from bounded retained input, and publish outputs independently using typed persisted state. Retries do not count accepted samples or results twice.Terminology follows Planner #462: Planner enumerates legal maintenance lifecycle choices and compiles physical candidates; Backend selects and binds a feasible candidate. The executable graph is the precompute Physical DAG.
Implementation scope
The initial continuous realization is serial bounded replay, with an atomically replaced, fsynced checkpoint protected by a single-writer lock. It trades update cost for straightforward correction semantics; it does not claim incremental-update efficiency or a bound on total process RSS. Serialized checkpoint size and operator workspace are bounded separately. Selected raw producers must have a supported native recovery codec, checked before admitting input.
The finite-input path remains available. This PR does not add cross-worker shuffle, external producer watermark coordination, cross-version state reuse or ad-hoc SDS discovery.
The execution design, SDS consistent-read contract and publication-completeness document are updated together.
Validation
All run locally:
-D warnings.Test logs remain outside the repository; no evidence bundle or binary logs are committed.
Base:
main(includes #749 and #774). The next dependent PR is #765. Planner remains pinned to4b0839ce1733aab8231eb90944c876063a4551f9.