Skip to content

feat: execute precompute DAGs with bounded corrections and consistent SDS reads - #763

Merged
zzylol merged 304 commits into
mainfrom
fix/precompute-post-asap-dag
Sep 29, 2026
Merged

zzylol merged 304 commits into
mainfrom
fix/precompute-post-asap-dag

Conversation

@zzylol

@zzylol zzylol commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Deployments need to execute Planner-provided precompute DAGs and publish the exact SDS outputs referenced by their QueryPlans. Continuous Remote Write also needs a defined correction and recovery contract: late input must replace affected results, and a query must not combine incompatible revisions from independently published outputs.

Before this PR

Precompute could follow aggregation configuration independently of the selected DAG. Completion depended on closing finite input. There was no durable continuous-input snapshot path for rebuilding affected windows, retaining an older readable revision during partial publication, or recovering those revisions after restart.

For example, after publishing Sum([2, 3]) = 5, admitting a late 4 requires a replacement result of 9. Appending the full replacement to the previous result would incorrectly produce 14.

After this PR

  • Execute installed precompute DAGs through the shared physical operators. Backend supplies input bindings, window/population boundaries and publication. Compatible derived sinks share one dependency cache per DAG, window and input revision.
  • Enable continuous local Remote Write with --remote-write-revision-dir. Capture accepted input durably, rebuild affected results from bounded retained input, and publish outputs independently using typed persisted state. Retries do not count accepted samples or results twice.
  • Reject a whole request containing an expired, future-dated or invalid sample before durable admission. Configure the correction horizon, snapshot freshness and byte budget explicitly. Periodic capture evaluates newly elapsed windows without manufacturing a source watermark.
  • Pin a compatible snapshot before query input preparation. All branches and range evaluation points use that view. When A:r2 is ready but B:r2 is pending, A+B can still use fresh, complete r1; affected r1/r2 branches are never mixed.
  • Recover input admission revisions, committed output bytes and partial publication for the same installed generation and policy. New plan versions start with separate history and require warm-up. Recovery validates output references, semantic definitions and typed payloads.
  • Preserve native cancellation and memory errors as execution failures. The finite worker path closes failed-worker admission and still shuts down healthy workers.

Terminology follows Planner #462: Planner enumerates legal maintenance lifecycle choices and compiles physical candidates; Backend selects and binds a feasible candidate. The executable graph is the precompute Physical DAG.

Implementation scope

The initial continuous realization is serial bounded replay, with an atomically replaced, fsynced checkpoint protected by a single-writer lock. It trades update cost for straightforward correction semantics; it does not claim incremental-update efficiency or a bound on total process RSS. Serialized checkpoint size and operator workspace are bounded separately. Selected raw producers must have a supported native recovery codec, checked before admitting input.

The finite-input path remains available. This PR does not add cross-worker shuffle, external producer watermark coordination, cross-version state reuse or ad-hoc SDS discovery.

The execution design, SDS consistent-read contract and publication-completeness document are updated together.

Validation

All run locally:

  • 946 data-plane library tests passed. Revision regressions cover partial publication, pinned reads, freshness and coverage selection, recovery, periodic capture without new input, whole-batch rejection and explicit memory/cancellation failures.
  • 20 process E2E tests passed. New one- and two-source cases compile real Planner-generated DAGs, install the production binary, ingest over HTTP Remote Write, execute derived summaries and bound queries, verify late corrections and retries, reject a mixed expired request, restart the same version, and verify new-version warm-up. Numerical assertions use the selected sketch's error bound. Existing finite-input process tests remain passing.
  • Workspace all-target Clippy passed with -D warnings.
  • Backend workspace formatting check passed.

Test logs remain outside the repository; no evidence bundle or binary logs are committed.

Base: main (includes #749 and #774). The next dependent PR is #765. Planner remains pinned to 4b0839ce1733aab8231eb90944c876063a4551f9.

@zzylol zzylol changed the title refactor: execute precompute from Planner DAGs and preserve exact families refactor: execute precompute DAGs with shared physical operators Sep 23, 2026
@zzylol
zzylol changed the base branch from test/promql-exact-function-coverage to feat/shared-operator-foundation September 23, 2026 19:27
@zzylol
zzylol changed the base branch from feat/shared-operator-foundation to main September 28, 2026 22:52
@zzylol zzylol changed the title refactor: execute precompute DAGs with shared physical operators feat: execute precompute DAGs with bounded corrections and consistent SDS reads Sep 29, 2026
@zzylol
zzylol merged commit e45280b into main Sep 29, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant