Skip to content

feat: cut physical candidates by lifecycle timing from one compilation - #479

Open
zzylol wants to merge 8 commits into
feat/lifecycle-timingfrom
feat/compile-once-cuts
Open

zzylol wants to merge 8 commits into
feat/lifecycle-timingfrom
feat/compile-once-cuts

Conversation

@zzylol

@zzylol zzylol commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #482 (on #476, on #475).

Why

The lifecycle layer decides timing (#482), and physical compilation reads it. A
deployment should compile each query DAG once and turn each lifecycle
assignment into a cut of that compilation. Right now compile_candidate
re-lowers the whole Post-ASAP DAG for every frontier, and callers work out the
frontier from timing themselves (#482's e2e test does this in a hand-written
helper).

What

  • Helper operators take their ID from their Planner node (u64::MAX - (node_id << 16) - index,
    at most one helper per node) instead of from a running counter. Every
    frontier then gives a subgraph of the same lowering, so a cut needs no
    renumbering.
  • cut_candidate(&compiled, frontier) splits one compile result into the
    precompute and query DAGs of a PhysicalCandidate. Nothing is lowered again.
  • New frontier_from_timing(&PostAsapDag) -> Result<Vec<NodeId>> finds the
    frontier in a timed DAG from execution_timed_dag: the ingestion-time nodes
    that query-time nodes read, plus the root if it runs at ingestion time. It
    rejects a DAG where a query-time node feeds an ingestion-time node. It is
    public because the deployment owns the compile/cut loop and needs this
    mapping. Without it, every caller writes its own copy of the rule.
  • enumerate_compiled_frontiers is no longer public. Placement now comes from
    lifecycle timing, so the only callers were enumerate_frontiers and tests.
  • compile_candidates lowers once and cuts once per frontier.
  • In feat: derive execution timing from a summary lifecycle plan #482's e2e test, frontier_from_timing replaces the hand-written frontier
    helper.

Before this PR

let dag = plan.execution_timed_dag()?;
let frontier = /* caller-derived from timing */;
compile_candidate(&dag, inputs, &roots, &frontier)?;      // re-lowers per assignment

After this PR

let compiled = compile(&dag, inputs, &roots)?;            // lowers once
for plan in lifecycle_plans {
    let frontier = frontier_from_timing(&plan.execution_timed_dag()?)?;
    let candidate = cut_candidate(&compiled, &frontier)?; // partitions only
}

For quantile(0.99, latency) and sum by(job)(rate(m[1m])), the
ContinuouslyMaintained frontier is the retained state and the Ephemeral
frontier is empty. Both cuts come from one compilation.

Compatibility

  • Helper operator IDs are now u64::MAX - (node_id << 16) - index. Encoded physical DAGs that
    contain helpers are different byte for byte from before, so fixtures or
    quotes that embed them must be regenerated.
  • An ingestion-time Binary lowers differently from a query-time one. For that
    node, the timing at compile time must match the assignment being cut. This
    is documented on frontier_from_timing.
  • Temporal pane candidates still use their own lowering.

Validation

  • summary_maintenance_lifecycle_e2e::lifecycle_timing_cuts_one_compilation:
    covers the KLL quantile fixture and grouped Rate→Sum. One compilation of the
    Ephemeral-timed DAG is cut by both the ContinuouslyMaintained and the
    Ephemeral timed DAGs. Each cut serializes byte for byte like
    compile_candidate on that timed DAG, and each frontier is the expected one.
  • Unit tests in physical_planner::candidates: across two timings, each
    Planner node is lowered exactly once (counted) and both cuts match
    compile_candidate. An ingestion-time root is the frontier. A query-time node
    feeding an ingestion-time node is rejected.
  • precompute_candidates.rs: every enumerated frontier cuts to the same bytes
    as per-frontier recompilation. The top-k Sort helper is u64::MAX - root
    and is rejected as a boundary.
  • cargo fmt --check and cargo clippy --workspace --all-targets -- -D warnings
    pass. cargo test --workspace --no-fail-fast: 1378 passed, 0 failed.

🤖 Generated with Claude Code

@zzylol
zzylol force-pushed the feat/lifecycle-candidates branch from 59b41bd to 4bb8c86 Compare September 30, 2026 02:14
@zzylol
zzylol force-pushed the feat/compile-once-cuts branch from 781d02b to 215f4d7 Compare September 30, 2026 02:14
zzylol and others added 3 commits September 30, 2026 02:18
`compile_candidate` re-lowered the whole Post-ASAP DAG for every
materialization frontier, and `enumerate_frontiers` compiled it once more.
Number helper operators from their Planner node (`u64::MAX - node_id`,
at most one helper per node) so every boundary choice is a subgraph of
one lowering. `cut_candidate` partitions a `compile` result for one
frontier and `enumerate_compiled_frontiers` enumerates over it; both
produce candidates byte-identical to per-frontier recompilation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The #462 split no longer exposes PhysicalCandidate::encode; its serde form
gives the same byte-for-byte comparison.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 3 commits September 30, 2026 02:47
The lifecycle layer assigns each node's timing; physical compilation now
reads it. `frontier_from_timing` returns the ingestion-time nodes read by
query-time nodes (or an ingestion-time root) and rejects a query-time node
feeding an ingestion-time one, so each lifecycle assignment is a
`cut_candidate` of one `compile` result. `enumerate_compiled_frontiers` is
private: placement comes from timing, and its only caller is
`enumerate_frontiers`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For the KLL quantile and grouped Rate->Sum fixtures, ContinuouslyMaintained
and Ephemeral timed DAGs cut one compilation into exactly the candidates
`compile_candidate` builds. The hand-written timing frontier in the chosen
lifecycle test now uses `frontier_from_timing`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol changed the title feat: derive physical candidates from one compilation feat: cut physical candidates by lifecycle timing from one compilation Sep 30, 2026
@zzylol
zzylol force-pushed the feat/compile-once-cuts branch from 215f4d7 to abb0a85 Compare September 30, 2026 02:48
@zzylol
zzylol changed the base branch from feat/lifecycle-candidates to feat/lifecycle-timing September 30, 2026 02:48
zzylol and others added 2 commits September 30, 2026 03:14
Number helpers as u64::MAX - (node << 16) - index so a node that lowers to
an operator chain keeps deterministic, traversal-independent helper IDs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With several retained states, only those read by a query-time node or
forming the root are cut points.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant