Skip to content

feat: compile PromQL fallback subtrees in the physical planner - #486

Open
zzylol wants to merge 4 commits into
feat/physical-compile-coveragefrom
feat/physical-compile-promql-fallback
Open

zzylol wants to merge 4 commits into
feat/physical-compile-coveragefrom
feat/physical-compile-promql-fallback

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #484.

Rebuilt into the linear stack on main. Conflict resolutions are recorded in the messages of: "feat(physical): add a PromQL per-series window operator"; "feat(physical): compile PromQL fallback subtrees from typed expressions".

Why

physical_planner::compile accepted a Fallback{QueryExpr} node only when the node itself was supplied as an input. ASAPQuery-backend therefore re-parses the PromQL text into its own operator list and maps each operator to promql_values::compile_* (coverage rows 1, 9, 11, 12). This PR is priority item 1 of docs/develop_docs/physical-compile-coverage.md.

What

  • compile lowers a Fallback node from its typed expression. The deployment supplies the raw rows of its one selector at promql_fallback::raw_series_input(node), with the schema from promql_fallback::raw_series(expr). Supplying the node's own ID still means "whole result supplied", for example by an external exact engine. A Fallback that reads no selector, such as vector(1), needs no input.
  • New Operator::series_window evaluates each series at the query time, or on a subquery step grid, over the left-open window (t - offset - range, t - offset]. It covers instant selection and rate/increase/delta/{sum,avg,min,max,count}_over_time, reusing the temporal kernels. Only delta is new.
  • Shapes compiled by the lowering:
    • selectors with offset
    • by aggregates (a global aggregate over no series returns an empty vector)
    • sort, topk by/limit
    • scalar(), vector(literal)
    • arithmetic with one literal
    • f(sel[R:S]) and f(g(sel[r])[R:S]) subqueries, including subquery offset, capped at 100000 steps
  • with_series_identity now accepts these shapes.
  • Frontend bug fix: subquery offset/@ were silently dropped. They are now kept as a TimeShift around the subquery.

Before this PR

max_over_time(m[5m:1m]) retained as a Fallback fails to compile: physical operation has no native binding. The same happens for rate(m[5m]), scalar(m) and similar queries. max_over_time(m[5m:1m] offset 1m) lowered to the same tree as the query without the offset.

After this PR

Take the samples m{job="a"} = (50s: 1), (110s: 7), (170s: 3), (290s: 4). The Fallback compiles to SeriesWindow(instant, 60s lookback, steps 60s..300s) → SeriesWindow(max, 5m). At 300s it returns {job="a"} 7. The step at 240s has no sample in its lookback, so it contributes nothing.

Remaining

  • Fallbacks with more than one selector, such as vector-vector binaries.
  • histogram_quantile: the IR emits by () and drops output labels.
  • Subquery operands other than one per-series function.
  • Implicit subquery resolution.
  • @ modifiers.
  • without grouping.
  • irate, changes, and other range functions.
  • The comparison, summary-algebra and precompute items already listed in the coverage doc.

Validation

  • cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, and cargo test --workspace --no-fail-fast all pass.

  • New tests/promql_fallback.rs compares against hand-computed Prometheus results. Cases: counter resets and extrapolation, left-open boundaries, offsets, lookback and stale markers, NaN, empty inputs, scalar() on 0, 1, or 2 series, subquery grid alignment and offset, and the step cap and input contracts.

  • Regression tests that fail before their fix: subquery offset retention in promql_lowering.rs, and topk by partitioning.

  • A separate reviewer agent that did not write the code reviewed the change. Fixed findings, each with a test:

    • topk by lost its partition.
    • Subquery offset was dropped.
    • The step grid was unbounded.
    • A consumed bare selector could be turned into instant selection.

    Documented rather than changed: output rows keep __name__ in the identity (the adapter applies metric-name rules), and the raw-row extent a subquery requires.

🤖 Generated with Claude Code

@zzylol

zzylol commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Integration note: #477 moves with_series_identity into asap-types as with_promql_series_identity (so asap-aware-mapping can use it), and promql_rows::with_series_identity becomes a delegate. This PR and #487 extend the physical-layer copy (TimeShift, subqueries, scalar(), literal and vector-vector arithmetic, without). When these land after #477, the extended rules must move into asap-types (with scalar_literal inlined there). The integration/planner-for-backend branch already does this port and passes the full workspace tests.

zzylol added a commit that referenced this pull request Sep 30, 2026
#477 moved series-identity resolution into asap-types; #486 and #487 extended
the physical-layer copy. Port the extended rules into asap-types and keep
promql_rows delegating.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 4 commits September 30, 2026 18:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Evaluates instant selection and range functions over left-open windows
at the query time or on a subquery step grid.

Conflicts with earlier stack changes resolved to the integration tree:
- crates/asap-physical-operators/src/operators/mod.rs: a7ff3ae Merge remote-tracking branch 'origin/feat/physical-compile-promql-fallback' into integration/planner-for-backend

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflicts with earlier stack changes resolved to the integration tree:
- crates/asap-physical-operators/src/physical_planner/promql_rows.rs: a7ff3ae Merge remote-tracking branch 'origin/feat/physical-compile-promql-fallback' into integration/planner-for-backend

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the feat/physical-compile-coverage branch from 05d0ecb to f9d8a3a Compare September 30, 2026 18:27
@zzylol
zzylol force-pushed the feat/physical-compile-promql-fallback branch from a4d7e16 to fd0bb0a Compare September 30, 2026 18:27
zzylol added a commit that referenced this pull request Sep 30, 2026
…, without and @

Each Fallback selector reads its own raw-series slot. Vector-vector
arithmetic uses PromQL one-to-one matching with on/ignoring via new
series_labels and series_binary operators; without grouping rewrites the
series identity; @ <timestamp> fixes selector and subquery evaluation.

Conflicts with earlier stack changes resolved to the integration tree:
- crates/asap-physical-operators/src/operators/mod.rs: 9a13ae4 integrate: extend asap-types series identity with #486/#487 shapes
- crates/asap-physical-operators/src/physical_planner/promql_rows.rs: 9a13ae4 integrate: extend asap-types series identity with #486/#487 shapes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Sep 30, 2026
#477 moved PromQL series-identity resolution into asap-types. The
fallback shapes compiled by #486 and #487 (time shifts, subqueries,
scalar bridges, and arithmetic between series) also need identity
realization there. Taken from integration commit 9a13ae4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant