Skip to content

feat: extend physical compilation coverage for deployment computation - #484

Open
zzylol wants to merge 7 commits into
feat/population-lifecyclefrom
feat/physical-compile-coverage
Open

zzylol wants to merge 7 commits into
feat/population-lifecyclefrom
feat/physical-compile-coverage

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #491.

Rebuilt into the linear stack on main. New integrate: commit(s) fold in integration-branch resolutions this PR needs on top of the earlier stack: "let coverage lowering chains use per-node helper indices".

Why

ASAPQuery-backend still lowers and evaluates much of its computation itself. It rebuilds PromQL operators from query text (query_time.rs, physical_values.rs), computes current-series readouts in its store, and assembles value graphs by hand. In the agreed layering, the Planner physical compile (physical_planner::compile) owns all computation. The backend owns ingestion, storage, readout I/O, external exact engines, and scheduling.

What

  • Inventory: docs/develop_docs/physical-compile-coverage.md maps 31 backend computation sites to Planner payloads and marks their status at feat: compile Planner selections into physical DAG candidates #475: 11 supported, 4 partial, 14 missing, 2 backend-owned.
  • ReadPopulation Sum / Count / Average / Quantile now compile to grouped aggregation. A new exact Reduction::Quantile uses PromQL interpolation and puts NaN first. Global readouts of an empty population return an empty vector.
  • Query-time Binary over logical value rows:
    • A scalar-literal operand is folded into a projection. This covers x * 2, 100 - x, and -x.
    • Two grouped vectors are matched one-to-one on their label columns (inner equi-join), then the operator is applied.
    • Comparisons and per-series rows fail at compile time instead of guessing.
  • Exact count finalization converts Int64 to the declared Float64 value. Before, compile rejected it.

After this PR the inventory reads 17 supported, 4 partial, 8 missing, 2 backend-owned.

Before / After

Each case below goes PromQL → Planner selection → compile_post_asap_dag → compile, with raw scans as the only inputs:

Query Before this PR After this PR
sum by (job) (m) (maintained population) native population readout does not support this operation {api: 10, db: 5}
quantile by (job) (0.25, m) same error {api: 1.5, db: 5}
sum by (job) (sum_over_time(m[5m])) * 2 node 3: native Planner binding currently requires a unary operation {api: 28, db: 10}
sum by (job) (sum_over_time(a[5m])) / sum by (job) (sum_over_time(b[5m])) vector binary requires Float64 scalars or complete label-map vectors {api: 3, db: +Inf} (unmatched groups dropped)
sum by (job) (count_over_time(m[5m])) node 2: native output type differs from Planner output {api: 4, db: 1}

These cases are the tests in crates/asap-physical-operators/tests/deployment_computation.rs. Each test was confirmed to fail on the base branch.

Remaining

The coverage doc lists the rest in priority order:

  • lowering PromQL-shaped Fallback subtrees: range functions over matrices, scalar(), histogram_quantile, subquery grids
  • comparison filters and bool comparisons (needs return_bool in Binary)
  • per-series vector matching with a name-free identity
  • constant-weight and EntityIdentity precompute updates
  • label-map sketch-state readout
  • summary join, subtract, and delete

Validation

  • cargo fmt --all --check: clean
  • cargo clippy --workspace --all-targets -- -D warnings: clean
  • cargo test --workspace --no-fail-fast: 1363 passed, 0 failed. dag_export::workload_node_ids_make_smaller_topk_reuse_explicit does not exist on this base.
  • A separate review agent, not the implementer, read the diff. It found the empty-population and NaN-ordering issues, and both are fixed in 05d0ecb. No human has reviewed this yet.

🤖 Generated with Claude Code

zzylol and others added 7 commits September 30, 2026 18:09
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ReadPopulation Sum/Count/Average/Quantile now compile to a grouped
aggregate over the population snapshot, so deployments no longer
evaluate these readouts in their current-series store. Quantile uses a
new exact Reduction::Quantile with PromQL rank interpolation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A query-time Binary with a PromQL scalar-literal operand folds the
literal into a projection, which also covers unary negation. Two grouped
row inputs match one-to-one on equal label columns through an inner
equi-join before the operator is applied. Comparisons and per-series
rows still fail at compile time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exact count readout yields Int64, but PromQL declares a Float64
sample, so compile rejected count finalization. Convert exactly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A global population aggregate with no members emitted one row; PromQL
returns an empty vector. Quantile now orders NaN samples first, as
Prometheus does. Found in independent review.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#479 (d41f201) allows several helper operators per Planner node, so the
coverage lowerings added here advance `auxiliary`; make it mutable.
Taken from integration commit da77b78.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the split/462-f-physical-planner branch from bb1abdd to 0ced125 Compare September 30, 2026 18:26
@zzylol
zzylol force-pushed the feat/physical-compile-coverage branch from 05d0ecb to f9d8a3a Compare September 30, 2026 18:27
@zzylol
zzylol changed the base branch from split/462-f-physical-planner to feat/population-lifecycle September 30, 2026 18:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant