Tracking: asap-planner query coverage for olly-bench

[milindsrivastava1997]

Track asap-planner-rs coverage of the olly-bench PromQL queries (benchmarks/metrics_observability/queries/olly-bench/handwritten_v2.txt).

Current status: 3 / 27 queries planned.

Method: each query planned individually with asap-planner --input_config ... --streaming_engine precompute --data-ingestion-interval-ms 15000, repetition_delay_ms: 60000, metric label hints job, instance, status, le, method, path. A query counts as planned if it appears in the generated inference_config.yaml.

Query buckets (handwritten_v2.txt)

Each query is placed in one bucket by its outermost blocker.

Bucket	Count	Lines	Planned?
sum(rate), including by/offset	6	8, 14, 15, 19, 20, 22	no
Binary: sum(increase) / sum(increase), including offset	3	3, 4, 16	no
Binary sum(increase) ratio wrapped in topk/sort_desc	3	1, 2, 17	no
Subqueries (...)[6h:]	4	10, 13, 24, 27	no
topk(sum by (job) (rate))	2	11, 26	no
topk(sum by (job) (metric)), a nested aggregation	1	12	no
topk(max_over_time), a nested aggregation	1	23	no
topk(sum(increase) / scalar(sum(increase)))	1	25	no
Comparison: sum(increase) > 0	1	18	no
histogram_quantile	1	21	no
Bare selector	1	7	no
max_over_time(metric[range])	2	5, 6	yes
sum(metric)	1	9	yes

Coverage by capability added

Add support for	Newly planned	Cumulative
(today)	lines 5, 6, 9	3 / 27
Nested aggregation only	12, 23	5 / 27
sum(rate) / sum(increase) only	8, 14, 15, 19, 20, 22, 3, 4, 16	12 / 27
Both	above + 1, 2, 17, 11, 26	19 / 27

The rows above assume that the outer operator (topk, sort_desc) can run at query time on top of a planned inner result. That still needs to be checked against the query engine.

Still failing with both: subqueries (4), scalar() (1), comparison > 0 (1), histogram_quantile (1), bare selector (1).

Blocker notes

• sum(rate) / sum(increase): rejected on purpose because the combination is non-collapsable (asap-planner-rs/src/planner/patterns.rs, test non_collapsable_combinations_are_rejected). It is the largest blocker: 15 of the 24 failing queries.
• Subqueries: need the inner expression planned as a per-step sliding computation, with the outer *_over_time over that series.

Tasks

• sum(rate) / sum(increase) support
• Nested aggregation / query-time topk and sort_desc over planned results
• Subqueries
• scalar(), comparison operators, histogram_quantile, bare selectors

🤖 Generated with Claude Code

[milindsrivastava1997]

Nested aggregation design and test contract

Scope and ownership

• This feature is PromQL-only.
• asap-planner-rs selects the precomputed anchor; the query engine executes the anchor and the remaining query-time aggregation pipeline.
• This does not depend on executing refactor(query-engine): execute native query DAGs #729 first. refactor(query-engine): execute native query DAGs #729 may later move the same semantics into executable DAG nodes.
• Planner-emitted full physical DAGs are out of scope.

Query config contract

Every QueryConfig must explicitly contain:

planned_subquery: "..."
query_time_aggregations: []

There are no implicit/default fields and no auto-normalization. Missing either field is a config-load error, including handwritten fixtures.

All planners emit these fields:

• PromQL emits the selected anchor plus the unplanned aggregation pipeline.
• SQL and Elastic emit the full existing query as planned_subquery and an empty pipeline. Their existing execution behavior remains unchanged; nested SQL/Elastic support is not included.

Configured PromQL queries use one path:

find exact config
→ execute planned_subquery
→ execute query_time_aggregations, possibly empty
→ format

Queries with no exact config fall back to Prometheus.

Anchor selection

For a nested PromQL aggregation chain, the planner peels outer aggregation nodes until it finds the largest existing planner-supported subtree. That subtree is planned_subquery; the peeled nodes are emitted inner-to-outer in query_time_aggregations.

Examples:

topk(3, sum by (job) (memory))
  anchor: sum by (job) (memory)
  pipeline: [topk(3)]

topk(3, sum by (job) (rate(cpu[5m])))
  anchor: rate(cpu[5m])
  pipeline: [sum by(job), topk(3)]

avg(sum(rate(cpu[5m])))
  anchor: rate(cpu[5m])
  pipeline: [sum, avg]

If no supported anchor exists before reaching a binary expression, subquery, or unsupported leaf, no config is emitted and the query falls back to Prometheus. Binary children and subqueries are not part of this feature.

Operators and semantics

Supported query-time aggregation operators:

sum, count, avg, min, max, quantile, topk

All support global grouping, by(...), and without(...).

• Normal aggregations drop __name__.
• topk preserves complete input labels; by and without partition ranking rather than changing output labels.
• topk sorts descending and resolves equal values by lexicographic full-label-set order.
• quantile is exact PromQL-compatible aggregation over materialized f64 inner results, with phi validation, NaN handling, and interpolation.
• Instant queries apply the pipeline once; range queries apply it independently at every evaluation timestamp.
• A runtime store/execution error, missing data, or a non-finite anchor result falls back for the whole query; no partial answer is returned.
• Static config errors are rejected at config load: unknown operator, missing k/phi, invalid grouping mode, non-integer or zero k, and phi outside [0, 1].

CMS-with-heap optimization

topk(k, sum by (...) (metric)) intentionally uses the generic anchor-plus-query-time-pipeline path first. The separate optimization to collapse it into a CountMinSketchWithHeap with the inner by labels as aggregated labels and sum subtype is tracked in #744.

TDD test contract

Implement vertical red-green slices, not a bulk test pass.

First tracer bullet:

topk(3, sum_over_time(http_requests_total[5m]))

It must use planned_subquery == query and an empty pipeline while preserving existing CountMinSketchWithHeap planning and execution.

First new nested semantic slice:

topk(3, sum by (job) (memory))

It anchors at sum by (job) (memory) and applies query-time topk(3).

Test layers:

1. Planner tests assert anchor selection and emitted pipeline YAML.
2. Public query-engine tests through handle_query_promql cover all seven operators across global, by(job), and without(instance) grouping (21 cases).
3. Focused tests cover top-k tie-breaking, k = 0, k = 1, over-large k, phi = 0, phi = 1, and invalid phi.
4. Range tests verify per-timestamp pipeline application, including a changing top-k winner.
5. Config-load tests reject missing or malformed plan fields.
6. Differential E2E tests compare Prometheus and ASAPQuery.

The E2E matrix contains every operator at one and two aggregation levels:

• One level: 9 variants: six non-quantile operators plus quantile at phi 0.5, 0.75, and 0.9.
• Two levels: 81 variants: all ordered pairs, expanding either quantile position across those three phi values and quantile-over-quantile across all nine ordered phi pairs.
• Total: 90 queries, each tested as both instant and range.

Use topk(k = 3) and a shared fixture with 3 jobs × 4 instances (12 distinct, time-varying source series), so ranking is meaningful and changes across range steps. The E2E matrix uses by(...); focused public-engine tests cover global and without(...).

Check in the fixture and suite, run it via make run-all and before paper experiments, and initially keep it outside the fast CI set.

[milindsrivastava1997]

Coordination with #729

If #729 is implemented concurrently, it becomes the preferred foundation for the query-engine half of nested aggregation; the planner/config work remains independent.

Work that can proceed now:

- QueryConfig schema and required YAML fields
- PromQL planner anchor selection and emitted pipeline
- Planner/config TDD tests

After #729 lands:

- Add AggregateVector nodes to the executable DAG
- Execute query_time_aggregations through those nodes
- Add engine, range, and differential E2E tests

The nested feature should extend the executor with:

StoreRead → ComposeWindows → ResolveKeys → Estimate
          → AggregateVector(sum/count/avg/min/max/quantile/topk)
          → AggregateVector(...)
          → Format

The test contract does not change. The only implementation difference is that the aggregation pipeline becomes executable DAG nodes immediately rather than a temporary hand-written evaluator in promql.rs.

The #729 executor must preserve an unformatted intermediate result—values plus label schema—between Estimate and Format, because nested aggregation must run before formatting to use labels for by/without grouping and top-k partitions.

Do not wait to start planner/config work, but wait to implement the PromQL execution half until #729’s executor interface is settled or merged, avoiding overlapping changes in query_plan.rs and simple_engine.