Skip to content

test(promql-compliance): add olly-bench suite and p75 quantile queries - #739

Merged
milindsrivastava1997 merged 1 commit into
mainfrom
test/promql-compliance-olly-bench
Sep 23, 2026
Merged

milindsrivastava1997 merged 1 commit into
mainfrom
test/promql-compliance-olly-bench

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Summary

  • Add suites/olly-bench.yaml + datasets/olly-bench.yaml: the 25 unique olly-bench handwritten_v2 queries tracked in Tracking: asap-planner query coverage for olly-bench #738. Query structure is unchanged; durations are shrunk to fit the fixture ([1h]/[6h]/[12h]/[24h] → [5m], offsets → 5m, [6h:] → [10m:]). Duplicate lines 17 and 22 are omitted; names carry the original line numbers.
  • Split the runner Makefile: make run-all runs every suite, make run-for-ci runs only CI_CASES. CI now uses run-for-ci, since most olly-bench queries are expected to fail until Tracking: asap-planner query coverage for olly-bench #738 lands.
  • Add p75 quantile_over_time / quantile / quantile by (job) / quantile by (job, instance) queries to the quantiles suite.
  • Register the olly-bench pair in the checked-in fixture parse test.

Notes

  • Dataset is generated by a Python script (not checked in); the formulas are documented in the YAML header. No series or jobs tie, so topk/sort_desc have unique answers, and job worker has zero 5xx so > 0 actually filters.
  • sort_desc ordering is not verified: the comparator sorts series before comparing.

Test plan

  • go test ./... in runner and seeder
  • Live quantiles suite: passes, including all four p75 queries
  • Live olly-bench suite: 3/25 pass (lines 5, 6, 9), matching Tracking: asap-planner query coverage for olly-bench #738; the rest fail with No result for query from ASAPQuery. Prometheus reference parity passes on every query.

Refs #738

🤖 Generated with Claude Code

Add the olly-bench handwritten_v2 queries (#738) as a separate suite with
a stand-in dataset. Durations are shrunk to fit the fixture; query
structure is unchanged. Most queries are not yet planned, so the suite is
excluded from CI: `make run-all` now runs every suite and the CI workflow
uses the new `make run-for-ci`.

Also add p75 quantile_over_time / quantile queries to the quantiles suite.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997
milindsrivastava1997 merged commit af9fa26 into main Sep 23, 2026
1 check passed
@milindsrivastava1997
milindsrivastava1997 deleted the test/promql-compliance-olly-bench branch September 23, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant