Skip to content

test: select, install and execute every admitted query-ensemble candidate - #775

Open
zzylol wants to merge 67 commits into
732-test-add-prometheus-remote-write-promql-differential-suitefrom
test/physical-execution-e2e
Open

zzylol wants to merge 67 commits into
732-test-add-prometheus-remote-write-promql-differential-suitefrom
test/physical-execution-e2e

Conversation

@zzylol

@zzylol zzylol commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Before this PR

Structural and synthetic-ranking tests did not prove that each admitted physical candidate could be compiled into deployment plans, installed, fed data, and return the expected query results. Testing only the default winner could leave other legal candidates unexecuted.

After this PR

For each individual query and query ensemble, enumerate admitted physical candidates and change synthetic costs to make each candidate win. Compile the selected Physical DAGs into deployment plans, install them, ingest Remote Write data, drain precompute, and compare query results with Prometheus. Each ensemble shares one installation per candidate.

This is the data-plane acceptance step after #728 (candidate structure) and #742 (synthetic ranking). It does not measure production costs or implement online ERP/replanning.

Alignment with #737

  • Planning supplies explicit logical dataset identity before Planner semantic export. Installed bindings and cost manifests retain that identity; mismatched bindings are rejected.
  • Same-version restart recovers eligible persisted state without re-ingestion. A new version with identical definitions starts cold and must populate its own state.
  • Publication between query branches invalidates the entire result through the revision fence.
  • Snapshot version 3 and catalog schema 6 reject the preceding contract. Cross-version state adoption and ad-hoc SDS discovery remain deferred.

Validation

The dataset-bound rerun passed 23 candidate executions and 66 query comparisons, covering all ten individual queries, shared-rate, shared-quantiles, and the full ten-query ensemble. These are all admitted fixture candidates; rejected candidates and their reasons remain visible. This is not a claim to cover every theoretical physical plan.

Also passed: four Level 1 tests, full synthetic Level 2 ranking, focused dataset/install/recovery/warm-up/query-fence tests, and strict all-target Clippy. Deferred test branches compile; their production-cost acceptance has not been rerun.

Candidate admission reports and human-review instructions. Generated archives are kept outside version control.

Run make candidates in promql-compliance/runner. Plan exports support human review; automated success does not replace it.

Additional regression: individually certified quantile readouts do not certify their ratio. Deployment admission checks the complete query guarantee and rejects uncertified compositions before pricing or installation.

@zzylol zzylol changed the title test: isolate installed physical-plan execution correctness test: compile selected physical candidates into deployment plans and verify data-plane results Sep 28, 2026
@zzylol zzylol changed the title test: compile selected physical candidates into deployment plans and verify data-plane results test: select, install and execute every admitted query-ensemble candidate Sep 28, 2026
# Conflicts:
#	docs/design_docs/physical-operators.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant