Conversation
zzylol
marked this pull request as ready for review
September 21, 2026 16:13
zzylol
added a commit
that referenced
this pull request
Sep 22, 2026
Integrate PR #749 reader/writer bindings and maintenance projections while preserving data-partition worker ownership and plan-derived configuration. Keep selected and maintenance DAG schemas distinct and adapt shared-sink execution to the projected graph.
This was referenced Sep 22, 2026
zzylol
force-pushed
the
docs/physical-plan-design
branch
from
September 28, 2026 13:47
b0d77ce to
5f1eebf
Compare
zzylol
force-pushed
the
refactor/backend-plan-split
branch
from
September 28, 2026 16:14
12896bd to
70a8d88
Compare
CompileAndPublishPhysicalPlanRequest gained a required `dataset_identity` field in this branch, but two api_tests fixtures build their request body as JSON by hand and were never updated, so both failed deserialization with `missing field dataset_identity` before reaching the handler. Take the value from the planning snapshot's own `environment.dataset_identity` rather than inventing one, so the fixture keeps describing the same dataset the rest of the snapshot describes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Deployments need one coherent installed plan that binds queries and producers to stored state with known meaning. Routing fingerprints alone cannot distinguish equal metric expressions over different logical datasets, or represent two deployed outputs with the same semantics safely.
Before this PR
Backend configuration mixes plan concerns and does not establish the SDS identity contract from #737. A query binding cannot independently express the semantic computation and the authorized deployed output.
After this PR
Install a coherent
PrecomputePlan,QueryPlan, transmission configuration and immutable catalog snapshot. Planner exports the persisted output's canonical typed dependency closure, including logical dataset identity; deployment binding connects it to a stored output.Two
KLL(latency)outputs named hot/rebuild may share a definition but are independently bound. Another dataset changes the definition; relocating the same dataset preserves it. Same-version restart restores valid state. A new plan version remains cold until fresh input is produced and never adopts previous-version payloads.Catalog schema 6 and planning snapshot schema 3 establish this identity contract here. Typed Count/Rate support formerly in #771 is included because those semantics must remain distinct. Shared physical execution and its precompute execution design document continue in #774.
Obsolete installed-plan aliases, untyped projection decoders, the unused materialization wire adapter, and cross-version adoption metadata are removed. Recovery accepts only the current sidecar schema and propagates malformed or unsupported metadata errors. Producer partition rosters are explicit on the wire.
Dependency changes
asap_sketch_codeccontains the envelope encoding/decoding helpers used by backend ingest, DDSketch/KLL accumulators and wire-format tests. It supports removing theasap-precompute-rsCollector runtime dependency; it adds no sketch algorithms. The workspace member and data-plane dependency are intentional. With the Collector dependency removed, its Sketchlib patch block is unused and is removed too.cd7e9e0tobccc837because this implementation usesLogicalDatasetIdentityandSummarySemanticFragment::from_stored_output_in_dataset. Both APIs are absent at the old revision. All four Planner dependencies use the same immutable revision.Validation and scope
The SDS implementation passed locally: 117 type-library tests, 431 control-plane library tests, 1,165 data-plane library tests, 8 HTTP API tests, 13 serving integration tests, the production-process restart/warm-up test, strict all-target Clippy and formatting.
Coverage includes dataset changes, endpoint relocation, independent hot/rebuild bindings, tampered definitions, writer/read consistency, same-version restart without re-ingestion, and new-version warm-up. Earlier downstream validation also passed Level 1, exhaustive synthetic Level 2 selection, 332 storage tests and 13 serving integration tests.
Discovery, calibration and workload replay now preserve version-3 dataset identity. Process fixtures use typed deployment configuration instead of removed flat aggregation documents. The 81 Python tests and seven focused sketch/component/monitor/forwarding process tests pass locally; the complete workspace and required CI are rerunning.
PR-specific evidence directories and generated archives are removed from the repository. Test summaries belong here; raw logs remain local or in CI.
Ad-hoc discovery, cross-version adoption and production-cost validation remain out of scope. Restricted native configuration helpers support explicit imported-state fixtures; production Planner compilation requires dataset identity. No manual deployment or human approval is claimed.