Skip to content

refactor: own SDS stored-state decoding and dataset-bound semantic identity - #791

Open
zzylol wants to merge 1 commit into
mainfrom
refactor/own-sds-and-dataset-identity
Open

zzylol wants to merge 1 commit into
mainfrom
refactor/own-sds-and-dataset-identity

Conversation

@zzylol

@zzylol zzylol commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Companion to the ASAPPlanner #462 split (ProjectASAP/ASAPPlanner#472–#475). Based on main; the Planner pin (4b0839c) is unchanged.

Before this PR

Backend used two things that are deployment concerns but lived in ASAPPlanner:

  • SDS stored-state code from asap_physical_operators::stored_state: sketch frame decoding (modified-OTLP proto and msgpack), delta reconstruction, sketch readouts and native batch frames.
  • Dataset-bound semantic identity from planner_types::post_asap::SummarySemanticFragment v2, which embedded LogicalDatasetIdentity inside the Planner fragment.

After this PR

Backend owns both. The new Planner layers drop them.

  • sketch_db/query/{decoders,delta_apply,sketch_readout}.rs and sketch_db/data/native_batch.rs hold the SDS code, which used to be re-export shims.
  • asap_types::semantic_fragment holds the v1 SemanticFragment (a structural dependency closure) and LogicalDatasetIdentity.
  • The dataset is recorded next to the fragment:
    • PrecomputeMaterialization.dataset_identity;
    • SummarySemantics::Planner { dataset, fragment }.
  • So tenant A's KLL(latency) and tenant B's KLL(latency) still get different SummaryDefinitionIds. The check against ingest.dataset_identity now compares the materialization's dataset.

Not in this PR

  • Exact readouts (stored_state::readout::exact_readout[_optional]) are Planner semantics and are still called from the Planner at this pin.
  • The physical plan byte format (CompiledPhysicalDag::encode/decode) moves when Backend bumps its Planner pin.

Validation

  • cargo test --workspace: 1556 passed.
  • cargo clippy --all-targets -D warnings and fmt --check: clean.
  • New or updated tests:
    • dataset_identity_is_part_of_definition_id;
    • dataset_identity_survives_binding_and_rejects_wrong_input, which now also rejects a forged materialization dataset.
  • The moved semantic_fragment and native_batch unit tests still run.

control_plane/build.rs needs Go on PATH, as before.

🤖 Generated with Claude Code

…entity

Backend now owns SDS frame decoding, delta reconstruction, sketch readouts
and native batch frames, and the stored-definition semantic fragment. The
logical dataset is recorded on the materialization and in
`SummarySemantics::Planner`, so equal fragments over different datasets get
different definition IDs. Planner stays deployment-agnostic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant