Skip to content

Move summary kernel serialization formats out of Planner #471

Description

@zzylol

Background

While splitting #462, we set the boundary that ASAPPlanner should not contain SDS or deployment-specific storage/wire formats. Each deployment picks its own formats. #462 already moves these back to ASAPQuery-backend:

  • types/post_asap/semantic_definition.rs (v1 and v2)
  • stored_state/{delta_apply,decoders,native}.rs
  • the encode/decode version envelopes of CompiledPhysicalDag and PhysicalCandidate

Serialization inside the summary kernels was deliberately left out of #462 to keep the split scope bounded. This issue tracks it.

Scope

In crates/asap-physical-operators on the #462 branch:

  • SummaryKernel in summary_kernels/traits.rs requires serialize_to_json and serialize_to_bytes.
  • Every sketch kernel has its own msgpack/proto encode/decode, e.g. count_min_sketch.rs, count_sketch.rs, dd_sketch.rs, hll_sketch.rs, datasketches_kll.rs, *_with_heap.rs.
  • summary_kernels/sketch_envelope.rs and the asap_sketch_codec crate.
  • serialize_to_bytes / deserialize_from_bytes on KeyByLabelValues and Measurement.

This code was moved from ASAPQuery-backend/data_plane/src/precompute_engine/operators/*_accumulator.rs with only minor changes.

Observation

On the #462 branch, apart from summary_kernels/ itself and stored_state/ (which is moving back to Backend), no Planner code calls these serializers. Build/merge/readout work on in-memory state. So once stored_state moves back, the byte formats are probably used only by Backend.

Questions

  1. Can the serialization methods be dropped from SummaryKernel, keeping only the in-memory state API (build/merge/readout)?
  2. Should msgpack/proto encoding and asap_sketch_codec move to Backend, next to the SDS wire decoding in sketch_db?
  3. Does Backend need an accessor to read or rebuild kernel state (for example, exposing the underlying asap_sketchlib structure)? The goal is to avoid adding a new abstraction layer.

Non-goals

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions