Background
While splitting #462, we set the boundary that ASAPPlanner should not contain SDS or deployment-specific storage/wire formats. Each deployment picks its own formats. #462 already moves these back to ASAPQuery-backend:
types/post_asap/semantic_definition.rs (v1 and v2)
stored_state/{delta_apply,decoders,native}.rs
- the
encode/decode version envelopes of CompiledPhysicalDag and PhysicalCandidate
Serialization inside the summary kernels was deliberately left out of #462 to keep the split scope bounded. This issue tracks it.
Scope
In crates/asap-physical-operators on the #462 branch:
SummaryKernel in summary_kernels/traits.rs requires serialize_to_json and serialize_to_bytes.
- Every sketch kernel has its own msgpack/proto encode/decode, e.g.
count_min_sketch.rs, count_sketch.rs, dd_sketch.rs, hll_sketch.rs, datasketches_kll.rs, *_with_heap.rs.
summary_kernels/sketch_envelope.rs and the asap_sketch_codec crate.
serialize_to_bytes / deserialize_from_bytes on KeyByLabelValues and Measurement.
This code was moved from ASAPQuery-backend/data_plane/src/precompute_engine/operators/*_accumulator.rs with only minor changes.
Observation
On the #462 branch, apart from summary_kernels/ itself and stored_state/ (which is moving back to Backend), no Planner code calls these serializers. Build/merge/readout work on in-memory state. So once stored_state moves back, the byte formats are probably used only by Backend.
Questions
- Can the serialization methods be dropped from
SummaryKernel, keeping only the in-memory state API (build/merge/readout)?
- Should msgpack/proto encoding and
asap_sketch_codec move to Backend, next to the SDS wire decoding in sketch_db?
- Does Backend need an accessor to read or rebuild kernel state (for example, exposing the underlying
asap_sketchlib structure)? The goal is to avoid adding a new abstraction layer.
Non-goals
Background
While splitting #462, we set the boundary that ASAPPlanner should not contain SDS or deployment-specific storage/wire formats. Each deployment picks its own formats. #462 already moves these back to ASAPQuery-backend:
types/post_asap/semantic_definition.rs(v1 and v2)stored_state/{delta_apply,decoders,native}.rsencode/decodeversion envelopes ofCompiledPhysicalDagandPhysicalCandidateSerialization inside the summary kernels was deliberately left out of #462 to keep the split scope bounded. This issue tracks it.
Scope
In
crates/asap-physical-operatorson the #462 branch:SummaryKernelinsummary_kernels/traits.rsrequiresserialize_to_jsonandserialize_to_bytes.count_min_sketch.rs,count_sketch.rs,dd_sketch.rs,hll_sketch.rs,datasketches_kll.rs,*_with_heap.rs.summary_kernels/sketch_envelope.rsand theasap_sketch_codeccrate.serialize_to_bytes/deserialize_from_bytesonKeyByLabelValuesandMeasurement.This code was moved from
ASAPQuery-backend/data_plane/src/precompute_engine/operators/*_accumulator.rswith only minor changes.Observation
On the #462 branch, apart from
summary_kernels/itself andstored_state/(which is moving back to Backend), no Planner code calls these serializers. Build/merge/readout work on in-memory state. So oncestored_statemoves back, the byte formats are probably used only by Backend.Questions
SummaryKernel, keeping only the in-memory state API (build/merge/readout)?asap_sketch_codecmove to Backend, next to the SDS wire decoding insketch_db?asap_sketchlibstructure)? The goal is to avoid adding a new abstraction layer.Non-goals
asap_sketchlib.