Figure 1. Integrated ASAPPlanner–ASAPQuery-backend architecture and workflow.
- -```mermaid -flowchart TD - subgraph PlannerBoundary["ASAPPlanner boundary — reusable optimization"] - Canonical[Canonical QueryExpr and workload semantics] - Canonical --> Strategies[CSE and reusable replacement strategies] - Strategies --> Candidates[Candidate post-ASAP workload DAGs] - Candidates --> Ranking[Semantic legality, accuracy and evidence-based ranking] - end - - subgraph BackendBoundary["ASAPQuery-backend boundary — observability application"] - Inputs[PromQL registrations, QueryWorkload and DataWorkload] - Evidence[Runtime capabilities and complete deployment cost evidence] - Commit[Control plane commits a feasible post-ASAP workload DAG] - Compile[Physical binding and deployment selection] - Bundle[One versioned physical plan bundle] - Activate[Validate, stage and activate] - Precompute[Ingest, precompute and summary store] - Serve[Bound query execution and explicit exact fallback] - Feedback[Readiness, accuracy and resource observations] - Inputs --> Commit - Commit --> Compile --> Bundle --> Activate - Activate --> Precompute - Activate --> Serve - Precompute --> Serve - Precompute --> Feedback - Serve --> Feedback - Feedback --> Evidence - end - - Inputs -->|Planning request| Canonical - Candidates -->|Implementation evaluation request| Evidence - Evidence -->|Feasibility and cost evidence| Ranking - Ranking -->|Legal ranked post-ASAP alternatives| Commit - Bundle -->|CollectorPlan in distributed profile| Collector[ASAPCollector — external runtime] - Collector -->|Planned data or summary frames| Precompute - Clients[PromQL clients] --> Serve - Serve -->|Configured exact route| Exact[Prometheus or archive query service] +# Binding Planner Physical DAGs to Backend Deployment Plans + +Status: target design. Audience: developers implementing deployment compilation +and the precompute/query engines. This document defines required behavior, not +completed backend integration. + +## 1. Problem and goals + +Precompute and query execution must agree on what state is produced, where it +is stored and which query inputs may consume it. A full logical DAG embedded in +PrecomputePlan obscures these boundaries. Reconstructing computation independently +in the backend also duplicates Planner's lowering and permits operator or +materialization decisions to diverge. + +The backend will consume Planner-compiled Physical DAGs and bind their typed +boundaries into one coherent deployment plan. PrecomputePlan and QueryPlan reuse +those DAGs and the shared executor; they do not define another computation IR. + +Goals: + +- Preserve Planner's selected operators, sharing and materialization boundaries. +- Bind every physical input/output to an explicit source, stored output or result. +- Install matching producer and consumer contracts atomically. +- Execute through the shared physical library while keeping storage, scheduling, + readiness and serving backend-owned. + +Non-goals are backend operator lowering, a second maintenance-selection model, +CollectorPlan/TransmissionPlan compilation, distributed activation and new +transport protocols. Backend integration must not depend on ASAPCollector. + +## 2. Architecture and ownership + +The authoritative boundary is [Physical Planning, Summary Maintenance, and +Deployment at e9390031](https://github.com/ProjectASAP/ASAPPlanner/blob/e9390031fcecd7bc0d611127eddc5c6603a281e5/docs/design_docs/physical-planning-and-deployment.md). + +```text +ASAPPlanner + Logical Post-ASAP candidates + legal Summary Maintenance Lifecycles + ↓ Physical Plan Compiler + Supported physical candidates + typed input/output boundaries + ↓ +Backend + Feasibility and cost selection over workload candidates + ↓ + Deployment Plan Compiler + sources/store + operational policy + ↓ + PrecomputePlan + QueryPlan + summary definitions + ↓ atomic installation + Deployment engines → shared physical executor ``` -**ASAPPlanner's selected post-ASAP workload DAG is the authoritative semantic -plan. ASAPQuery-backend binds and executes that decision through its control -plane and data plane.** Backend physical plans remain necessary, but must be -traceable projections of that DAG, not independently optimized replacements -for its dependencies, shared state, or query-result semantics. - -Planner provides reusable legal alternatives and ranking. The backend owns -deployment commitment, concrete realization, and operational policy. A -deployment choice cannot silently change Planner-owned grouping, statistic, -summary parameters, logical window, accuracy, or lifecycle: it must return to -the legal candidate-selection boundary. - -## Architecture boundaries and reuse - -ASAPQuery-backend is the observability downstream application, including the -MetricsObservabilityQuery use case. DQC (the proposed name for the current -asap-fusion repository) is a separate downstream application, not an execution -dependency of this backend. - -| Responsibility | ASAPPlanner | ASAPQuery-backend | -| --- | --- | --- | -| Query semantics | Canonical expressions, equivalence, grouping and time semantics | PromQL API, workload registration and profile restrictions | -| Optimization | CSE, legal sharing, rollup, decomposition, summary and accuracy alternatives | Feasibility evidence, deployment commitment and concrete assignments | -| Time and state | Logical windows, abstract window framework and maintenance lifecycle | Panes, retention layout, update implementation and placement | -| Plan identity | Logical producer identities and result dependencies | Plan versions, physical materializations, SID bindings and runtime handles | -| Execution | Deployment-independent semantic contract | Ingest, precompute, store, serving, readiness and fallback | -| Operations | Reusable models consuming scoped evidence | Activation, rollback, telemetry, freshness and resource enforcement | - -Reuse works in both directions. The backend consumes Planner strategies; -general-purpose rules discovered while optimizing repeated observability -queries belong in Planner so DQC and other applications can reuse them. -Prometheus staleness handling, SID resolution, Collector placement, and OpAMP -publication remain downstream responsibilities. - -## Inspection: what already exists - -Inspected backend main at -[`95131d83972bb7a07d338e2a5af925a20c15ddce`](https://github.com/ProjectASAP/ASAPQuery-backend/tree/95131d83972bb7a07d338e2a5af925a20c15ddce), -using its pinned Planner revision -[`cb50219c582d43f53ab77d3a595bd1ea4a9aa119`](https://github.com/ProjectASAP/ASAPPlanner/tree/cb50219c582d43f53ab77d3a595bd1ea4a9aa119). -The baseline is merged code, not the completion of open PRs. - -| Area | Existing foundation | Consolidation needed | -| --- | --- | --- | -| Frontend and selection | Planner dependency, canonical query parsing, backend selection from Planner alternatives | Make workload-wide sharing and strategy composition explicit across supported entry points | -| Physical compilation | One bundle with precompute, transmission, backend and query projections; Collector projections when applicable | Preserve all selected shared producers and provenance through every projection | -| Serving | Bound QueryPlan execution, exact materialization identities and explicit fallback | Audit remaining compatibility paths; serving must not make a new summary choice | -| Deployment | Versioned staging and activation, runtime capability and evidence checks | Verify profile-specific failure and readiness behavior end to end | -| Compatibility | Backend-local ASAPQuery profile alongside distributed collection | Keep distinct deployment profiles on the same semantic contract | - -Evidence: -[selection adapter](../../control_plane/src/planner_selection.rs), -[physical compiler](../../control_plane/src/physical/compiler.rs), -[legacy workload adapter](../../control_plane/src/physical/workload_planner.rs), -[shared QueryPlan](../../crates/asap_types/src/query_plan.rs), -[query lowering](../../control_plane/src/query_plan.rs), and -[bound serving executor](../../data_plane/src/query_engines/asap_query_engine/post_asap_readout.rs). -The selection adapter explicitly commits a ranked Planner candidate downstream. -Consequently, the figure does not imply that the Planner library deploys or -commits a complete backend configuration by itself. - -This is an extension of existing integration, not a proposal to replace it -wholesale. Implementation guides sometimes describe a broader target than an -individual runtime path supports; migration acceptance must be demonstrated -against executable paths, not inferred from interface names. - -## One authoritative semantic DAG, derived runtime plans - -The shared contract must preserve sources and filters, label/grouping identity, -exact operators surrounding summaries, summary build/merge/readout, shared -producers, query roots, logical time coverage, accuracy, and maintenance -requirements. Audit the pinned post-ASAP representation for genuine gaps; -extend Planner semantics where necessary. - -Do not put concrete engine or implementation IDs into Planner IR. The backend -retains a binding from logical producer identity to implementation, placement, -materialization, state schema, and active generation. This follows the -[planner-runtime contract](https://github.com/ProjectASAP/ASAPPlanner/blob/f46cbf6c5738db2f4460d419baa8af5572f5276a/docs/design_docs/architecture/planner-runtime-contract.md). - -One selected DAG can produce several execution projections: - -- PrecomputePlan: how the selected state is built and maintained. -- TransmissionPlan and optional CollectorPlan: how distributed producers - implement and deliver that state. -- SummaryCatalog: canonical summary/data descriptors and stable materialization identities. -- QueryPlan: executable reads, merges, readouts and remaining exact operations. - -These projections may expand one semantic node into several physical tasks. -They must not invent a different semantic sharing graph. QueryPlan need not be -a byte-for-byte serialization of post-ASAP IR, nor should ingestion and query -serving literally run an identical task schedule. They implement different -phases of the same selected computation. - -Sharing has explicit scope: maintain a shared producer once per compatible -source/window/plan generation; reuse its state across query roots. Memoizing a -query DAG within one request is useful but does not, by itself, prove -cross-query or cross-request sharing. - -## End-to-end workflow - -1. **Register demand.** Collect canonical queries, evaluation cadence, time - windows, accuracy scope, source arrival facts and optimization horizon. -2. **Generate alternatives.** Planner applies legal rewrites and sharing, - choosing among summary, abstract-window and lifecycle alternatives. -3. **Evaluate implementations.** The backend checks runtime feasibility and - supplies complete, fresh costs over the same workload horizon. -4. **Commit and bind.** The control plane selects a legal workload alternative, - retains its concrete realization, and compiles one coherent plan bundle. -5. **Publish.** Validate and stage matching projections. For distributed - deployment, require the corresponding Collector application evidence - before activation. A failed rollout preserves the prior active generation. -6. **Maintain and serve.** Ingest updates the selected state; a request uses one - active snapshot and exact bindings. Warm execution requires complete, - fresh coverage. Otherwise follow the configured exact route or return an - explicit failure if that route is unavailable. -7. **Observe and replan.** Attribute cost, readiness and accuracy evidence to - the plan generation and producer. Semantic changes require a new planning - decision and activation, not an ad-hoc serving-time substitution. - -The backend-local profile uses Remote Write, local precompute and Prometheus -fallback without requiring Collector/OpAMP. The distributed profile may use -Collector-maintained summaries and configured archive services. Neither -profile's optional infrastructure becomes a prerequisite for the other. - -## Example: repeated dashboard queries sharing one state producer - -Consider a gauge `request_size_bytes`, one scalar series per -`(service, instance)`, without extra labels. Register these instant-query -expressions repeatedly at the same evaluation cadence: - -```promql -# Q1: sum of observed sample values per service over the last five minutes -sum by (service) (sum_over_time(request_size_bytes[5m])) - -# Q2: sample-weighted mean per service over that same interval -sum by (service) (sum_over_time(request_size_bytes[5m])) -/ -sum by (service) (count_over_time(request_size_bytes[5m])) +| Owner | Decisions | +| --- | --- | +| Planner logical and maintenance candidate construction | Computation semantics, guarantees, window/retention/reuse requirements | +| Planner Physical Plan Compiler | Concrete operators, schemas, dependencies, roots, sharing and materialization frontiers | +| Backend candidate selection | Deployable workload candidate, using capabilities and scoped cost inputs; shared work is costed once within that candidate | +| Backend Deployment Plan Compiler | Concrete source/state bindings, stored-output identities, placement, scheduling and installation version | +| Backend engines | Resolve inputs, drive execution, publish results, check actual readiness and apply installed fallback policy | +| Shared physical library | Operator execution, per-run sharing, backpressure, cancellation and resource contracts | +| SummaryStore | Committed definitions and stored records, lookup, recovery and reclamation | + +The Deployment Plan Compiler binds a realization satisfying Planner requirements; +it does not repair an unsupported candidate or repeat lifecycle planning. For +example, Planner may require retention of at least ten minutes, the compiler +may bind a permitted fifteen-minute retention configuration, and the runtime +actually retains and reclaims records. If the requirement specifies an exact +policy rather than a minimum, the binding must preserve that policy. + +**Build, merge and readout are reusable operators, not deployment-phase classes.** +Planner may place a build in a query DAG or a readout before a persisted scalar +output. Backend execution respects the selected graph boundaries. + +Planner exposes supported, semantically legal physical candidates for the workload. +Backend evaluates deployment feasibility and compares their scoped costs, then +selects a candidate and binds its deployment. Binding failures exclude candidates; +missing costs must not silently become zero. Backend may evaluate binding while +pricing candidates, but cannot change their operators, windows or boundaries. +The current validation uses synthetic costs. Online resource collection and +feedback-driven replanning are deferred. + +A maintenance lifecycle is a contract associated with computation, not another +operator IR. A deployment plan is an operational wrapper around Physical DAGs, +not another lowering stage. + +## 3. Deployment plan structure + +One installed version contains: + +| Part | Content | +| --- | --- | +| Summary definitions | Persisted semantic definitions referenced by stored outputs | +| PrecomputePlan | Planner-provided maintenance Physical DAGs, input/output bindings, schedules and retention/publication policy | +| QueryPlan | Planner-provided query Physical DAGs, input bindings, query associations and explicit fallback policy | + +A physical graph may be embedded or referenced within the bundle; either way, +its operator vocabulary and computation remain Planner-owned. The backend does +not copy it into a second set of Build/Merge/Estimate node variants. + +Two concrete decisions govern these bindings: + +| Design question | Decision and example | +| --- | --- | +| What identifies the source dataset? | Tenant A's and tenant B's `KLL(latency)` have different definitions. Moving tenant A's dataset to another endpoint preserves its definition. See [source identity examples](summary-catalog-sds-architecture.md#source-identity). | +| Can a new plan version reuse old state immediately? | Version 43 populates its own state even if version 42 has the same definition. Same-version restart can recover eligible records. See [recovery examples](summary-catalog-sds-architecture.md#recovery-and-plan-version-changes). | + +Bindings attach only to declared physical boundaries: + +```text +physical input slot → concrete raw source or stored-output reference +physical output → persisted output or query result ``` -Q2 is deliberately not the unweighted mean of per-instance means. Its -denominator counts actual observations, which matters when instances have -different sample counts. These are gauge samples, not counter increases. +Backend resolves a stable logical dataset identity before requesting Planner +semantic definitions. Physical source bindings must realize that identity; changing +an endpoint or replica does not change it. Binding a different dataset requires a +new semantic definition, not reuse of a matching field name. -A legal target alternative is: +The compiler assigns each persisted output a `stored_output_id` within the plan +version. Its writer and all readers refer to the same definition and compatible +format. A `StoredOutputReference` is a binding, not a separately managed catalog +object. [SDS](summary-catalog-sds-architecture.md) defines the storage contract. + +Logical-to-physical provenance comes from Planner and remains available for +inspection. It does not drive backend semantic-node classification or re-lowering. +There is no backend `MaintenanceInput`/`QueryInput` decision in this target model. + +Source, filter, grouping and window describe input-data semantics; they are not +an exhaustive computation schema. The DAG also preserves value expressions, +upstream transformations, operation parameters and typed output semantics. +[Summary-definition completeness](summary-catalog-sds-architecture.md#3-summarydefinition-what-does-this-state-mean) +defines what storage compatibility must preserve. The example below abbreviates +these contracts rather than replacing them with a fixed field list. + +## 4. Worked example: shared KLL state + +Suppose p50 and p99 use KLL with `k=200` over aligned five-minute windows. Planner +selects one-minute panes and compiles: ```text -Selected samples and logical five-minute coverage - | - Shared state per (service, instance) - SUM(value), COUNT(observations) - | - Merge/reduce by service - SUM(sum), SUM(count) - | - +------+------+ - | | - sum -> Q1 sum / count -> Q2 +Maintenance Physical DAG Query Physical DAG + +raw-pane input compatible-pane input + ↓ ↓ +NativeKllBuild(k=200) NativeKllMerge(k=200) + ↓ ┌───┴───┐ +kll-state output ↓ ↓ + p50 readout p99 readout +``` + +The raw input must contain the complete set of input samples for the one-minute pane. The query input +requires compatible panes covering the requested aligned five-minute interval. +These are Planner contracts, not a backend decision to cut the logical graph. + +The backend adds operational bindings. This YAML illustrates ownership and is +not a proposed Rust or wire schema: + +```yaml +precompute: + dag: planner.maintenance_dag + inputs: {raw-pane: latency_source} + outputs: {kll-state: stored_output.latency-panes} + +query: + dag: planner.query_dag + inputs: {compatible-pane: stored_output.latency-panes} + outputs: {p50: query_p50, p99: query_p99} ``` -Planner recognizes the common sum computation and can propose aggregate-state -fusion with per-consumer readouts. The backend implements the selected window -framework with compatible runtime state and binds both query roots to the -same producer. It must preserve PromQL range boundaries, labels, absent-series -behavior and division semantics; a missing denominator is not invented as -zero. Physical panes may be used only when their coverage matches the selected -logical interval, including boundary handling. - -This diagram is a target acceptance example, not a claim that today's compiler -already fuses these complete PromQL expressions. If an operator or window -cannot be realized end to end, the current supported behavior is explicit -fallback rather than partial warm execution with changed semantics. - -For the first milestone, use exact sum/count state and compare against -Prometheus at identical timestamps. Verify both numerical/label equivalence -and one maintained producer shared by the two roots. Exact aggregate state -does not eliminate the separate requirement to verify data completeness. - -Approximate extensions must declare what epsilon measures and what delta -covers. For a whole 20-row result with failure probability at most 0.05, -20 valid per-row failure bounds of at most 0.0025 suffice by the union bound; -independence is not required. Per-row 95% intervals alone do not establish -95% confidence for the complete result. Multiple dashboard evaluations need -their own declared scope; a result-level guarantee is not automatically -session-wide. Shared state also does not make separate errors independent. - -## Capabilities, costs and feedback - -Capabilities answer **can this deployment faithfully execute this alternative?** -Costs answer **which feasible alternative is preferable?** - -| Capability question | Why it constrains selection | +SDS defines semantic identity, format, coverage and version validation. The +installed bundle also binds the selected maintenance schedule, retention and +unavailability policy; those fields are omitted here to show the shared-output +connection clearly. DAG references resolve within the installed bundle, not to +live Planner objects. + +For `(12:00, 12:05]`, the query engine resolves five one-minute records for the +requested group, validates their format, coverage and revision compatibility, +and supplies them to the query DAG. Merge runs once for its two consumers within +that run. Separate query runs do not implicitly share mutable execution state. + +Each maintained pane contributes once. Replacing a pane snapshot does not add +the same input samples again to a query merge. Missing, overlapping or incomplete panes +cannot be treated as the requested complete range. + +A delayed build keeps its original coverage interval; publication time does not +change query semantics. If required state is missing, the installed fallback or +unavailability policy applies. The backend must not substitute older state or +change the maintained window to make a read succeed. + +## 5. Deployment compilation contract + +Inputs are selected Physical DAGs and boundaries, the selected lifecycle, +query associations, source/store capabilities and installation context. +Compilation opens no readers and does not establish future state readiness. + +For each selected physical candidate, the compiler: + +1. Verifies that the backend runtime can supply every input and fulfill the selected + maintenance requirements without changing their semantics. +2. Binds raw inputs and assigns identities to persisted physical outputs. +3. Connects stored-state inputs to those outputs, with matching definitions, + grouping, coverage rules, revision scope and supported format. +4. Binds schedules and retention that satisfy the selected lifecycle, then + packages the provided DAGs and bindings into precompute/query plans. +5. Validates the complete bundle before it can be staged. + +Backend feasibility includes persisting the selected output type. Planner may +produce scalar/result frontiers as well as sketches; this does not imply the +backend supports all of them. An unsupported output is rejected or excluded +during Backend candidate selection, never silently replaced with another +frontier. + +A query-only candidate can build state during a query; a precompute candidate +can finalize values before persisting them. The backend follows the selected +Physical DAGs rather than enforcing build-only/estimate-only phase rules. + +## 6. Installation and execution contracts + +| Contract | Requirement | | --- | --- | -| Can the producer build/update the selected family and parameters? | A readout implementation alone does not make a state maintainable | -| Can storage and readout preserve the selected windows and labels? | A tumbling-only path cannot silently implement arbitrary sliding coverage | -| Are merge operations and full/delta encodings compatible? | Distributed producers must construct the same logical state | -| Can the runtime perform every exact operator after readout? | A supported sketch is insufficient for an unsupported full expression | -| Can readiness, staleness and exact fallback be enforced? | Mathematical legality does not establish runtime answerability | - -Costs include initialization, ingestion updates, overlapping/retained state, -transmission, storage, merges, readouts, recurring queries, and shared producer -construction once. Compare alternatives over the same data and demand scope. -Missing evidence is not zero cost; stale or incomplete implementation evidence -cannot justify selection. - -Runtime observations reference the concrete binding and selected semantic -producer. Physical controls may vary only within already-authorized -guardrails. Changing grouping, family, parameters, windows or sharing returns -to planning. - -## Reuse across various ASAP workload scenarios - -| Scenario | Reusable Planner strategy | Application-specific responsibility | -| --- | --- | --- | -| Repeated dashboards (MetricsObservabilityQuery) | Shared aggregates and prepared/maintained state | PromQL semantics, freshness and serving | -| Multiple dashboard resolutions | Legal rollup and window alternatives | Compatible retention and exact time coverage | -| Distributed telemetry aggregation | Mergeable summary and grouping alternatives | Collector placement, transmission and activation | -| DQC analytical workloads | CSE, aggregate fusion and rollup | DQC engine adapters and batch execution policy | - -General semantic rules belong in Planner. Backend-local metric-name fixtures, -SID lookup or deployment-specific placement must not become universal Planner -rules. No dependency on DQC is needed to reuse strategies contributed by it. - -## High-level migration - -See the [migration delivery plan](asapplanner-migration-plan.md) for PR-sized -implementation slices, dependencies, regression fixtures and completion gates. - -| Milestone | System outcome | Acceptance | -| --- | --- | --- | -| 1. Audit the shared contract and entry points | Current canonical compilation and compatibility paths have explicit ownership | Document supported operators, sharing scope, profile limits and true IR gaps | -| 2. Complete one workload-wide semantic path | Registered queries use Planner alternatives with preserved shared producers | The two-query example has one selected producer and both result roots | -| 3. Preserve bindings through all projections | Precompute, storage and serving implement the same selected decision | No duplicate maintenance; exact state/schema/window and generation agreement | -| 4. Consolidate reusable strategies | Missing general fusion/rollup rules extend Planner | Rules work without backend metric names, SID objects or placement assumptions | -| 5. Close capability and cost feedback | Only fully executable, properly costed alternatives are committed | Unsupported or stale evidence fails closed; estimated and observed costs are traceable | -| 6. Validate profiles and retire redundant selection paths | Serving executes installed bindings without independent semantic planning | Prometheus parity, sharing, readiness, fallback and activation-failure tests pass | -| 7. Broaden coverage (ProjectASAP-wide; not required for this repository) | Other applications, engines, sketches and lifecycles reuse the contract | Each participating provider demonstrates capability and semantic conformance | - -The first milestone demonstration should use backend-local ingestion and the -exact two-query example. Distributed rollout follows the same contract with -additional producer and activation checks. Existing paths may remain as -comparison baselines until parity is established; remove duplicate semantic -selection, not necessary physical plans or profile-specific runtime adapters. - -Step 7 is an ecosystem extension, not a prerequisite for completing this -backend's scoped consolidation through steps 1–6. - -## Related contracts and implementation guides - -- [Physical compiler](../developer_docs/control-plane/physical-compiler.md) -- [Plan publication](../developer_docs/control-plane/plan-publication.md) -- [Catalog-backed physical-plan runtime](../developer_docs/query-engine/catalog-physical-plan-runtime.md) -- [ASAPQuery compatibility profile](asapquery-compatibility-profile.md) -- [Runtime accuracy feedback](../developer_docs/control-plane/runtime-accuracy-feedback.md) +| Preserve computation | Binding does not change physical operators, ordered edges, roots or sharing. | +| Bind completely | Every required boundary resolves to one compatible input/output contract. | +| Install atomically | Definitions and both plans become active as one version; failed staging leaves the previous version active. | +| Distinguish readiness | Installation authorizes a plan; actual state coverage and readiness are checked when resolving inputs. | +| Execute once per run | Shared physical producers are driven by the shared runtime, not duplicated by separate backend traversals. | +| Publish consistently | Stored metadata and payload become visible together under the authorized output binding. | +| Fail explicitly | Unsupported bindings or unreadable state follow rejection, fallback or unavailability policy without changing computation. | + +The precompute engine schedules work, resolves bounded inputs, invokes the shared +executor and commits output. The query engine resolves request-specific inputs, +invokes the same executor and adapts results. Both propagate cancellation and +resource limits. Neither interprets logical Post-ASAP nodes at runtime. + +Cleanup respects retention and active readers/dependent producers. Storage lookup +uses installed references; it does not search for a substitute summary at +serving time. See SDS for record eligibility and recovery requirements. + +Initial recovery is limited to the same installed plan version. A new version +populates its own state, even when definitions match the previous version. During +warm-up it uses its installed fallback or unavailability policy. Cross-version +state adoption is deferred; equal definitions do not authorize it. See the +[SDS recovery contract](summary-catalog-sds-architecture.md#recovery-and-plan-version-changes). + +## 7. Design choices and tradeoffs + +Re-lowering logical nodes in the backend would duplicate physical selection and +allow deployment and Planner graphs to drift. Consuming Physical DAGs avoids that +second compiler, at the cost of requiring an explicit capability/replanning +boundary when the backend cannot realize a candidate. + +Keeping one full logical DAG under PrecomputePlan would require runtime phase +filtering and obscure which inputs are stored. Separate Planner-provided physical +subgraphs make execution ownership explicit without inventing separate operator +systems for precompute and queries. + +A separate catalog Materialization object would repeat fields already owned by +definitions, boundary bindings and stored records. Two stored object types and +plan-local references are sufficient for the selected scope. + +## 8. Validation and acceptance + +Tests must establish: + +1. Deployment binding preserves Planner's operators, boundaries and shared + dependencies; unsupported bindings fail before activation. +2. The KLL example summarizes each pane’s input samples once and serves both readouts with + one merge per shared run. Missing/overlapping panes and incompatible revisions + fail read eligibility. +3. A supported query-only build and precomputed readout/result follow their + selected phases. Unsupported persisted types are rejected explicitly. +4. Multiple queries can reference one producer, and one query can consume multiple + compatible outputs. Derived maintenance checks its source completeness. +5. Compilation, installation and runtime agree on identity, schema and version. + Staging failure, restart and version switching preserve consistency. +6. The complete path runs without an ASAPCollector checkout or process. + +These are acceptance requirements, not claims of completed deployment tests. +The [migration plan](asapplanner-migration-plan.md) defines delivery gates. + +## 9. Scope and follow-up work + +Backend work binds and operates Planner computation. It does not add an execution +IR, alter the Planner API's ownership, or introduce another maintenance model. +Distributed activation, Collector and transmission plans, and new checkpoint +protocols remain separate work. Changes to physical algorithms or materialization +frontiers belong in Planner and its shared physical library. + +A future unregistered-query path may ask Planner to search available SDS +definitions and rewrite the query over reusable state. Backend then resolves +authorized outputs and binds the selected Physical DAG normally. This is +planning before execution, not substitute-summary search inside an installed +reader. See [SDS semantic discovery](summary-catalog-sds-architecture.md#7-future-discovering-sds-for-an-unregistered-query). +It remains outside the initial deployment rollout. diff --git a/docs/design_docs/asapplanner-migration-plan.md b/docs/design_docs/asapplanner-migration-plan.md index babe652cc..0c51cda9a 100644 --- a/docs/design_docs/asapplanner-migration-plan.md +++ b/docs/design_docs/asapplanner-migration-plan.md @@ -1,313 +1,188 @@ -# ASAPPlanner integration: migration delivery plan - -Status: implementation sequence for the -[system architecture proposal](asapplanner-integration.md). A checked milestone -requires executable evidence; publishing this plan or opening a PR does not -complete migration. - -## Baseline and completion definition - -The inspected baseline is backend `95131d83972bb7a07d338e2a5af925a20c15ddce`. -The compiler already deduplicates backend PrecomputePlan state by physical -fingerprint and binds QueryPlan leaves explicitly. It still builds Collector -materialization declarations per query, and lifecycle selection builds a -single-query demand. Therefore, do not describe all sharing as absent, or -treat existing fingerprint deduplication as workload-wide optimization. - -Migration is complete for a declared supported workload/profile when: - -- one Planner-authorized semantic decision governs all result roots; -- compatible shared producers have one physical maintenance path per source - partition and generation; -- unsupported sharing or operators are rejected or explicitly fall back; -- activation, readiness, query execution and feedback refer to matching - bindings and generations; -- supported entry points no longer independently select a different summary; -- parity and producer-update tests pass for the promised deployment profile. - -Backend-local and distributed profiles have separate acceptance evidence. -Neither arbitrary PromQL coverage nor ProjectASAP-wide engine coverage is a -completion prerequisite. - -## Delivery sequence and dependencies - -Implementation tracking (PRs are not merged automatically): - -| PR | Implemented scope | -| --- | --- | -| [Backend #513](https://github.com/ProjectASAP/ASAPQuery-backend/pull/513) | A: compatible physical producer deduplication and conflicting deployment-contract rejection | -| [Backend #514](https://github.com/ProjectASAP/ASAPQuery-backend/pull/514) | B prerequisite: port the backend from its divergent historical pin to merged Planner APIs, including typed summary inputs | -| [Planner #356](https://github.com/ProjectASAP/ASAPPlanner/pull/356) | B: reusable, scope-local typed post-ASAP subtree interning; includes schemas and guarantees in equivalence | -| [Backend #515](https://github.com/ProjectASAP/ASAPQuery-backend/pull/515) | B: workload search, shared producer bindings and persistent query-root mapping | -| [Backend #516](https://github.com/ProjectASAP/ASAPQuery-backend/pull/516) | C: backend-local packed SUM/observation-count state, exact readouts, additive reductions and constrained arithmetic; production HTTP acceptance | -| [Backend #517](https://github.com/ProjectASAP/ASAPQuery-backend/pull/517) | E: current distributed publication/frame protocol, actual Collector validator, two shared readouts, failed staging and inactive-generation rejection | -| [Backend #518](https://github.com/ProjectASAP/ASAPQuery-backend/pull/518) | B/F: one workload-selection adapter for canonical startup and compile-and-publish; query-scoped accuracy certificates | -| [Backend #519](https://github.com/ProjectASAP/ASAPQuery-backend/pull/519) | D component: joint producer lifecycle demand, incompatible-evidence rejection and identity-keyed lifecycle estimates | -| [Backend #520](https://github.com/ProjectASAP/ASAPQuery-backend/pull/520) | E: published config drives the actual Collector Rust update/window/emission loop; N raw observations yield N updates and one shared output | -| [Backend #521](https://github.com/ProjectASAP/ASAPQuery-backend/pull/521) | E: failed staging cleanup permits retry; concurrent readers survive successful same-semantic generation cutover; retired frames are rejected | -| [Backend #522](https://github.com/ProjectASAP/ASAPQuery-backend/pull/522) | D: provider-priced complete bound-workload selection, strict v2 startup evidence, read-only quote preparation, live publication/reporting and process acceptance | - -The backend PRs form a sequential review stack from #513 through #522; -#515 uses merged Planner #356 at revision -`378a7547ede629a64e84c9f7c810226ce196cce9`. #516 includes the fail-closed -arithmetic regression fix, propagated through its dependent branches. -The backend-local dashboard and distributed single-partition quantile examples -have executable acceptance evidence, including complete cost-based selection -and same-semantic generation cutover. The supported-profile implementation -is in the review stack, not yet merged or deployed. Production calibration, -platform-specific rollout and broader semantic workload replacement are not -claimed complete by these fixtures. - -Local verification of the original combined migration stack: 654 control-plane -library tests, 28 control-plane binary tests, one control-plane integration -test, 977 data-plane library tests and three production-process tests passed. Planner -#356 passed its 156 type-library tests and GitHub formatting/lint/test checks. -The backend process tests cover the actual binaries and Collector Rust library, -not production traffic or every Collector platform adapter. Local passes do -not replace PR CI, review or the remaining migration gates. - -| Slice | Repository | Depends on | Deliverable and acceptance | -| --- | --- | --- | --- | -| A. Safe physical state sharing | ASAPQuery-backend | Existing compiler | Deduplicate Collector declarations for compatible state; reject conflicting implementation/layout/lifecycle contracts; keep both query roots bound to one backend state | -| B. Workload semantic planning adapter | ASAPQuery-backend, with Planner changes only for demonstrated gaps | A and Planner API audit | Batch registered canonical roots through reusable Planner search; preserve root mapping and producer identity; do not implement backend-local semantic CSE | -| C. Aggregate-state fusion and readouts | ASAPPlanner for rules; backend for execution | B | SUM/COUNT example with per-consumer projections, label/time equivalence and fully executable division; reuse existing decomposition/rollup rules | -| D. Workload-wide implementation evidence | ASAPQuery-backend and Planner evidence boundary | B; C for fused states | Compare complete alternatives with shared build/update cost once and per-consumer read costs; joint state lifecycle/implementation agreement | -| E. Bound execution and lifecycle acceptance | ASAPQuery-backend; Collector only where public runtime gaps require it | A–D | Producer update counts, readiness/fallback, generation isolation, failed rollout, and distributed projection tests | -| F. Compatibility-path retirement | ASAPQuery-backend | E for each affected profile | Route supported entry points through the validated path; remove duplicate selection only after call-site and parity audit | - -Slices are reviewable PR units, not an instruction to open empty placeholder -PRs. If a slice spans semantic changes and physical execution, split by -repository and stack the dependent PR explicitly. Do not merge automatically -or make one unverified pin bump cover unrelated Planner changes. - -## A. Safe physical state sharing - -The immediate regression fixture is two different quantile readouts over the -same source, parameters and window. It exercises existing supported operations -without depending on future SUM/COUNT fusion. - -Implementation scope: - -1. Compare concrete contracts when multiple selected leaves resolve to the - same physical fingerprint. Include algorithm/parameters, grouping, window - framework, implementation, pane layout and lifecycle. Runtime transmission - policies must also agree. -2. Emit one Collector producer declaration for a compatible shared state while - preserving every query's binding and readout. -3. Keep evidence conservative: differing evidence cannot silently disappear - during deduplication. A future certificate-union design is a separate step. -4. Reject conflicting contracts before any plan is published. Do not pick - whichever query happened to be visited first. - -> Historical note: this acceptance text predates the SummaryCatalog migration; -> the former BackendPlan state is now represented by a catalog materialization -> and its execution-plan references. - -Acceptance: both query roots exist; one catalog materialization and one PrecomputePlan -state exist; each Collector has one producer declaration; both bindings point -to that state. A different implementation/layout for the same fingerprint -fails compilation. Distinct source/window/parameters must remain distinct. - -This slice establishes deployment consistency, not workload search or a claim -that all query-time computations execute once across separate HTTP requests. - -## B. Workload semantic planning adapter - -Audit the pinned Planner workload/search APIs before defining another backend -plan representation. Inputs must preserve canonical query identity, source -selection, requirements, recurrence and time scope. - -The result must retain all original roots and shared logical producers. -Backend bindings must be keyed by workload-scoped producer identity, not only -a per-query pointer. Physical IDs stay downstream. Preserve explicit mappings -from each query root to its required materializations and fallback. - -Acceptance fixtures: - -- identical producers used by two different roots; -- a diamond within one query and sharing across queries; -- incompatible filters, grouping, windows or accuracy do not share; -- round-trip compilation retains roots and sharing; -- unsupported alternatives cannot become partially executable warm routes. - -Pointer sharing in memory alone is not persistent identity. A serialized -execution projection must preserve the relationship explicitly. - -## C. Aggregate-state fusion and complete readouts - -Use the system document's sample-weighted mean example as the target. -Planner owns the equivalence rule: union compatible SUM/COUNT states and -project the needed results to consumers. The backend owns physical state -implementations and exact output operators. - -First inspect existing AVG decomposition, CSE and rollup rules. Add only missing -semantics upstream; do not copy DQC transformation objects or hard-code metric -names in Planner. - -Acceptance includes uneven per-instance sample counts, missing/stale series, -multiple services, exact interval endpoints, range evaluation steps and -denominator edge cases. Query results must match Prometheus labels, timestamps -and numeric semantics. Until the whole expression is supported, preserve -explicit fallback rather than claiming partial integration. - -The implemented backend-local example uses one raw accumulator that retains -both sum and observation count. This is native physical packing of selected -Planner operations, not a new backend semantic rewrite. The process test has -two services: observations `[10]` and `[2, 4, 8]` across two API instances give -SUM = 24, COUNT = 4 and weighted mean = 6; worker observations `[9, 15]` give -SUM = 24, COUNT = 2 and mean = 12. Three registered consumers still configure -one producer; a Remote Write retry does not double the counts. Range steps, -output labels/timestamps and unaligned-window fallback are checked. - -Do not generalize that execution contract to `sum(sum_over_time(m) / -count_over_time(m))`: summing per-instance means cannot pool samples first. -Non-additive entity reduction, mismatched operand grouping/windows, shifted -selectors and unverified instantaneous/temporal combinations remain explicit -fallbacks. Unknown legacy observation counts also fail closed. Distributed -observation-count readout is not advertised by this implementation. - -## D. Workload-wide evidence and selection - -Today per-query lifecycle inputs are not proof of joint workload costing. -Aggregate demand for each shared producer while retaining consumer-specific -requirements. Compare alternatives over one horizon and data scope. - -Charge shared initialization and maintenance once, account for all consumer -readouts and live/retained state, and include applicable placement and -transmission costs. Feasibility checks cover the entire selected DAG, not -only a summary family. The winning evidence must resolve to the same concrete -implementation that compilation installs. - -Acceptance: a shared alternative wins when its complete cost is lower, loses -when retention/materialization overhead dominates, and is unavailable when -any required capability/evidence is absent or stale. Adding another consumer -must not double-count the producer's update stream. - -Implemented component: #519 gives each unique physical producer a -`WorkloadDemand` containing all its consuming query entries. For a 300-second -horizon, 100 updates/second and two consumers reading every 10 and 20 seconds, -the demand is 30,000 updates and 45 reads. With build = 10, update = 0.001, -read = 0.1, retention/second = 0.001 and retirement = 1, the lifecycle cost is -45.8. Adding the second consumer increases cost by 1.5, not another build and -update stream. Publication reports this component against the materialization -and implementation identities; it is not a complete-plan total. - -Implemented selection: #522 compares complete bound alternatives before -commitment. A provider prices source upkeep, each shared state's build/update/ -residency/retirement per location, transport, every reachable query operator, -and results over one common horizon. Query work is multiplied by recurrence; -shared maintenance is not multiplied by consumer count. Native exact fallback -includes its service's input upkeep as well as full native query execution. - -The default inventory is the Planner-selected continuously maintained workload -and its whole-workload exact alternative. The comparison interface also accepts -additional Planner-authorized, bindable forests; this is not exhaustive search -over all engines or lifecycle variants. Tests prove both the sharing win and -high-retention loss, and reject missing, stale, mismatched or infeasible quotes. - -Implementation refinement: pricing uses a flat coverage manifest over the -existing bound physical projection, not another semantic DAG. It does not -populate `PlannerPhysicalPlanProvider` with guessed source statistics or split -the older opaque per-query window scalar into fabricated components. Providers -must quote the actual source scope, state layout, implementation and capability -generation. The selected plan and report retain those identities. - -Version-2 canonical snapshots require complete evidence. Live requests can -obtain requirements from the read-only `cost-manifests` endpoint before -publication. Version 1 and live requests without quotes remain explicitly -uncosted compatibility paths. See the [provider workflow in #522](https://github.com/ProjectASAP/ASAPQuery-backend/blob/feat/complete-workload-cost-selection/docs/examples/workload-cost-evidence.md). - -Production calibration still requires evidence from the intended deployment; -the deterministic fixture costs are not production measurements. The provider -attests exact-backend access and resource feasibility; a low cost alone does -not establish either. - -## E. Runtime and deployment acceptance - -Start backend-local, then validate the distributed profile independently. - -- Replay deterministic raw samples through production ingestion. -- Count state creation and updates: one compatible producer per generation, - with no duplicated updates when a second query subscribes. -- Query both roots through HTTP and compare with an exact reference. -- Test incomplete coverage, stale state, absent routes and unavailable fallback. -- Stage a successor while requests run; each request observes one generation. -- Fail staging or producer acknowledgement and verify the active generation - remains unchanged. -- For distributed collection, decode emitted plans through the actual Collector - validator and assert one producer per source partition, not one producer - globally across independent sources. - -Unit-level declaration counts do not replace runtime update-count tests. - -Current evidence combines real backend executables with the actual Collector -Rust runtime library. The test's host adapter supplies OpAMP acknowledgements -and frame metadata; it does not launch a platform-specific Collector binary. -In #521, failed Collector staging is discarded without touching the active -snapshot; the same successor version can then be retried successfully while -queries run. Old-generation frames are rejected after cutover and successor -frames become queryable. #522 exercises this flow with costed publication. -This verifies same-semantic runtime generation replacement, not arbitrary -semantic workload replacement or a platform-specific production rollout. -Platform adapter rollout remains a deployment acceptance step. - -## F. Retire duplicate selection safely - -Inventory canonical startup compilation, explicit compile-and-publish, -legacy workload adapters and serving-time binding helpers. Distinguish dead -code from intentionally supported profiles using call-site inspection. - -For each path, either route it through the selected workload contract, retain -it as an explicitly unsupported/fallback adapter, or remove it after parity. -Parsing and canonicalization at serving time are fine; family/parameter, -grouping or lifecycle reselection is not. - -Do not remove QueryPlan, PrecomputePlan, physical deployment selection, -exact fallback, or profile-specific adapters merely because their types are -different from post-ASAP IR. - -Call-site audit: production instant/range serving already requires an active -physical QueryPlan and declines absent or unregistered routes. The old -summary-selection serving branches in `engine.rs` are `cfg(test)` fixtures. -#518 unifies the two first-class compilation entry points. Legacy flat-workload -demo/configuration adapters remain separate compatibility paths; they must not -be presented as migrated canonical-workload entry points or removed without -their own parity/retirement decision. - -## Existing PR coordination - -At the baseline inspection, open PRs -[#505](https://github.com/ProjectASAP/ASAPQuery-backend/pull/505), -[#506](https://github.com/ProjectASAP/ASAPQuery-backend/pull/506), -[#509](https://github.com/ProjectASAP/ASAPQuery-backend/pull/509) and -[#511](https://github.com/ProjectASAP/ASAPQuery-backend/pull/511) cover PromQL, -process-E2E and TopK-related work. Re-check their status and changed files -before touching overlapping paths. Their presence is not evidence that the -workload-sharing migration is complete. - -Review follow-up (2026-09-08): #505 is now stacked on #522 and uses the merged -Planner revision above. Typed TopK update weights belong to the selected -producer, not its readout. Its multi-series fixture distinguishes count ranking -(`api=4`) from value ranking (`worker=200`). #509 compares complete vectors at -each range step, including changing winners. #506 tests unregistered-query -fallback; it is not evidence that registered arithmetic is unsupported. - -#515 preserves duplicate algorithm candidates during cost ranking; removing -them violates Planner's candidate-multiset contract and can panic. #522 quote -preparation enumerates bindable alternatives without requiring the default -warm alternative to compile, so missing warm implementations do not hide an -available exact quote. Publication still requires a selected, validated plan. - -#511 retains evidence-aware legacy binding and preserves count update semantics -in emitted heap configuration. Its two heap TopK acceptance tests now use -registered `topk(3, count_over_time(top_endpoint_qps[5s]))`, a compiled physical -QueryPlan, and the production backend-local Remote Write path. Both CMS-with-heap -and CountSketch-with-heap return gamma=200, zeta=150 and alpha=100 over two -windows, with exact item identities, timestamps and retry deduplication checked. -Unregistered instantaneous TopK still follows the explicit exact fallback. -This replaces the two obsolete no-QueryPlan tests; it does not restore that -serving contract or claim migration of other legacy OTLP fixtures. - -The [architecture PR #512](https://github.com/ProjectASAP/ASAPQuery-backend/pull/512) -tracks the design and this delivery plan. Implementation PRs should report the -slice they complete, tests actually run, and remaining acceptance gaps. +# Migration to Planner Physical DAG Deployment + +Status: delivery plan for the [target design](asapplanner-integration.md). +Audience: backend implementers. This plan does not introduce a second operator +IR or backend lowering path. + +## 1. Outcome + +PrecomputePlan and QueryPlan carry Planner-compiled Physical DAGs and backend +boundary bindings. Both engines execute through the shared physical library. +The backend owns storage, scheduling, installation and serving; Planner owns +operator selection and materialization frontiers. + +The migration also removes the backend dependency on ASAPCollector. Distributed +activation, CollectorPlan/TransmissionPlan compilation and new transport +protocols are outside this delivery. + +## 2. Migration boundaries + +```text +Before + full logical DAG + backend semantic-node bindings + → backend-specific computation and phase interpretation + +After + Planner-provided maintenance/query Physical DAGs + → backend input/output bindings and operational policy + → shared physical execution +``` + +The runtime accepts the new deployment artifact. Obsolete plan schemas are +rejected before activation rather than interpreted through a parallel logical-DAG +executor. Producers and fixtures move together. + +Plan-format migration is separate from stored payload compatibility. Supported +historical payloads retain versioned decoders and fixtures; this does not require +retaining obsolete plan readers. Do not change sketch byte formats as a side +effect of moving execution code. + +## 3. Delivery stages + +| Stage | Work | Exit condition | +| --- | --- | --- | +| Inventory | Record current supported computation, state formats and deployment behavior. | Each supported path has a fixture or an explicit unsupported result. | +| Shared dependencies | Adopt shared operator/runtime and codec contracts; remove Collector dependencies. | Backend builds and tests without ASAPCollector. | +| Deployment binding | Consume Planner Physical DAGs; bind their typed boundaries and lifecycle. | No backend logical lowering or frontier selection remains in the new path. | +| Execution and installation | Drive both kinds of DAG through the shared executor and install one coherent bundle. | Identity, resource, failure and readiness tests pass. | +| Retirement | Switch publications and remove superseded computation paths. | Full-path and recovery tests pass; obsolete plans are rejected. | + +### 3.1 Inventory and shared dependencies + +Capture fixtures for full/delta decoding, reconstruction, maintenance and +readout, completion, restart, staging, activation and fallback. Record revision +and schema provenance. Use semantic checks when randomized bytes are unstable. +Fixtures may originate from Collector but must run independently of it. + +Reuse `asap-physical-operators`, `asap_sketch_codec` and sketch-library APIs for +neutral execution and encoding work. Storage adapters, scheduling and publication +remain backend-owned. Remove reconstruct-serialize-decode detours and duplicate +family execution paths when replacing them, with parity evidence. + +Inspect manifests, lockfiles, build scripts and tests for direct or transitive +Collector dependencies, including `asap-precompute-rs` and Collector patches. + +### 3.2 Deployment binding + +Adopt the Planner-owned semantic-description export and versioned canonicalization +contract for SDS definitions, independent of internal executable IR serialization. +Persist only the semantic dependency closure needed to interpret each output before records +can reference semantic fingerprints. Definitions derived from incomplete legacy +metadata must be reconstructed from authoritative plans or rejected for rebuild; +do not infer missing expressions from source and grouping alone. + +Supply stable logical dataset identities to Planner before semantic export, and +validate that concrete source bindings realize those identities. Different datasets +must not acquire equal definitions merely because expressions use the same names. + +Planner exposes physical workload candidates. Backend evaluates binding feasibility +and scoped costs before selecting a deployment; it does not rewrite candidate DAGs. +The initial tests inject synthetic costs. Online measurements and feedback-driven +replanning remain deferred. + +Consume the selected Physical DAGs, physical boundary identities, query +associations and maintenance requirements. Replace semantic-node classification +with mappings from declared input/output boundaries to deployment resources. + +- Bind raw slots to readers satisfying source, filter, grouping, window, schema and boundedness requirements. +- Assign version-scoped stored-output identities to persisted physical outputs. +- Bind stored inputs to matching outputs and validate grouping, format, coverage + and revision requirements. +- Package the original physical computation with schedules, retention, result + routing and publication policy. + +The old `Materialization`, `MaintenanceInput`, `Query` and `QueryInput` semantic +classification is not a target contract. Logical provenance is diagnostic data +from Planner, not an instruction to rebuild operators or split a graph. + +No build/readout phase whitelist is introduced. Follow the selected physical +candidate. If the backend cannot persist a selected scalar or result output, +report that capability limitation instead of moving operators across a boundary. + +A new plan schema version expresses this boundary. Do not reinterpret an old +field under an unchanged version. Normalize supported legacy stored identities +during migration with an explicit mapping; preserve payload identity and reject +unresolved/conflicting mappings. + +### 3.3 Installation and execution + +Validate definitions and boundary bindings against the supplied Physical DAGs. +Verify all stored-output references, schemas, encodings, partitions and versions, +then perform deployment resource and capability checks. + +Stage definitions and both plans as one snapshot. Failed staging leaves the +active version unchanged. Activation does not establish state readiness; runtime +input resolution checks actual committed state and applies the installed fallback +or unavailability policy. + +Precompute and query engines resolve inputs and drive the shared executor. They +must not retain a second node traversal that recomputes shared producers. Plan +visualizations show the supplied DAGs connected by deployed stored-output bindings. + +### 3.4 Retirement + +Migrate publishers and consumers together with pinned dependencies and matching +rollback artifacts. Remove obsolete plan adapters, full-logical-DAG execution +and duplicated operators after the new path passes its gates. + +Storage payload readers remain governed by the supported format policy. Initial +recovery supports the same installed plan version. New versions populate +their own state and use their installed fallback/unavailability policy during +warm-up. Cross-version state adoption is deferred independently of binary rollback. + +## 4. Acceptance evidence + +Record tested revisions, supported families/output types and fixture results. +Acceptance includes: + +- Planner computation and boundaries are preserved through installation. +- One producer serves multiple queries without duplicate maintenance; one query + can consume multiple compatible outputs. +- Shared producers execute once per run; separate runs remain isolated. +- Supported query-time construction and precomputed finalized outputs follow the + selected phases; unsupported output bindings fail explicitly. +- Missing, overlapping, incomplete or incompatible state fails eligibility. +- Dataset identity changes alter definitions; endpoint/replica changes do not. +- Query branches preserve the whole-query revision fence during publication. +- Same-version recovery validates bindings and completeness; new-version reads + never silently adopt old state and follow warm-up failure policy. +- Staging failure, cancellation, resource limits, restart and version switching + preserve documented behavior. +- Obsolete plans are rejected and backend builds/tests do not require Collector. + +Trace a query from Planner candidate construction through Backend selection, deployment +binding, state publication and query execution. Verify exact operations against +independent results and sketches against their supported guarantees. The design +is not accepted solely because example schemas parse or unit tests pass. + +## 5. Bound-query SDS implementation across the PR stack + +The SDS contract separates semantic identity from deployed-output identity. +The bound-query path locates state by plan version, output and group, then +selects its time range and validates semantics, format, revision and coverage. +Ad-hoc discovery is deferred. + +| Implementation owner | Required change | Regression/acceptance gate | +| --- | --- | --- | +| Planner shared types and physical integration (#462) | Export a versioned canonical semantic description for a selected persisted output; exclude placement and temporary node IDs. | Different input expressions differ; renumbering preserves identity; state definitions exclude downstream readout parameters. | +| Backend plan/schema foundation (#749), completed with the shared semantic contract in #774 | Separate semantic definitions from deployed-output bindings; remove the requirement that stored-output ID equals definition ID; version the changed plan contract. | Same-version hot/rebuild outputs can share one definition without aliasing; tampered definitions and mismatched bindings fail installation. | +| Planner dependency integration (#774) | Consume the shared semantic export and propagate it from selected physical outputs into deployment compilation. | No backend expression normalization or synthetic semantic fingerprint from incomplete config fields. | +| Precompute/storage integration (#763) | Persist definitions and output-scoped records; authorize writes against installed bindings and recover them consistently. | Restart retains semantic descriptions; wrong-output writes fail; replacement metadata and payload remain consistent. | +| Query integration (#765) | Resolve the installed deployed output and validate definition, revision, format and coverage before invoking shared execution. | A hot-bound query never reads rebuild state; stale, missing or incompatible records take the explicit failure route. | +| Acceptance PRs (#728, #742, #775) | Update fixtures and process tests for the new contract; validate individual queries and ensembles using synthetic costs. | End-to-end producer → persisted definition/record → recovery → bound read, with negative identity and coverage cases. | + +These are implementation responsibilities and acceptance gates. PR ordering +must follow actual dependency commits, not an outdated stack list. +A semantic definition cannot be replaced by a policy fingerprint containing +physical layout or cadence. Conversely, relaxing an output-reference validator +without changing storage keys and authorization is insufficient and unsafe. + +The implementation must preserve supported payload decoders independently of +plan-schema retirement. Keep implementation guides accurate to the code until +each stage lands; then update the APIs, persistence descriptions and test evidence +in the same implementation PR. + +The open shared-library integration PR is #774, replacing the already merged +#770. The active order after #771 is #774 → #763 → #765 → #761 → #728 +→ #742 → #775. Real-evidence work in #776, #777, #778 and #759 is deferred; +old #770 base metadata is not part of this chain. diff --git a/docs/design_docs/planner-backend-glossary.md b/docs/design_docs/planner-backend-glossary.md new file mode 100644 index 000000000..3d86abf29 --- /dev/null +++ b/docs/design_docs/planner-backend-glossary.md @@ -0,0 +1,82 @@ +# Planner/backend design glossary + +For developers reading the [integration](asapplanner-integration.md), +[SDS](summary-catalog-sds-architecture.md), and +[migration](asapplanner-migration-plan.md) designs. Definitions describe the +proposed boundary; they do not imply that every proposed field already exists +in the serialized API. + +## Computation and execution + +| Term | Meaning | +| --- | --- | +| Selected post-ASAP DAG | The Planner-produced logical computation underlying the physical candidate selected by Backend, including summary producers, shared dependencies and query readouts. | +| Physical DAG | Planner-owned concrete operators, typed input boundaries, dependencies and roots; no storage identities or placement. | +| Physical candidate | A Planner-produced physical realization of a workload; Backend checks deployment feasibility and selects using scoped costs. | +| Deployment plan | System instantiation of physical computation with concrete source/state bindings and operational policy. | +| Summary producer | An operation or subgraph that builds summary state. Multiple queries may share its stored output. | +| `SummaryMaintenanceLifecyclePlan` | Planner result associating a post-ASAP root with selected maintenance requirements for its unique reachable summary producers, plus workload and costing context. | +| Selected deployment guarantee and schedule/retention | The selected `SummaryMaintenanceLifecycleGuarantee` for one producer, together with its concrete scheduling and retention binding. Planner supplies the selected lifecycle requirements; backend scheduling and retention must realize them without changing their semantics. | +| Maintenance | Work that constructs, refreshes or derives stored summary state, including batch rebuilds and incremental updates. | +| `PrecomputePlan` | Planner maintenance Physical DAGs plus backend input/output bindings, scheduling and publication policy. | +| `QueryPlan` | Planner query Physical DAGs plus backend input bindings, query/result associations and fallback policy. | +| Readout / `SummaryEstimate` | Operation that obtains a query value from summary state, such as p99 from KLL. | +| Derived summary state | Stored summary state computed from existing summary states. Earlier discussion calls this a “derived materialization”; it does not require a separate catalog object. | +| Exact computation around summaries | Part of the selected query computed exactly around summary operations, such as supported filtering or arithmetic after readout. It does not make the whole approximate result exact. | +| Exact fallback | Configured execution of the original query through an exact route when the summary plan cannot serve it. | + +For example, merging five compatible one-minute KLL summaries and storing the +five-minute result produces derived summary state in a separate destination +stored output. Merging them only to answer a query is a query-time operation. Both require +compatible grouping, coverage and accuracy. + +## State and identity + +The storage contract has two stored objects: `SummaryDefinition` and +`StoredSummary`. `StoredOutputReference` belongs to installed boundary bindings, +not a third storage table. Metadata and payload form one logical stored record. +Migration of existing types and fields is covered in the +[migration plan](asapplanner-migration-plan.md). + +| Term | Meaning | +| --- | --- | +| `summary_definitions` | Logical table inside `SummaryStore`: semantic fingerprint → persisted canonical semantic description. The compiler supplies a snapshot for validation and registration during installation. | +| `stored_summaries` | Logical table inside the same store: `(plan_version, stored_output_id, group_key, window)` → `StoredSummary`. | +| SDS (Self-Describing Summary) | The description and metadata needed to interpret and validate stored summary state. It is not a separate execution engine or payload store. | +| `SummaryDefinition` | Immutable, versioned Planner-defined semantic description of the persisted output and only its necessary dependencies; not an executable plan. | +| `stored_output_id` | Compiler-assigned binding ID for a persisted PrecomputePlan DAG output within one plan version. Writers and shared readers use it to name the same output; it is not a memory slot or independent catalog object. | +| `StoredOutputReference` | Plan reference identifying a stored output and summary definition within the enclosing plan version. Reader configuration selects the required state instances and constrains format and coverage. | +| `StoredSummary` | One committed record containing instance metadata and payload, such as one service's completed five-minute KLL snapshot. | +| `SummaryStore` | Persistence authority for summary definitions and concrete stored results; Planner defines semantics and deployment installs them. The current implementation is `SketchStore`; no separate metadata or payload service is required. | +| `plan_version` | Version shared by an installed plan bundle and its catalog bindings. Creating or updating state instances does not itself change this version. | +| Schema / encoding | Schema describes the state structure; encoding describes how that structure is represented as bytes. | +| Semantic discovery | Future Planner search for legal query rewrites over persisted definitions; distinct from fingerprint equality and record lookup. | +| Deployment resolution | Backend selection of authorized stored outputs realizing a selected definition. | +| Logical dataset identity | Stable semantic source identity supplied before Planner definition export; distinguishes datasets independently of endpoints or replicas. | +| Definition ID | Fingerprint of a versioned canonical semantic description; different input expressions must remain distinguishable. | +| Provenance | Mapping from physical plan operations back to the selected Planner computation. | + +A materialization boundary is a Planner-selected physical output consumed +through typed inputs. Backend binding assigns its storage identity without +reclassifying logical nodes or changing that boundary. + +## Time, selection and validation + +| Term | Meaning | +| --- | --- | +| Logical range | Input interval required by the computation. In the example, `range: 5m` means `(T - 5m, T]` at evaluation time `T`. | +| Pane | Physical time partition of stored state. Several compatible panes may serve one logical range; pane size need not equal that range. | +| Refresh cadence | How often the producer is scheduled to build or refresh state. | +| Retention | How long state remains available; distinct from its input range and refresh cadence. | +| Readiness | Whether the required state is available with valid format and sufficient coverage/completeness for a read. Plan installation alone does not establish readiness. | +| Backend capability | Declaration of supported implementation combinations: algorithm/parameters, maintenance mode, input kind, window behavior and format. | +| Physical cost evidence | Scoped measurements or estimates used to compare executable candidates; includes workload and implementation context. | +| Compiler contract | Required inputs, outputs, validation rules and guarantees, including matching writer/reader definitions, formats, partitions and plan versions. | + +## Compilation ownership + +The [canonical Planner design at e9390031](https://github.com/ProjectASAP/ASAPPlanner/blob/e9390031fcecd7bc0d611127eddc5c6603a281e5/docs/design_docs/physical-planning-and-deployment.md) +defines physical lowering and frontier selection as Planner responsibilities. +The backend Deployment Plan Compiler binds declared physical inputs and outputs; +it does not reinterpret the logical DAG. Operator kind alone does not determine +maintenance versus query placement. diff --git a/docs/design_docs/summary-catalog-sds-architecture.md b/docs/design_docs/summary-catalog-sds-architecture.md index 2522bb026..afad2a1fc 100644 --- a/docs/design_docs/summary-catalog-sds-architecture.md +++ b/docs/design_docs/summary-catalog-sds-architecture.md @@ -1,578 +1,509 @@ -# Summary Catalog and Self-Describing Summary Architecture +# Self-Describing Summary: Semantic Definitions and Stored Results -This design defines three logical layers for summary producers and consumers. +Status: target design. Ad-hoc discovery is a future extension, not implemented +behavior claimed by this document. -| Layer | Describes | Changes when | -| --- | --- | --- | -| **Summary Descriptor** | Summary operator and fidelity guarantees | Algorithm, configuration or guarantee contract changes | -| **Data Descriptor** | Summarized source and population | Source binding or population definition changes | -| **Summary Instance** | Instance metadata and summary state | A concrete materialization is created or updated | +## 1. Why SDS? + +Stored summary bytes are not enough to determine what they mean. -Separating these layers lets many materialized instances reuse the same operator -configuration and data scope. A new time interval creates a new instance without -copying or redefining either descriptor. +For example: -## Proposed ownership +```text +KLL(latency) +``` -The descriptor vocabulary is a shared contract in `asap_types`. The control -plane owns the authoritative `SummaryCatalog`; Collector and backend receive the -same immutable catalog snapshot. Planner reasons about operators, fidelity, -source and population semantics, while runtime components bind catalog identities -to producers and stored instances. +and -| Layer | Responsibility | -| --- | --- | -| Summary Descriptor | Shared semantic definition used by Planner and backend | -| Data Descriptor | Shared source/population definition; backend resolves concrete runtime bindings | -| Summary Instance | Backend owns metadata, state, updates, storage and retirement | - -Planner may observe instance availability, covered time ranges and descriptor -references as planning evidence. It does not need the encoded summary state. -SDS describes summaries; an installed QueryPlan specifies how to execute a query -using them. The current backend fields are an incremental implementation of this -model. They must converge on the identities and invariants below rather than add -operator-specific stores beside `SketchStore`. - -## Target semantic model - -The target model has descriptor registries plus pane instances. Descriptor IDs -are derived from canonical semantic content; display names and runtime SIDs are -not descriptor identities. `SummaryDescriptorId` and `DataDescriptorId` currently -contain versioned canonical semantic strings. `SummaryDefinitionId` is a distinct -typed policy fingerprint, and `CatalogGeneration` identifies a publication using -its digest and plan version. A physical `SeriesId` identifies one storage lifetime -of a definition/group; it is neither a descriptor ID nor a pane instance ID. -Changing descriptor encoding to a hash must preserve content identity and handle -collisions explicitly. - -```rust -struct SummaryDescriptor { - id: SummaryDescriptorId, - operator: SummaryOperator, - fidelity: Vec