Skip to content

[Design] Let the backend decide among candidates with missing evidence #454

Description

@zzylol

Motivation

ASAPPlanner currently treats missing evidence differently across candidate families: some candidates are generated with unknown guarantees/costs, while others are suppressed during construction or removed by target-aware search. This can prevent ASAPQuery-backend from seeing an alternative and applying evidence or policy that it owns. PR #449 addresses only direct DDSketch quantile ratios; this issue defines the broader intended Planner/backend interface.

Intended interface

For every candidate family whose semantic/execution shape can be constructed without external evidence, Planner should retain the candidate in PlanSpace even when evidence needed to establish accuracy, cost, or deployment feasibility is missing. Planner must expose what is known, unknown, or disproven in a machine-readable way. A missing proof must never be represented as a satisfied accuracy target or a zero/finite cost. ASAPQuery-backend inspects its own evidence and capabilities and makes the final accept/reject and deployment decision.

This does not require Planner to emit malformed DAGs, unsupported operators, or candidates with a known violated invariant. The design should distinguish “unknown because evidence is absent” from “known invalid” and define which checks belong to Planner versus backend.

Work

  • Inventory evidence-gated candidate paths (accuracy/domain, cost, workload statistics, and runtime capability) and document which currently suppress candidates.
  • Define one consistent candidate-state contract for missing, supplied-valid, and supplied-invalid evidence, including how root AccuracyTarget interacts with an unknown guarantee. Prefer existing types where sufficient; add a new abstraction only if a concrete consumer needs it.
  • Make search_workload_with_targets, ranking, whole-plan selection, materialization, and export preserve the intended candidate state without silently certifying or deploying it.
  • Document the Planner output and the backend responsibility for final evidence/capability checks, with an end-to-end integration example.
  • Add cross-family regression tests, including no evidence, partial evidence, invalid evidence, and explicit accuracy targets.

Acceptance criteria

  • A backend can inspect all constructible evidence-dependent alternatives before making its own selection, rather than losing them during Planner candidate generation or root-target filtering.
  • Unknown accuracy/cost/feasibility is explicit; no default ranking or selection path mislabels an uncertified candidate as satisfying a requirement.
  • Known-invalid evidence and structurally/semantically impossible candidates remain rejected with a reason.
  • The behavior and ownership boundary are documented for library users and validated with at least one end-to-end backend-facing workflow.

Related: #446, PR #449, #428, #427, ASAPQuery-backend#734.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions