Skip to content

Retire PolicyFingerprint as StreamingConfig's routing key #785

Description

@zzylol

PolicyFingerprint describes itself as legacy, in its own module doc:

//! Legacy routing wrapper for a deployed stored output.
//!
//! An explicit `PrecomputeMaterialization::stored_output_id` takes precedence.
//! Otherwise the compiler allocates a deterministic default from the existing
//! policy fields (including pane layout and cadence). This identifier is not
//! semantic identity: `SummaryDefinitionId` hashes the versioned semantic
//! definition, and several deployed outputs may share that definition.

Its two original jobs have already been split out: semantic identity belongs to SummaryDefinitionId, and explicit routing belongs to stored_output_id.

The migration is real but cannot finish on its current path

stored_output_id adoption is climbing through the stack:

branch stored_output_id occurrences
main 157
issue-752 (#761) 194
test/physical-execution-e2e (#775) 196
test/issue754-level3 (#759) 196

But PolicyFingerprint occurrences do not fall. Counted across every open PR branch, they sit at 300-302 against main's 301, and several branches are at 302. No open PR deletes crates/asap_types/src/policy_fingerprint.rs.

The reason is a loop in PolicyFingerprint::from_config:

pub fn from_config(cfg: &PrecomputeMaterialization) -> Self {
    if let Some(output) = cfg.stored_output_id {
        return output.fingerprint();   // the explicit id becomes a fingerprint too
    }
    ...
}

Adopting an explicit id does not remove a fingerprint; it changes how that fingerprint is derived. The computation is being retired. The type is not, because it is still the routing key.

What actually blocks retirement

StreamingConfig keys its materializations by the fingerprint as a bare u64:

HashMap<u64, PrecomputeMaterialization>

StreamingConfig is what the control plane POSTs to the data plane as YAML (control_plane/src/backend_client.rs). So the key is part of a published configuration format, and changing it is a format change rather than an internal refactor.

Proposed work

  1. Key StreamingConfig's materializations by StoredOutputId instead of a bare u64.
  2. Require stored_output_id, so from_config's hashing fallback has no callers, then delete the fallback and the module.
  3. Bump the published schema version, which now has a single owner and a test enforcing that every producer agrees (WORKLOAD_SNAPSHOT_VERSION, added in fix(control-plane): give the snapshot schema version one owner #782).

Step 3 is the reason this is worth doing now rather than later: #782 made a schema bump one edit plus whatever its test reports, instead of a literal that some producers follow and others quietly do not.

Not in scope

SummaryDefinitionId stays. It is semantic identity, a different thing from routing, and several deployed outputs may legitimately share one definition.

Related

Separately worth checking while in this area: plan_id is computed with std::collections::hash_map::DefaultHasher (control_plane/src/physical/compiler.rs), whose algorithm the standard library does not guarantee across Rust releases, yet plan_id is compared across processes and contract tests reject a changed installed generation. A toolchain upgrade would change it.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions