Skip to content

[Change] Calibrate Code Review into a truthful blocking gate #417

Description

@djm81

Why

specfact code review can currently emit error findings and still return zero because severity, score, fixability, enforcement mode, and core fallback behavior are independent. C15 makes the signed policy/profile the sole severity-calibration authority and derives truthful assurance and exit behavior from effective severity, lifecycle, waiver, differential state, and enforcement mode without changing C14 scope or differential contracts.

Parent and coordination

Blockers and dependency order

Implement these prerequisites first:

  1. [Change] Implement truthful Code Review scope and differential enforcement #416 — C14 truthful scope and differential enforcement, including its signed schema 1.6 handoff.
  2. Policy Packs & Enforcement Modes (Advisory/Mixed/Hard) #158 — authoritative policy packs and enforcement modes.
  3. [Change] Runtime Exception Management And Enforcement Follow-Up #167 — modules-side exception/suppression runtime.
  4. Profile and Validation Config Layering specfact-cli#237 — authoritative profile/config layering.
  5. Exception Management — Time-Bound, Tracked Policy Exceptions specfact-cli#248 — time-bound, trusted exception authority.

Native blocked-by relationships include all five prerequisites, including the two cross-repository core authorities.

What changes

  • Upgrade ReviewReport to schema 1.7 and preserve raw analyzer severity separately from effective severity.
  • Block non-shadow changed or full enforcement on every open, applicable, unwaived effective error.
  • Make required analyzer/policy uncertainty UNKNOWN and fail non-shadow enforcement closed.
  • Keep warning/info advisory, keep info score-neutral, and prohibit score or autofix availability from determining verdict or exit code.
  • Preserve truthful assurance_status in shadow while returning zero.
  • Reuse C14 identity, source-span, multiset, scope, immutable snapshot, and differential contracts.
  • Implement D1–D4: scoped MISSING_ICONTRACT, hierarchy-aware optional-param analysis, AST-backed attribution/exact deduplication, and deterministic calibrated contexts.
  • Add audited suppression evidence authenticated against trusted, active, scope-matched exception records.
  • Freeze and sign the initial rule/severity mapping and schema compatibility fixtures.
  • Dogfood in shadow, remediate both repositories, rerun the frozen measurement, then explicitly activate blocking mode only if the precision target passes.

Acceptance criteria

  • Open, applicable, unwaived effective errors yield FAIL and exit 1 in non-shadow changed/full enforcement.
  • Required uncertainty yields UNKNOWN and exit 1 in non-shadow enforcement.
  • Warning/info-only and waived/advisory reports do not block; info is score-neutral.
  • Score and fixable never determine verdict or exit.
  • Shadow preserves truthful assurance while exiting zero.
  • Suppression directives never delete findings and cannot self-authorize candidate-added exceptions.
  • Attribution ambiguity that could affect blocking becomes UNKNOWN; multiplicity is never silently collapsed.
  • Schema 1.7 compatibility, signed module identity, dogfood remediation, and the frozen audit gates all pass before blocking activation.

Measurement and rollback

  • Repeat the frozen 40-PR replay and n=150 stratified adjudication with seed 20260819.
  • Require weighted effective-error actionable precision ≥80%, Wilson 95% lower bound ≥70%, and at least 40 adjudicated effective-error findings.
  • Require zero silently collapsed multiplicity cases and zero unresolved blocking attribution conflicts.
  • If targets miss, keep shadow mode, demote failing rules, and rerun the frozen measurement.
  • Rollback is configuration-only: restore shadow while retaining schema 1.7 evidence.

Non-goals

  • No new analyzer family or AI reviewer.
  • No blanket blocking of warning/info findings.
  • No candidate-controlled self-waiver.
  • No change to C14 Git scope, immutable snapshots, differential matching, or protected-envelope ownership.
  • No generalization claim beyond the measured Python repositories.

OpenSpec Change Proposal: code-review-15-signal-calibrated-blocking-gate

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

change-proposalProposal for a new changecodebaseSpecfact codebase related topicenhancementNew feature or requestopenspecopenspec change

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions