Skip to content

fix: make the verifiers able to refuse, and the citations able to be wrong - #19

Merged
arst merged 1 commit into
mainfrom
fix/fourth-review-followups
Sep 1, 2026
Merged

arst merged 1 commit into
mainfrom
fix/fourth-review-followups

Conversation

@arst

@arst arst commented Sep 1, 2026

Copy link
Copy Markdown
Owner

Fourth review pass. Six findings, all about what a sample is entitled to claim rather than whether it runs.

1. Least-to-Most accepted an answer its verifier had already failed (medium-high)

The real bug. A final answer that failed the deterministic billing check twice was still added to solved and printed as === Final answer === — directly under the log line proving it wrong.

Steps now carry Accepted / Unverified / Contested. A contested step ends the chain rather than becoming an established fact for the steps that would cite it, and the run prints the candidate beside the host's figure with status: CONTESTED instead of a total.

A verifier that cannot block acceptance is only telemetry.

Both paths were run live — the passing one, and the failing one by temporarily perturbing the host's schedule so the check could not agree.

2. GraphRAG traded fabricated provenance for inflated provenance (medium)

The previous fix dropped invented ids, then attached every document in the community to the summary — so documents the summary never rested on appeared to support it.

The citation is now claimed ∩ actual. Unclaimed community members print as "in this community but were not claimed as used — not cited as support", and a summary with no surviving id is not carried to the global answer at all.

Kept single-summary provenance rather than moving to claim-level SummaryClaim — that is the right next step, but it is a redesign of the synthesis stage, not a fix.

3. CoVe promoted a self-report into factual authority (medium)

check CONFIDENT and disagrees -> correct the draft let the same deployment's own confidence label decide which factual value wins. No evidence enters the system between the draft and the check, so preferring the check is only preferring the model's later guess.

Disagreement now resolves to contested either way, stating both values and what would settle them. The CONFIDENT:/UNCERTAIN: prefix is reported, not obeyed. Live run:

Founding year: AD 15 [CONTESTED: the blind check said around 15 BC (self-reported confident)]

4. Dual-LLM stale comment (minor)

A comment said the injected 48000.00 "would file", two lines above the gate that holds it. Now: it passes the type gate untouched, and the separate unattended-value policy holds it for a person.

5. Clarification screening rediscovered the slot it had already resolved (minor)

ScreenedQuestion carries TargetSlot, so the lexical match happens once in Screen and the merge stops repeating it. No behaviour change today; it removes the drift.

6. Refusal.NoSubscriber was dead taxonomy (minor)

"Nobody subscribes" was ambiguous between the workflow finishing and a topic name nothing will ever match — so DecisionMdae became a successful terminal event. Terminal topics are now declared (bus.RegisterTerminal("DecisionMade")); anything else unsubscribed is a NoSubscriber dead letter. The run publishes one misspelled event at the end to show a typo failing loudly.

One unasked extra there: the terminal event's payload — the approver's actual decision — was never printed, so the workflow's own output was invisible. One line.


Six pattern docs updated to match. 441 tests pass (+1), solution builds clean, all six changed samples re-run against the live deployment.

🤖 Generated with Claude Code

https://claude.ai/code/session_0161UFxvL3zPhufoYaQh27Ss

…wrong

Fourth review pass. Six findings, all about what a sample is entitled to
claim rather than whether it runs.

Least-to-most: a verifier that cannot block acceptance is only telemetry.
A final answer that failed its deterministic check twice was printed as
the run's answer anyway, directly under the log line proving it wrong.
Steps now carry Accepted/Unverified/Contested, a contested step ends the
chain rather than becoming an established fact, and the run prints the
candidate beside the host's figure instead of a total.

GraphRAG: the fabricated-citation fix had traded fabricated provenance
for inflated provenance - it dropped invented ids, then attached every
document in the community, so documents the summary never rested on
appeared to support it. The citation is now claimed ∩ actual, unclaimed
community members are reported as not-cited, and a summary with no
surviving id is not carried to the global answer at all.

Chain of verification: "check CONFIDENT and disagrees -> correct the
draft" let a self-report decide which factual value wins. No evidence
enters the system between the draft and the check, so a disagreement is
a flag, not a correction. Disagreement now resolves to contested with
what would settle it; the confidence label is reported, not obeyed.

Event bus: terminal topics are declared, not inferred. "Nobody
subscribes" was ambiguous between the workflow finishing and a topic
nothing will ever match, which made DecisionMdae a successful outcome.
RegisterTerminal declares outcomes; anything else unsubscribed is a
NoSubscriber dead letter, and the run publishes one misspelled event to
show it. Revives an enum member that had gone dead.

Proactive clarification: ScreenedQuestion carries the slot the screen
resolved it to, so the merge stops re-deriving it from question text.

Dual-LLM: a comment still said the injected figure "would file" two
lines above the gate that holds it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0161UFxvL3zPhufoYaQh27Ss
@arst
arst merged commit f2f2e68 into main Sep 1, 2026
2 checks passed
@arst
arst deleted the fix/fourth-review-followups branch September 1, 2026 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant