fix: make the verifiers able to refuse, and the citations able to be wrong - #19
Merged
Merged
Conversation
…wrong Fourth review pass. Six findings, all about what a sample is entitled to claim rather than whether it runs. Least-to-most: a verifier that cannot block acceptance is only telemetry. A final answer that failed its deterministic check twice was printed as the run's answer anyway, directly under the log line proving it wrong. Steps now carry Accepted/Unverified/Contested, a contested step ends the chain rather than becoming an established fact, and the run prints the candidate beside the host's figure instead of a total. GraphRAG: the fabricated-citation fix had traded fabricated provenance for inflated provenance - it dropped invented ids, then attached every document in the community, so documents the summary never rested on appeared to support it. The citation is now claimed ∩ actual, unclaimed community members are reported as not-cited, and a summary with no surviving id is not carried to the global answer at all. Chain of verification: "check CONFIDENT and disagrees -> correct the draft" let a self-report decide which factual value wins. No evidence enters the system between the draft and the check, so a disagreement is a flag, not a correction. Disagreement now resolves to contested with what would settle it; the confidence label is reported, not obeyed. Event bus: terminal topics are declared, not inferred. "Nobody subscribes" was ambiguous between the workflow finishing and a topic nothing will ever match, which made DecisionMdae a successful outcome. RegisterTerminal declares outcomes; anything else unsubscribed is a NoSubscriber dead letter, and the run publishes one misspelled event to show it. Revives an enum member that had gone dead. Proactive clarification: ScreenedQuestion carries the slot the screen resolved it to, so the merge stops re-deriving it from question text. Dual-LLM: a comment still said the injected figure "would file" two lines above the gate that holds it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0161UFxvL3zPhufoYaQh27Ss
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth review pass. Six findings, all about what a sample is entitled to claim rather than whether it runs.
1. Least-to-Most accepted an answer its verifier had already failed (medium-high)
The real bug. A final answer that failed the deterministic billing check twice was still added to
solvedand printed as=== Final answer ===— directly under the log line proving it wrong.Steps now carry
Accepted/Unverified/Contested. A contested step ends the chain rather than becoming an established fact for the steps that would cite it, and the run prints the candidate beside the host's figure withstatus: CONTESTEDinstead of a total.Both paths were run live — the passing one, and the failing one by temporarily perturbing the host's schedule so the check could not agree.
2. GraphRAG traded fabricated provenance for inflated provenance (medium)
The previous fix dropped invented ids, then attached every document in the community to the summary — so documents the summary never rested on appeared to support it.
The citation is now
claimed ∩ actual. Unclaimed community members print as "in this community but were not claimed as used — not cited as support", and a summary with no surviving id is not carried to the global answer at all.Kept single-summary provenance rather than moving to claim-level
SummaryClaim— that is the right next step, but it is a redesign of the synthesis stage, not a fix.3. CoVe promoted a self-report into factual authority (medium)
check CONFIDENT and disagrees -> correct the draftlet the same deployment's own confidence label decide which factual value wins. No evidence enters the system between the draft and the check, so preferring the check is only preferring the model's later guess.Disagreement now resolves to contested either way, stating both values and what would settle them. The
CONFIDENT:/UNCERTAIN:prefix is reported, not obeyed. Live run:4. Dual-LLM stale comment (minor)
A comment said the injected
48000.00"would file", two lines above the gate that holds it. Now: it passes the type gate untouched, and the separate unattended-value policy holds it for a person.5. Clarification screening rediscovered the slot it had already resolved (minor)
ScreenedQuestioncarriesTargetSlot, so the lexical match happens once inScreenand the merge stops repeating it. No behaviour change today; it removes the drift.6.
Refusal.NoSubscriberwas dead taxonomy (minor)"Nobody subscribes" was ambiguous between the workflow finishing and a topic name nothing will ever match — so
DecisionMdaebecame a successful terminal event. Terminal topics are now declared (bus.RegisterTerminal("DecisionMade")); anything else unsubscribed is aNoSubscriberdead letter. The run publishes one misspelled event at the end to show a typo failing loudly.One unasked extra there: the terminal event's payload — the approver's actual decision — was never printed, so the workflow's own output was invisible. One line.
Six pattern docs updated to match. 441 tests pass (+1), solution builds clean, all six changed samples re-run against the live deployment.
🤖 Generated with Claude Code
https://claude.ai/code/session_0161UFxvL3zPhufoYaQh27Ss