conformance: --pointer can address every element of an array - #11
Merged
Merged
Conversation
…ject nobody chose An envelope carries one document per comparison its producer performed, so a real CI report's `raw` is a LIST. `--pointer` resolved to ONE document, and the obvious repair — pointing at `/raw/0/diff` — is not a smaller check. It judges the first entry and prints the same confident CONFORMANT over the rest. Measured on a four-entry envelope whose only violation sits at entry 2: `/raw/0/diff` reads exit 0, CONFORMANT; `/raw/-/diff` reads exit 1 and names `report.json/raw/2/diff`. Both readings are pinned as fixtures in one manifest, so the contrast is executable rather than remembered. `-` in an ARRAY position now selects EVERY element, each judged and counted as its own document. The token is safe to give that meaning because RFC 6901 §4 already gives it exactly one in an array — the element after the last — and that element never exists, so no pointer that used to resolve can start resolving differently. In an OBJECT position it stays a literal member name: this tool does not overrule the standard it cites in order to be helpful. Two ways the corpus can shrink are refusals, not verdicts, and both are exit 2: a pointer written for a wire shape the envelope no longer has, and a pointer that resolves onto an EMPTY array. The second is the shrink no exception-shaped guard can see — every path taken is correct and the subject is nothing. The document count is now printed per file as well as in total, so a file that stopped carrying what it used to is visible without a roster. Nothing moves for a single-document subject: a pointer without the token selects one node and labels it by FILE, unchanged. Verified by re-running the three live invocations of this tool before and after — identical exit codes, identical findings, the only difference being the per-file count added to the corpus line. Three new controls, each derived from a mutation the previous fixture set passed: pointer-array-every-element, pointer-empty-selection, pointer-token-literal-in-object. The manifest gains two demands derived from a fixture's own shape rather than from a list: a fixture whose pointer expands must say WHICH element each finding came from, and a fixture expecting a refusal on a pointer must say WHICH refusal — exit 2 is a class, not a reason. Ten mutations red the suite; both demands verified by bypassing their partner (remove the demand and the key it asks for, and the suite goes green on a strictly weaker oracle). Unmutated control: 22/22, eight controls armed, against 16/16 and five before this change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect this closes, and why the obvious repair is worse than the bug
--pointerresolves to one document. An envelope carries one document percomparison its producer performed, so a real CI report's
rawis a list —raw[0].diff,raw[1].diff, …The obvious repoint,
/raw/0/diff, is not a smaller check. It judges the firstentry and prints the same confident
CONFORMANTover the rest — a green coveringa subject nobody chose. Measured on the four-entry fixture this PR adds, whose
only violation sits at entry 2:
/raw/0/diffCONFORMANT— over a file that contains a violation/raw/-/diffNONCONFORMANT, namingreport_with_raw_diff_array.json/raw/2/diffBoth rows are fixtures in the same manifest. The second is the fix; the first
is pinned deliberately, so the contrast is executable rather than remembered and
so the fix cannot later be read as an improvement in degree.
The change
-in an array position selects every element; each selected element isjudged and counted as its own document.
The token is safe to give that meaning because RFC 6901 §4 already gives it
exactly one meaning in an array — the element after the last — and that element
never exists, so no pointer that used to resolve can start resolving differently.
In an object position
-stays a literal member name, exactly as RFC 6901says: this tool does not overrule the standard it cites in order to be helpful.
A fixture carrying a member spelled
-pins that half.Two ways a corpus can shrink are refusals, not verdicts, and both exit 2:
-against anobject with no such member) — the message says so, rather than blaming the
document for a pointer fault;
exception-shaped guard can see: every path taken is correct and the subject is
nothing.
The document count is now printed per corpus file as well as in the total, so
a file that stopped carrying what it used to is visible without a roster. Each
finding names the concrete pointer of the element it came from —
1 of 40with noelement named is a number nobody can act on.
Nothing moves for a single-document subject
A pointer without the token selects exactly one node and labels it by file,
unchanged. Verified by re-running the three live invocations of this tool, before
and after: identical exit codes and identical findings, the only difference
being the per-file document count added to the
corpusline.In
--json,corpusbecomes a list of{path, documents, expanded}rather than alist of paths — one key with one meaning, rather than a second key repeating the
first.
Teeth
Ten mutations each red the self-test:
pointer-empty-selection/raw/0/diffcontrast pincontroltag deletedUnmutated control: 22/22, eight controls armed — against 16/16 and five before
this change.
The last two rows are the manifest's own demands, both derived from a fixture's
own shape rather than kept in a list beside it, and both verified by bypassing
the partner: remove the demand and the key it asks for, and the suite goes
green on a strictly weaker oracle. That is what each demand costs and why it is
derived rather than remembered.
Scope
conformance/only — noSPEC.md, noschema/. Not a format change: no documentchanges validity, and no producer becomes conformant or stops being so.