Repository navigation
Add conformance fixtures for the #504 and #507 evidence gaps - #543
johnml1135 wants to merge 11 commits into
Conversation
Introduces an engine-agnostic conformance suite for HermitCrab-formalism morphological parsers: a `conformance/` tree of language grammars and edge-case micro-grammars, each an HC-XML grammar.xml paired with a words.yaml ground truth (oracle-provenance, gloss, and per-word parse signatures). A new SIL.Machine.Morphology.HermitCrab.Conformance project and hc-conformance CLI tool run those fixtures against an engine adapter, producing coverage and rules CSVs, an interface inventory, dataflow- obligation ledgers, evidence cards, a FieldWorks-producibility table, and a JSON manifest for external consumers. PROTOCOL.md is the adapter contract a second engine (e.g. the in-progress Rust port) implements to plug into the same suite. Includes the FieldWorks-producibility classification campaign: for every HermitCrab construct the suite covers, a verdict on whether FieldWorks' own HC-XML export path can actually produce a grammar exercising it, distinguishing genuine data-model limits from mere loader gaps, with five previously-unresolved verdicts now classified and corrected write-ups for two more cases initially misread as exporter silence. Also lands one new edge-case fixture, edge-cases/chained-output-feature-override-loss, isolating an analysis- side OutputHeadFeatures retraction bug: a well-formed four-morph parse of `zudiua` that the founding C# oracle currently loses. The fixture is deliberately red against hc.dll today (expected, not a broken test) and documents the mechanism, provenance, and the HC-Rust divergence that motivated it. The suite is deliberately stopped at this point rather than finished: scope, known gaps, and next steps are documented in the suite's own tracking docs rather than left implicit. This commit squashes the full conformance-suite branch stack (integrate-conformance-framework through conformance/fieldworks- witnesses, 49 commits) into one commit on top of current master, for PR #480. Folds in three review fixes: AdapterEngine's contract in MaterializedRunner now credits an expect_crash fixture only once the same engine also completes an ordinary fixture from the same run, so a misconfigured adapter (bad path, rejected arguments) can no longer register as reproducing the pinned crash; GuesserSignatureTests' batch-path test is renamed for the overload it actually calls; and .gitattributes freezes project.fwdata as binary so its pinned SHA-256 is reproducible across platforms and CI runners, not just this one. Also folds in the follow-up PR commits: fuzz/break-attempt evidence; admitting PCRE.NET's source generator via a vetted content-hash allowlist; final-template and template-sharing controls; oracle-verified fixtures for per-piece environments, simultaneous mode and iterative epenthesis; reduplication x phonology cases; and a regenerated manifest, ledgers and evidence cards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ce suite The suite at the integrate-conformance-framework head was blind to #539: base, the PR head and a fixed variant gave identical results on every fixture and test, while on real grammars the PR head hangs on a simultaneous expansion rule and loses parses behind narrowing (merge) rules. Add fourteen synthetic edge-case fixtures that close that gap, each verified against the C# founding oracle at this commit and shown to fail (or time out) on the #539 head: - simultaneous-expansion-terminates (#539 head hangs), feature-rule-before-merge, epenthesis-fed-merge, geminate-under-merge and default-symbol-after-merge (engine-only: SymbolicFeature.defaultSymbol), ported from the PanGloss conformance-staging set. - boundary-first-in-narrowing-lhs, boundary-first-in-narrowing-lhs-strrep, boundary-leftover-in-narrowing-lhs-strrep, boundary-mid-in-narrowing-lhs-strrep, deletion-reinserted-feature-unapply, expansion-overlapping-targets, merge-subrules-pos-rtl-template, merge-then-metathesis and two-merges-cascade-env, from the #539 regression hunt. The two zero-feature boundary fixtures also fail on the no-dirty-marks variant of the PR. Regenerate every ledger the gates check (manifest, coverage.csv, rules.csv, interface inventory, interaction chains, rule-interaction pairs, counterfactuals, evidence, the traceability bundle and evidence cards) and move the measured-count pins in the ledger tests to the new corpus. Add the fourteen rows to fixtures.csv and correct the README fixture count. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The suite added Trace.FailureAllomorph so FailureRuleAttributor could credit an affix rule whose allomorph failed an environment or co-occurrence check. That is an engine API change riding in a test-suite PR, and #534 adds the same fact as Trace.Allomorph. Drop it here; the attribution returns with #534. No generated ledger changes: coverage.csv and rules.csv regenerate byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
--coverage-report printed dead rules and uncovered constructs but always exited 0, so the CI step that ran it could not fail. It now exits 1 when an in-scope construct has no coverage, when a rule no word exercises is missing from conformance/dead-rules-allowed.tsv, or when an allowed rule is exercised again. The allowlist holds today's 15 dead rules, each with its reason: inactive decoys, refused controls, and rules in crash-pin fixtures. The Conformance workflow also runs when SIL.Machine core or the HermitCrab tests change, since both can change parse results. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The suite added two steps to ci.yml: a fixture run, which ConformanceFixtureGateTests already performs under dotnet test, and parity-check.py, whose one remaining check the coverage report now enforces in the Conformance workflow. It also pinned SDK 10.0.303, which global.json's latestFeature roll-forward overrides. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
chained-output-feature-override-loss was written red: its header says the oracle loses zudiua today and that the loss survives PriorityUnion. Both are stale. Running the fixture against master shows zudiua red at 52d069f and green from 7109531 (#494) onward; the earlier note measured the pre-merge research branch. The header now describes the fixture as the regression pin it has become. Manifest regenerated for the new file hash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
local_check.sh ran `--semantic-coverage`, whose CLI exits 1 while the semantic catalog is an incomplete backlog -- the state the suite expects, and gates under dotnet test instead -- so the script always failed after the tests passed. It now runs what conformance.yml runs: self-check with pathological fixtures, the coverage report, and the CSV freshness check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A counterfactual mutant is parsed in a child process and its outcomes come back through stdout, while the baseline it is compared with is computed in-process. Neither end pinned an encoding, so on a non-UTF-8 console code page a surface form such as "katibɯd" came back as "katib?d", and "∅" as "φ". Every mutant of a fixture with non-ASCII output then looked like it changed a parse, and its surface counted as Evidenced. The checked-in ledger has two such verdicts, Gloss and HeadFeatures; Gloss is the surface the suite's own tests describe as never read by the parser. The child now writes UTF-8 and the parent reads UTF-8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With the encoding fixed, the counterfactual sweep agrees with the checked-in ledger on every verdict but two: Gloss is Unobservable (no word's parse reads it) and HeadFeatures is RequiredByDtd. Both had been recorded as Evidenced from mis-encoded output. Every other changed row keeps its verdict and only loses the garbled delta text. The ratchet's pinned gap count goes from 25 to 26 for Gloss, a gap that was always there and is now counted. The ledger-shape pins in FoldInCandidateLedgerTests and GrammarCoverageLedgerTests move with the regenerated files. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
syntactic-feature-inversion-controls (#504): 23 oracle-recorded words with positive and negative controls for analysis feature inversion over nested paths, a disjunctive allomorph pair, and a genuine two-rule merge. RequiredHeadFeatures is unification, not a presence check, so an absent path unifies with anything and only a conflicting value is refused. final-template-gap-coverage (#507): the three cases final-template-partial-discriminators names as uncovered -- a rule used both as a stratum rule and a template slot rule, a partial lexical entry, and a derivation continuing across strata. Both pass on master. Of the four ordering items they add, one has no word that evidences it, which moves the coverage-gap ratchet from 26 to 27. Ledgers and count pins regenerated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| <MorphemeId>NOKFULL</MorphemeId> | ||
| <Gloss>nokfull</Gloss> | ||
| </LexicalEntry> | ||
| <LexicalEntry id="eTikPartial" partOfSpeech="posB" partial="true"> |
There was a problem hiding this comment.
Minor: F1. Branch B's partial root turns off final-template pruning for Branches A and C as well. With any partial morpheme, Morpher.IsPartial is true for the whole grammar (Morpher.cs:68). That skips the FinalTemplateState writes at AnalysisAffixProcessRule.cs:83 and the prune at AnalysisAffixTemplateRule.cs:47. So dukom and zaboteshi are never analysed with pruning on, and the doc's "none of the three branches interacts" is false. Move Branch B into its own fixture.
|
|
||
| Unmodified grammar remains green. These are grammar-sensitivity tests, not proof that PR #491's analysis | ||
| optimization executes; the analysis-side `FinalTemplateState` pruning this fixture is meant to feed | ||
| evidence to does not exist on this fixture's own base branch at all (confirmed: zero references in |
There was a problem hiding this comment.
Minor: F2. The pruning this paragraph calls absent is already on the base branch. git grep FinalTemplateState 3a63132b -- src finds it at AnalysisStratumRule.cs:164, AnalysisAffixTemplateRule.cs:48 and AnalysisAffixProcessRule.cs:85. It is the code F1 disables here, so the "cannot be exercised from this repository state" conclusion is the wrong way round.
| } | ||
|
|
||
| private const int PinnedGapCount = 26; | ||
| private const int PinnedGapCount = 27; |
There was a problem hiding this comment.
Minor: F3. Raising the pin to 27 lets this PR's own new gap through. The class doc says gaps "may only go down". The gap is ordering:.../syntactic-feature-inversion-controls/.../mrNum~mrNumSg: semantic-coverage-evidence.tsv has rows for mrNumSg~mrPers and mrPers~mrGate, but none for this pair. Add a word that witnesses it, or drop mrNumSg from the linear rule list, and keep the pin at 26.
| root alone never closes -- not a rule rejection, a structurally incomplete derivation. | ||
| expect_fail: true | ||
|
|
||
| - word: dukomom |
There was a problem hiding this comment.
Minor: F4. Nothing in Branch A depends on mrDualA being in two places. The doc's own mutation (final-template-gap-coverage.md:43) shows that removing the stratum listing changes no row, and dukomom fails either way because of multipleApplication=1. The #507 "rule invoked both ways" gap is therefore still uncovered. Add a severance that changes a result, such as multipleApplication="2".
| THE main positive pin: mrNum, mrPers, then mrGate. mrGate's RequiredHeadFeatures reads THREE | ||
| nested cfAgr sub-paths at once -- agr.num=pl (mrNum's write), agr.pers=1 (mrPers's write), | ||
| agr.cls=clsA (never written by any rule; survives untouched from eSim's own | ||
| AssignedHeadFeatures). All three must be simultaneously recoverable for this to parse: nested |
There was a problem hiding this comment.
Minor: F5. simiao cannot fail if a write is dropped, despite this note's "all three must be simultaneously recoverable". By grammar.xml:25-29 and the passing simio/simao/basiao rows, an unset path unifies with mrGate. If mrNum's num, mrPers's pers, or the root's cls were lost, simiao and douao would still parse, so they do not show a MERGE or PASSTHROUGH happened. Unverified by mutation; the reasoning comes from the fixture's own rows.
| nested cfAgr structure (neither rule alone provides both); cls surviving | ||
| from the root's own AssignedHeadFeatures to mrGate, untouched by any rule | ||
| in between, is the PASSTHROUGH check. | ||
| No two rules ever write the SAME nested path, so this fixture does not re-probe the override- |
There was a problem hiding this comment.
FYI: F6. "No two rules ever write the SAME nested path" is false. mrNum (line 158) and mrNumSg (line 173) both write cfAgr/fNum, and they are adjacent in the linear order. That is the override-loss chain this sentence rules out, and the unwitnessed pair behind F3.
| the grammar through the C# founding oracle (`hc-dotnet-wrapper.sh`) produced byte-identical signatures and | ||
| counts each time. | ||
|
|
||
| Each mutation below keeps the fixture's original expected rows and makes it fail: |
There was a problem hiding this comment.
FYI: F7. "Each mutation below ... makes it fail" is contradicted by the first bullet, which reports IDENTICAL output. Branch A has no failing severance (see F4), and this sentence hides that.
| note: >- | ||
| Branch C negative control: the Root-stratum prefix alone, without Outer's final template, is a | ||
| non-final-template-parallel case for a rule that lives in a DIFFERENT stratum than its template -- | ||
| the derivation reaches the Outer stratum but the mandatory Slot there is skipped only if it |
There was a problem hiding this comment.
FYI: F8. This note gets the engine mechanism wrong in two places. SynthesisStratumRule.cs:82 resets IsLastAppliedRuleFinal = null when a word leaves a stratum, so nothing is forwarded "already final" (line 93). An applicable mandatory slot is never "skipped": SynthesisAffixTemplatesRule.cs:57-61 produces no output for it, and that is why zabot and bot fail.
| - signature: "SIM|sim" | ||
| rules: [] | ||
| exercises: | ||
| - "Syntactic feature agreement (RequiredHeadFeatures/OutputHeadFeatures/RequiredFootFeatures/OutputFootFeatures)" |
There was a problem hiding this comment.
FYI: F9. Bare-root rows with rules: [] (sim, and likewise do, dam and bas) claim to exercise "Syntactic feature agreement" when no rule applied. They still count as Confirmed in construct-claim-corroboration.tsv, which inflates the confirmed=243 pin.
| @@ -0,0 +1,331 @@ | |||
| # oracle-provenance: oracle-recorded | |||
There was a problem hiding this comment.
FYI: F10. oracle-recorded with no machine-commit= makes the oracle run hard to reproduce. Every other fixture records founding-oracle machine-commit=<sha> or forward-synthesis. Lines 5-6 name a local worktree (conformance/oracle-504) rather than a commit. Record the sha the signatures came from.
| # against the real oracle rather than sketched algebraically. See grammar.xml's own header for the | ||
| # full mechanism write-up (mrNum/mrNumSg/mrPers/mrGate, and the cfAgr.{num,pers,cls} nesting). | ||
| # | ||
| # A CORRECTION made DURING authoring, kept here rather than silently smoothed over (per this |
There was a problem hiding this comment.
FYI: F11. The new comments break .claude/skills/code-comments/SKILL.md. Lines 24-37 narrate authoring history ("the first draft ... predicted"), and grammar.xml lines 53, 117, 137, 194 and 248 use em dashes. comment-hygiene.ps1 does not scan .xml/.yaml, so --agent-strict will not catch this.
|
Verdict: request changes.
Findings: F1 new, F2 new, F3 new, F4 new, F5 new unverified, F6 new, F7 new, F8 new, F9 new, F10 new, F11 new. Reviewed at 499e154 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## integrate-conformance-framework #543 +/- ##
================================================================
Coverage 74.10% 74.10%
================================================================
Files 555 555
Lines 56041 56041
Branches 7622 7622
================================================================
Hits 41527 41527
Misses 12666 12666
Partials 1848 1848 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
3a63132 to
df2cf21
Compare
3 participants
Quick summary
Adds two conformance fixtures for the evidence gaps named in #504 and #507.
No parser code changes; both fixtures pass on master today and pin current behaviour.
Stacked on #480 (the conformance suite); review that first.
Where to look
conformance/edge-cases/syntactic-feature-inversion-controls(HermitCrab: track exact analysis feature inversion and chained override parse loss #504) -- positive and negative controls for analysis feature inversion: nested paths, a disjunctive allomorph pair, and a genuine two-rule merge.conformance/edge-cases/final-template-gap-coverage(HermitCrab: complete cross-engine correctness controls for final-template pruning #507) -- the three casesfinal-template-partial-discriminatorslists as uncovered: a rule used both as a stratum rule and as a template slot rule, a partial lexical entry, and a derivation continuing across strata. Reasoning inconformance/docs/final-template-gap-coverage.md.hc-conformance --write-*. One of the four ordering items the fixtures add has no word that evidences it, so the coverage-gap ratchet moves from 26 to 27.Deliberately not included
commuting-affix-order-collision(HermitCrab: establish discriminating conformance coverage for analysis-state and template/slot merges #505): its companion test measures analysis-memo hits, and Remove HermitCrab's analysis-cascade memo #522 removes the memo.zero-width-morpheme-identity-stability(HermitCrab: investigate unstable zero-width morpheme identities across fresh processes #506): it fails on master on purpose, and a CI gate cannot carry a red. It stays with the issue until a fix exists.Validation
hc-conformance --fixtures conformance --include-pathological-- 60 passed, 0 failed.hc-conformance --fixtures conformance --coverage-report-- exit 0; 15 dead rules, all allowed;coverage.csv/rules.csvunchanged.hc-conformance --check-manifest-- current (60 fixtures).dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests-- all pass after the re-pins (the full run had one failure, the ratchet, before it moved to 27).scripts/comment-hygiene.ps1clean;dotnet csharpier check .clean.🤖 Generated with Claude Code
This change is