diff --git a/docs/hermitcrab-2026-10-21-search-cost-round.md b/docs/hermitcrab-2026-10-21-search-cost-round.md new file mode 100644 index 000000000..3522a8a74 --- /dev/null +++ b/docs/hermitcrab-2026-10-21-search-cost-round.md @@ -0,0 +1,377 @@ +# HermitCrab search cost, 2026-10-21 round + +**Disposition: three large reductions in search work stand up. Two of them were bugs in the Rust +port, not algorithmic gains.** HC-Rust ([sillsdev/PanGloss](https://github.com/sillsdev/PanGloss)) +is a port of `SIL.Machine.Morphology.HermitCrab` under test against `hc.dll`. On one private +grammar, HC-Rust did far more search than C# because two kinds of shape node had lost the segment +identity that C# gives them. Fixing both made HC-Rust match C# and removed most of that grammar's +cost. The one algorithmic change that survived, sharing trailing affix-template slot batches, was +built in HC-Rust and then ported to C# as [Machine PR #546](https://github.com/sillsdev/machine/pull/546). +The lead ranked first in the [search-asymptotics research](hermitcrab-search-asymptotics-research.md), +lexically-dead branch pruning, was measured first and rejected on its ceiling. + +This record follows the [optimization ledger](hermitcrab-optimization-ledger.md): one entry per +attempt, the number that settled it, and what differs from earlier attempts. Ledger rows 24-30 +summarise it. + +## Data and units + +**Grammars.** The public reference grammars are Sena, Amharic, Mbugwe, Aweti and Indonesian. The +round's motivating grammar was a **private featureless project (Austronesian, ~30k lexical entries, +3 strata)**. It was measured in three variants: *base*, *v1*, and a *deletion variant* that adds a +rule deleting two stop classes after nasals and has no compounding. Each variant was run on three +word lists: 200 nonwords, a 563-word dev list and a 7,206-word held list. Only aggregate numbers +from it appear here; no word forms, glosses or lexical entries. + +**Units.** The owner rule for this round is that gains are stated as **parser work**: "N% less work +to parse the words". +- **Parser work** is HC-Rust's per-word `work_steps`, summed over a word list. A work unit is one + reservation against HC-Rust's per-word work cap. Every analysis attempt reserves one, and so does + every inner matcher step, segmentation step, root-lookup step and confirmation step. It is + deterministic, so it survives machine contention, unlike wall time. +- **Steps** are analysis attempts (`steps <= work_steps`). They are a secondary figure. +- **Milliseconds** appear only as labelled secondary figures. +- **C# figures** are in C#'s own counters (slot batches, lookups), never in HC-Rust work units. They + are not converted. + +**Parity comparators.** HC-Rust used `rust/tools/parse_compare.py`, which compares complete +analyses per word and reports each row IDENTICAL or not. Entries 090 and 103 ran it with +`--multiset`; entry 095 records only `parse_compare.py`. PR #546 compared ordered morpheme-ID/surface signature lists, +ordinal-sorted, with multiplicity. Neither is the legacy morph-count `WordAnalysisSignature` that the +September disposition calls out. + +**Runs.** Unless noted, HC-Rust runs were `pangloss batch --threads 1` with `HC_STEP_STATS=1`. +Change A's full-list parity runs used `--threads 2` with the default step cap and no wall timeout. + +## What stands up + +| # | Change | Kind | Where | Effect, in parser work | Parses | +|---|---|---|---|---|---| +| A | Analysis-side affix literals keep `StrRep` identity on wide featureless tables | HC-Rust port bug. C# already exact | PanGloss divergence entry 090 (analysis-side `Lhs` follow-up) | Private project, 7 spot words: base **98.5% less** (2,365,602 -> 35,913); worst variant **99.8% less** (50,778,583 -> 83,677). Steps on the full lists fell 46-55x | IDENTICAL on all three private lists and on Amharic, Sena, Indonesian | +| B | A re-inserted segment-class node keeps its members on a featureless grammar | HC-Rust port bug. C# already exact | PanGloss entry 103, [`414ac6b5`](https://github.com/sillsdev/PanGloss/commit/414ac6b5), [record](https://github.com/sillsdev/PanGloss/blob/main/docs/divergences/103-class-deletion-reinsertion-identity.md) | Deletion variant, held list: **94.7% less** (303,966,311 -> 16,220,190). Its 16 words that hit a 1 s limit: **99.9% less** | IDENTICAL on every list. Byte-identical work wherever it cannot fire | +| C | Shared trailing slot trie in the affix-template battery | Algorithmic, no memo | PanGloss entry 095, [record](https://github.com/sillsdev/PanGloss/blob/main/docs/divergences/095-shared-template-slot-batches.md); C# [PR #546](https://github.com/sillsdev/machine/pull/546) | Sena **11.5% less** (all 6,146 words); Mbugwe **18.1% less** (1,364-word partial run); Aweti 0.35% less; zero elsewhere | IDENTICAL in HC-Rust; zero signature differences in C# samples | + +The census instrumentation that settled the rejected lead is +[`d1690c13`](https://github.com/sillsdev/PanGloss/commit/d1690c13). It is gated behind +`HC_BRANCH_CENSUS=1` and inert when unset: parser work and parses are identical to main on six +lists. + +## The lesson: diff the port on the oracle's own export before optimizing the algorithm + +The bugs were found by a comparison, not by a profile. HC-Rust normally loads a FieldWorks project +through its own snapshot reader. Here it was also run on the XML that C# itself exports for the same +project. On the private project's slowest word: + +| Engine and input | Lexical lookups | +|---|---| +| C#, its own export | 49 | +| HC-Rust, C#'s export | 18 | +| HC-Rust, its own snapshot | 1,655 | + +These counts were supplied by the round's brief; this write-up did not reproduce them. A ~90x gap +between two inputs to the same engine is a loading or compilation difference, not a search-strategy +difference. Following it led to both identity bugs: + +- **A (entry 090 follow-up).** HC-Rust's analysis-side affix literals carried identity only through + a single 64-bit id lane. That lane does not exist above 64 character definitions. Provisional + letters pushed the private project to 70, and the grammar has no phonological features, so + nothing else distinguished segments. An enclitic literal therefore matched *any* final segment. + Every word long enough was unapplied against it, and synthesis rejected each bogus candidate. C# + gives a featureless segment a `StrRep` `EqualTo` constraint (`CharacterDefinitionTable.cs:68` at +`a4b29742`, the same snapshot the research doc's anchors use), so + its literal matches only itself. The fix applies HC-Rust's chunked `StrRep` matcher on this path, + only for featureless grammars. Featured wide tables (Amharic) keep the feature-only + over-approximation, which C# shares, because a featured node has no `StrRep`. +- **B (entry 103).** `NarrowAnalysisRewriteRuleSpec.Unapply` re-inserts a deletion rule's LHS + constraint as an optional node. For a `SegmentNaturalClass` that node carries the union of its + members' structures, which on a featureless grammar is "`StrRep` in members", and + `RootAllomorphTrie` unifies against it. HC-Rust gave the node no identity, and its root trie also + ignored a query node's member set. Analysis and root lookup could therefore take any segment at + every re-insertion site. The fix sets the member set on featureless grammars only, and the trie + refuses an edge only when C#'s unification would fail. + +**Neither fix is a C# optimization.** Before the fixes HC-Rust did more work than C#; afterwards it +does what C# does. They are recorded here because they decided where this round's effort went, and +because they explain several figures in the research brief: +- **The deletion variant's 1 s timeouts.** The brief reported 200-1,600 inputs reaching a 1 s + per-input timeout, from an earlier build that had neither fix; that count was not re-measured. + What was measured is narrower: after fix A and before fix B, 16 held-list words still hit a 1 s + limit, and fix B cut their parser work by 99.9%. Both port bugs are implicated, and the two counts + are not comparable. +- **Where the time went.** In a two-word profile of the deletion variant before fix B, analysis was + 0.1-0.3% of parser work, root lookup about 10% and synthesis about 90%. Zero branches were + lexically dead. No analysis-side pruning could have helped. + +One residual is **not explained**. On the slowest word, HC-Rust on C#'s export made 18 lookups where +C# made 49. Parses were identical, but an efficiency nobody can argue for is not a result. It is +listed under open leads rather than claimed. + +**Method rule.** Before measuring an algorithmic lead on a port, run the port on the oracle's own +export of the same grammar and compare a cheap count, such as lookups per word. A large gap between +the port's two inputs is a port bug, and its fix can dwarf any algorithmic change. This round, it +did so twice out of three. + +## C1: prune analysis branches that cannot reach a root (research lead #1) — rejected on its ceiling + +**Expected.** A sound summary of the remaining transformations would prune analysis nodes from +which no lexical lookup can find a root. This is the research doc's rank-1 candidate, residual +viability. It differs from ledger rows 10 and 11 only if compounding and the remaining morphology +are modelled. + +**Measured.** A hindsight census charged each unapplication's steps and work to its input node. +Folded alternatives alias their canonical. A node is *live* when a lookup on it, on an alias or on +a descendant found a root. Dead cost is split three ways: +- **Dead leaves:** a failing rule battery plus one failing lookup. +- **Dead, may compound:** fewer non-heads than `MaxStemCount`, with a compounding rule in this or a + deeper stratum. +- **Dead internal, plain:** the residual. + +Only the plain bucket is what a sound, compound-aware summary could actually remove at a profit. A +dead leaf costs about as much to prove dead as to run. A may-compound node is row 10's +disqualifier. + +| List (words) | Parser work | Analysis / lookup / synthesis | Lexically-dead ceiling | Dead leaves | Dead, may compound | **Dead internal, plain** | Repeat-lookup work | +|---|---|---|---|---|---|---|---| +| Private, base, held (7,206) | 50,345,810 | 52.7 / 11.8 / 35.2% | 22.6% | 6.1% | 14.6% | **1.9%** | 2.4% | +| Private, v1, held (7,206) | 105,768,202 | 70.5 / 6.7 / 22.6% | 38.7% | 19.9% | 6.6% | **12.2%** | 1.8% | +| Private, deletion variant, held (7,206), before fix B | 303,966,311 | 8.5 / 35.7 / 55.8% | 16.0% | 12.7% | 0.0% | **3.4%** | 10.4% | +| Indonesian (70) | 1,906,012 | 85.8 / 2.1 / 12.0% | 41.0% | 15.3% | 23.8% | **1.9%** | 0.7% | +| Amharic (669 completed) | 39,395,506 | 97.2 / 1.4 / 1.4% | 66.1% | 20.3% | 45.8% | **0.0%** | 0.6% | +| Sena (5,134 completed at 30 s) | 271,894,826 | 89.9 / 3.0 / 7.1% | 55.5% | 11.9% | 39.0% | **4.7%** | 2.2% | +| Mbugwe, first 400 (394 completed) | 610,569,774 | 73.6 / 3.3 / 23.0% | 45.5% | 9.7% | 35.8% | **0.0%** | 3.3% | +| Aweti (183 of 206 completed at 30 s) | 488,089,456 | 93.6 / 1.0 / 5.3% | 74.9% | 35.1% | 0.0% | **39.8%** | 1.0% | + +The census builds include fix A but predate change C. Template-heavy shares (Sena, and possibly +Amharic and Mbugwe) are therefore against an older, larger denominator. For scale, Sena was 271.9M +work units on 5,134 completed words then, and is 240.7M on all 6,146 now. + +**Verdict: not built.** +- **Plain bucket.** It is 0-5% of parser work on six of eight lists. The exceptions are v1 at 12.2% + and Aweti at 39.8%, the latter on 183 completed words. +- **Most dead work is unprunable.** It sits in leaves, or in nodes that could still split off a + compound non-head. Amharic, Sena and Mbugwe put 36-46% of parser work there. +- **Any sound summary has a large scope.** It must model reduplication, infixes, truncation, + deletion re-insertion and compounding before it is allowed to prune anything. + +**What is different from prior attempts.** +- **Row 11 ceiling.** Row 11 found 23.5% pooled dead steps. Dead *steps* here are 43-94% of steps, + so dead branches are common. But they are cheap: the ceiling falls once it is weighted by work + and the unprunable buckets are removed. +- **Row 11's other failure mode persists:** the root is reachable but synthesis fails. On the + deletion variant before fix B, 86% of synthesis work went to canonicals whose lookups hit but + never matched the surface. On Mbugwe the share is 98%. No lexical oracle can prune either. + +**C# transfer.** C#'s `Morpher`, `AnalysisStratumRule` and `AnalysisRewriteRule` run the same +search, so the same buckets should apply. No Machine PR unless the shadow experiment under open +leads shows a sound summary deciding a meaningful share of the plain bucket. + +### Per-parse distinct-shape lookup — rejected + +The brief's "97,575 lookups over 56 shapes" came from trace mode, where analysis merging is off. The +research doc warned that a trace-mode count cannot establish an untraced duplicate rate, and the +untraced census confirms it: +- **Calls per key.** The private project makes 1.13-1.49 lookup calls per distinct + (stratum, shape). +- **Ceiling.** Exact repeat-lookup work is 0.5-3.3% of parser work on every list, or 10.4% on the + deletion variant before fix B. +- **Policy.** Avoiding it would be a per-parse memo, which this work excludes. + +C#'s `LexicalLookup` repeats lookups the same way, so there is no Machine PR. Dead-candidate lookups +are 30-75% of lookup work, but lookup is only 1-12% of parser work, so they were not pursued either. + +## C2: share trailing slot batches inside one battery call (research lead #2) — kept, ported to C# + +**Expected.** The battery walks each template's slots last-first. Templates whose trailing slots +use the same rules, in the same order, with the same optionality, run the same slot batch on the +same word. Factoring the templates into a trie runs that batch once. This is the research doc's +rank-2 candidate. It is control factoring, not a cache: nothing is keyed by a visited word, and +nothing outlives one battery call. + +**Battery share of all parser work (HC-Rust, before C):** + +| Grammar | Battery share | +|---|---| +| Mbugwe | 61% (86 step-capped words are about 2/3 of the total) | +| Private project, v1, held | 45% | +| Sena | 23% | +| Amharic | 21% | +| Private project, base | 8% | +| Private project, loaded from C#'s export | 9% | +| Private project, deletion variant | 6.5% | +| Indonesian | 0 (no templates) | + +**Static census.** This counts rule applications on the battery's own input, before and after +sharing: +- **Shared tails:** Sena 111 -> 45, Mbugwe 397 -> 196, Aweti 143 -> 100. +- **No sharing possible:** Amharic 50 -> 50, and the private project 30 -> 30. Each has one template. + +**Dynamic ceiling, Sena.** A trie-exact key (same trailing slots and same full word state) predicts +448,573 repeated slot batches, which is 30.8M work units or 11.3% of all Sena work. The built trie +avoided exactly 448,573. + +### HC-Rust (PanGloss entry 095) + +Run with `--threads 1 --step-cap 2000000` (Aweti 500000) and no wall timeout. + +| Grammar | Words compared | Parses | Parser work | Steps | Fire count | +|---|---|---|---|---|---| +| Sena | 6,146 | IDENTICAL | **11.5% less** | 31.6% fewer | 171,319 shared, 448,573 avoided (= predicted ceiling) | +| Mbugwe | 1,364 (partial run) | IDENTICAL | **18.1% less** | 19.3% fewer | - | +| Aweti | 170 | IDENTICAL | 0.35% less | 0.16% fewer | 1,939 shared, 2,159 avoided | +| Private project (4 grammar variants: base, v1, deletion, C#'s export; x 3 lists) | all | IDENTICAL | 0 | 0 | 0 | +| Amharic, Indonesian | all | IDENTICAL | 0 | 0 | 0 | + +**Tests.** +- **Differential test.** Compares the shared walk with each template's own walk for all words up to + 5 segments. It fails when optionality is dropped from the trie key. +- **Fire-count test.** Fails with sharing disabled. +- **Conformance.** `-Scope all` passed 2043/2043. Fixtures at the PR #480 head showed the same + failures as the baseline. + +### C# (Machine PR #546) + +`AnalysisAffixTemplateRule` has the same shape, so the trie ported directly to sequential +`ApplySlots`. It is restricted to untraced, sequential walks with no alternative limit. Tracing, +`ParallelApplySlots`, finite alternative limits, custom compiled templates, changed slot +optionality, and grammars without shared tails all keep the original battery. +`AnalysisScope.TemplateMemo` (row 2) still wraps the battery unchanged: row 2 reduces how often a +battery runs, and #546 reduces repeats inside one run, so the two compose. + +The figures below are C# slot batches on deterministic 200-row samples, single-threaded and warmed, +with mode order alternated per input: + +| Input | Completed pairs | Slot batches before -> after | Fewer batches | Parse time, secondary (ms under concurrent load) | +|---|---|---|---|---| +| Synthetic shared-tail fixture (10 words x 100) | 1,000 | 2,000 -> 1,000 | 50.0% | 1.27 s -> 1.05 s | +| Sena (200 sampled rows) | 172 | 47,072 -> 32,287 | 31.4% | 20.07 s -> 16.67 s | +| Mbugwe (200 sampled rows; 61 pairs capped at 2 s in either mode, excluded) | 139 | 72,467 -> 58,486 | 19.3% | 58.43 s -> 51.33 s | + +- **Parity in the samples.** Every completed pair had identical signature lists, with zero + differences and zero errors. +- **Differential test.** `SharedWalkMatchesPerTemplateWalkInOrder` compares ordered `Word` equality + and complete `AnalysisStateKey` sequences across four grammar arrangements and 2,904 input/state + combinations. +- **Fire test.** `UntracedBatteryRunsSharedSlotBatchOnce` fails with sharing disabled (`Expected: 1; + But was: 2`). +- **Unit tests.** HermitCrab passed 126/126. +- **Conformance.** The PR #480-head fixtures were run on the branch assembly and on the unchanged + base `aab237b2`. With memoization on, both were 58 passed / 2 failed. With memoization off, the + branch was 58/2 and the base 59/1. Both assemblies fail traced-rule attribution in + `boundary-leftover-in-narrowing-lhs-strrep`. `deep-optional-affix-nesting` crossed its 15 s budget + in three runs; it has no shared slot tails and takes the original walk. + +These two engines measure different things, so do not compare the columns. C#'s "31.4% fewer slot +batches" on Sena and HC-Rust's "31.6% fewer steps" are not the same unit, and neither is HC-Rust's +11.5% less parser work. + +**Conformance pins.** The C# 60/60 self-check below is `hc.dll` at `72dbd4f9`, the PR #480 head. The +58/60 above is PR #546's base `aab237b2`. They are different assemblies, and the two counts are not +in conflict. + +**What is different from prior attempts.** +- **Row 2.** The battery memo reduces battery *invocations*. C reduces repeated slot batches + *within one invocation*, with no `(Word, state)` table. +- **Row 18.** C is not a cross-word cache. +- **The research doc's prediction holds.** It said this would not turn 43,641 battery calls into + 4,988. It does not; it shrinks each call. + +**Rejected alongside.** A rule-level memo per battery call was no bigger than slot-level sharing on +Sena (33.7M vs 32.9M work units of repeats), and it is a memo. + +## Conformance state at the round's close + +The run used the PR #480 head, Machine `72dbd4f9`. + +- **C# self-check.** `hc-conformance --include-pathological` passed **60/60 fixtures**: 609 entries + in the `words.yaml` files, none skipped. Upstream fixtures grew from 44 to 60 since the previous + pin; nothing was removed. +- **HC-Rust.** One fixture failed: `edge-cases/clitic-rule-designation`, 5 words. Machine PR #529 + added `MorphologicalRule@isCliticRule`, which HC-Rust has not yet ported. It is a parity gap, not a + divergence, and the port is under way. Every other new or modified fixture passes in both engines. + No fixture passes in HC-Rust and fails in C#. + +## How today's measurements bear on the six research reports + +The integrated ranking is in the [research doc](hermitcrab-search-asymptotics-research.md). This +section records only what changed. + +| Lead (report) | Status after this round | +|---|---| +| **1. Residual viability**: abstract interpretation with staged refinement (R6), lazy residual-lexicon product (R3), magic sets / goal propagation (R4), extension oracle (R5) | **Ceiling measured, rejected for now** (C1). The plain dead-internal bucket is 0-5% on six of eight lists. Reopens only through the Aweti/v1 shadow experiment below | +| **2. Template battery as a shared control trie** (derived from R2/R3) | **Built, kept, ported** (C; Machine PR #546) | +| **3. Trace-class POR with recovery**: Mazurkiewicz traces (R1), sleep/source-set DPOR (R1), critical-pair certificates (R1); verified POR (R6) | Not pursued. No new measurement. The earlier 0.00-0.05% eligibility still stands as the warning | +| **4. Exact finite-state compilation of a verified subgrammar** (R3, R6, R2 #3) | Not pursued as an exact engine. A narrower **FST-derived precheck** is the best remaining lead for single-template grammars (open leads) | +| Fixed-cap root-result cache (R5 #2; R4 aside) | **Rejected**: repeat-lookup work is 0.5-3.3% of parser work untraced, and the cache is a memo. The 97,575/56 premise was a trace-mode artefact | +| Count-vector chart DP with packed provenance (R2 #1); packed trail forest for synthesis (R2 #2); `AnalysisStateKey` graph with ZDD trails (R4 #1); analysis-DAG x synthesis-state product (R5 #1) | Not pursued. They are search-growing tables, a closed family. The finding that 86-98% of synthesis work is root-reachable failure is consistent with the reports' own warning that one output analysis does not bound product states | +| Proven rule-precedence DAG (R4 #2) | Not pursued. It needs complete-path certificates (research doc section 1) | +| Symmetry reduction (R1 #4) | Not pursued. The reports themselves rated it a probable no-fit | +| Meet-in-the-middle (R3, R4), A*, k-best and k-shortest paths, reverse search (R4, R5), Earley/CKY, scalar semirings (R2), e-graphs (R4), bimachines, two-level morphology, compile-replace (R3) | Still not recommended. Unchanged | + +**Prediction scored.** The research doc said the trace-mode lookup ratio could not establish an +untraced duplicate rate. That was right: 0.5-3.3%. + +**The finding the reports could not have made.** Where it was measured, much of the private +project's cost was two port bugs: 98.5-99.8% less work on the spot words, and 94.7% less on the +deletion variant's held list. No ranked lead addressed that, because each one assumed the port's +search matched the oracle's. It is not the whole story: with fix A in place, v1's held list still +costs 105.8M work units, 45% of it in a one-template battery that C cannot touch (open lead 1). + +## Open leads, not pursued + +Each lead is tagged **[C# analogue]** where the same question applies to `hc.dll`, and +**[HC-Rust]** where it concerns the port only. + +**Work reductions, with a ceiling to measure first:** +1. **FST-derived precheck before slot-rule traversal, for single-template grammars** [C# analogue]. + C cannot help a grammar with one template, and on the private project's v1 variant the battery + is still 45% of parser work. A necessary condition derived from each slot rule's compiled matcher + could reject a rule before traversal begins. This differs from row 22's edge-segment prefilter in + being judged in work units, since a traversal charges per arc, and in being derived from the + whole matcher rather than its first and last segments. Its ceiling, the work spent in + zero-match traversals, has not been measured. +2. **Shadow summary experiment on Aweti and v1** [C# analogue]. These are the only lists whose + plain dead-internal bucket is material (39.8% and 12.2%). Run a sound summary in shadow mode and + count the plain-bucket work it would actually decide, along with the summary's own cost. Close + lead 1 if the decided share is small. +3. **Mbugwe's step-capped words** [C# analogue]. 86 of 1,645 words hit the 2M step cap and account + for about 2/3 of Mbugwe's work. They have not been examined individually. +4. **Root-reachable synthesis failure** [C# analogue]. 86% (deletion variant, before fix B) and 98% + (Mbugwe) of synthesis work goes to candidates whose lookup hits but whose synthesis never matches + the surface. No lexical oracle reaches this. It is the largest unaddressed bucket on Mbugwe. + +**C# port follow-ons to #546** [C#]: + +5. **`ParallelApplySlots`** needs its own design for its layered breadth-first queue. +6. **Share compiled slot rules across templates**, in addition to slot batches. +7. **Measure in a deterministic work counter.** The C# gain is measured in slot batches on samples, + not in a work counter on full lists. + +**Time-only costs (no work units charged, so outside the owner metric):** +8. **`ana_allomorph_matches`** deep-clones segments with `segs.to_vec()` and rebuilds + `format!("p{i}")` part names on every allomorph call [HC-Rust]. About 5.1M segment-vector builds + are avoidable per private-v1 held run: 5.48M slot-rule attempts against 365K slot batches. + Hoisting `segs_of` or borrowing segments in `Transduce::new` removes them. +9. **`root_trie::search_segs_ident`** checks its active set with `Vec::contains`, which is quadratic + [HC-Rust]. A bitset would be a pure time win. + +**Identity over-approximations left in HC-Rust** [HC-Rust]. Synthesis re-validates all of them, so +none can change a parse, but each over-explores relative to C# on some grammar: + +10. An `InsertContext` (natural-class insertion) in analysis-side `Lhs` still carries no identity + above 64 character definitions (entry 090's remainder). +11. Narrowing leftovers from `ana_narrow_general` keep no class identity (entry 103's remainder). No + measured grammar exercises a class LHS there. +12. `union_narrowing_node` checks only `char_def`, so it ignores an LHS class node's member set. +13. RHS class nodes in synthesis (`syn_narrow`, `syn_epenthesis`) lack identity on featureless + grammars. They were left alone because they affect rendering. + +**Unexplained observations:** +14. **The lookup gap** [HC-Rust vs C#]. HC-Rust on C#'s export made 18 lookups on the slowest word + against C#'s 49. Parses were identical; the cause was not investigated. +15. **Sena SKIPPED rows** [HC-Rust]. A census run with a 30 s wall limit reported 1,012 SKIPPED + rows, while no-limit A/B runs completed all 6,146 words in about 50 s. Not investigated. + +## Data handling + +Private-project word lists, per-word logs and exports stay on the measuring machine. This record +carries only aggregate counts. Conformance fixtures are synthetic. diff --git a/docs/hermitcrab-optimization-ledger.md b/docs/hermitcrab-optimization-ledger.md new file mode 100644 index 000000000..4582360a2 --- /dev/null +++ b/docs/hermitcrab-optimization-ledger.md @@ -0,0 +1,318 @@ +# HermitCrab optimization ledger — tried, closed, do not retry + +**Purpose: stop this work being redone.** One row per optimization attempted, what was expected, +what actually happened, and the number that settled it. If you are about to try something on this +list, read its row first — several of these look irresistible on paper and three of them have +already been independently rediscovered and re-retracted. + +Deliberately in-repo and durable, not evicted into a PR accordion. A closed avenue is only closed +if the next person can find out cheaply that it is closed. + +**This is the only report.** The working docs it was distilled from +(`hermitcrab-packed-forest-research.md`, `hermitcrab-forest-memo-plan.md`, +`hermitcrab-forest-memo-ceiling.md`, `hermitcrab-synthesis-fold-probes.md`) are chronological +process records containing retracted intermediate claims, and are deliberately **not** merged. +They are preserved only as this record. **The evidence branches were deleted after the round +closed**, along with all instrumentation: research scaffolding is not worth maintaining in two +product assemblies, since the probes gate on `volatile bool` reads inside `Matcher.cs`, a hot inner +loop in `SIL.Machine` used by every consumer. + +**`docs/hermitcrab-probe-design.md` is the replacement.** It carries the probe architecture and its +two-assembly gate trap, the eight exclusive wall-time buckets and why `synForward` must be recorded +net of cascade and battery, the per-node decomposition, the census designs, all three key designs +with field-level audits, the classifier's direction trap, and the full A/B methodology with the +three measurement failures that forced each part of it — enough to rebuild any probe in a day. The +expensive part was never the code; it was learning what to measure and which measurements lie. + +**No instrumentation is merged, by design.** The probes gate on `volatile bool` reads inside +`Matcher.cs` — a hot inner loop in `SIL.Machine` used by every consumer — which is an acceptable +cost for a measurement run and not one to carry in product code indefinitely. Rebuilding a probe +from a branch is cheap; maintaining research scaffolding in two product assemblies is not. + +## Unmeasured search-asymptotics research + +[Analysis-search research](hermitcrab-search-asymptotics-research.md) integrates six independent reports, +checks their code claims, and separates new experiments from this ledger's closed families. It ranks +residual lexical viability, a grammar-compiled template battery, recoverable trace-class reduction, and +exact finite-state compilation for a verified subgrammar. It maps them to lexicon-driven pruning during +unapplication and making the affix-template battery cheaper without memoization. The research doc itself claims no measurement. Its first two candidates were measured on +2026-10-21 (rows 27 and 26): see [the search-cost round](hermitcrab-2026-10-21-search-cost-round.md). + +## Final disposition of the September follow-up + +The follow-up branch does deliver product improvements, retained on `perf/hc-optimization-archive` and not +merged to master. Its authoritative summary is `hermitcrab-perf-2026-09-counts.md`, and each retained +optimization has one record under `docs/optimizations/`. The retained set is: deferred template cloning, +feature-structure traversal-state reuse, frozen-flat-tree equality, frozen-flat-tree cloning, single-pass +`Shape.CopyTo` mapping, copy-on-write shape sharing, flat FST register arrays, the analysis edge-segment +prefilter, and copy-on-write syntactic feature structures. + +Four attractive experiments were deliberately excluded from the final product diff and are recorded under +`docs/rejected-optimizations/`: the negative two-pass nondeterministic traversal, the inert final-template +prune, the zero-value alternatives memo/hoist, and the low-ceiling compounding split prefilter. This final +disposition supersedes intermediate “ships” language for those experiments later in this historical ledger. + +The minimized branch passed all 603 runnable tests in the conformance-inclusive suite (one expected Windows +symlink-privilege skip). A fixed three-word sample matched on `WordAnalysisSignature` on Sena, Amharic, +Indonesian, Mbugwe, and Aweti; on Sena, Mbugwe and Amharic that signature compares morph counts, not +analyses (`hermitcrab-perf-2026-09-disposition.md`). Only Amharic (1.76x) and Mbugwe (1.83x) cleared the +recorded spread and timing reliability rules. The other three ratios are retained as directional evidence +only. + +## Earlier ledger round: net speed delivered none + +No optimization in the "closed" section shipped, and none of them made anything faster. The +fastest thing tried was 4% **slower**. The pre-existing wins in this codebase (rows 1–3) predate +this work. What this effort produced is the measurement, the closed family, the located target, +and this ledger. + +--- + +## Earlier shipped results + +| # | Optimization | Expected | Outcome | Number | +| --- | --- | --- | --- | --- | +| 1 | Memoize analysis morphological-rule cascade | Same state reached by many rule orders; cache it | Works, but the cascade was never the cost | 2,555 expansions vs a 2,546-state floor; 1.4 s of a 30.5 s word | +| 2 | Memoize affix-template battery (same key) | Battery was 93% of wall time | **The big win.** Reduced how *often* the battery runs | 38,840 runs -> 2,581; word 30.5 s -> **6.1 s** | + +### Retracted historical prototype + +Row 3 in the original ledger claimed that shape sharing through `CloneShareFrozenShape` had shipped. No +reachable production commit carried that implementation. Its historical prototype measured 4.5–7.6% fewer +allocated bytes and no wall-time change. The September follow-up independently built a different, retained +copy-on-write design through `CloneForEngine`; see `docs/optimizations/shape-sharing-on-clone.md`. + +## Closed — measured, does not pay + +| # | Optimization | Expected | Why it failed | Number | +| --- | --- | --- | --- | --- | +| 4 | **Narrow `AnalysisStateKey`** (drop strictly-shrinking rules' un-application counts) | `MaxApplicationCount` defaults to 1, so the count multiset is "which subset of rules was used" — 2^27 states. Dropping 19 of 27 rules gives 2^8 | Shape + syntactic FS + realizational FS *already* discriminate. The count component was implied by its neighbours. A worst-case bound collapsing is not the realised state count collapsing | Realised state collapse **Sena 1.12x** (state-weighted) vs a 1.3 gate; **4 of 7** heavy probes exactly 1.00, including the 2,556-state probe. The three that moved were 1.13x, 1.16x, and **1.38x**, and all returned zero parses: R tracks failure, not size. **Reopening condition:** a grammar with genuinely high R; the census is cheap and is the right first question to ask of any new grammar | +| 5 | **Fold-step sharing memo** (share deterministic synthesis fold steps by computed value) | Probe measured 3.22x and 8.10x shareable steps; ~50% wall ceiling on two reliable fixtures | A sound key must carry the **ordered remaining trail**. Two candidates then have to agree on their entire future to share a step, which almost never happens | **hits = 0** on the best fixture; **0.96x** (4.5% *slower* — pure key-construction cost); 1.06x on the second, inside 21.9% noise | +| 6 | **Order-insensitive synthesis-input dedupe** | 9,774x duplicate synthesis inputs | HermitCrab's morphological rules are **not order-invariant**: same rule multiset, different order, different output. Merging loses parses | 2 violations on Sena, 12 independently on Indonesian. Sound version: **15–40%** | +| 7 | **Trail-position indexing of synthesis rules** | 11,445,538 of 11,445,538 rejections are "this rule is not the pending trail rule" — 100% of the histogram | Each rejection is one array index plus one reference compare. Count is not cost | 11.4M x 29 ns = 0.33 s of 143 s = **0.2%** | +| 8 | Synthesis-side length bound (Gate A) | Reject candidates that cannot reach the surface length | At the comparison point the candidate is still the bare root; its affix trail applies later inside `_synthesisRule.Apply`. Rejected valid parses | Unit suite 64 -> 34 passing. Reverted | +| 9 | Analysis length ceiling (Gate B) | Same idea, analysis side | Sound and correct. No available corpus contains the pathology it prunes | Byte-identical output, **no measurable speedup** (~4–7% slower, within noise) | +| 10 | Lexical reachability gate (Phase 5) | Prune candidates from which no lexicon root is reachable | Both reference corpora have real compounding in their deepest stratum, which disqualifies the gate everywhere | Proven **no-op** on both corpora | +| 11 | Tandem lexical intersection (T2) | Kill doomed branches early via the lexicon | The oracle only sees lexical unreachability. The expensive words fail on checks that run *after* lexical lookup succeeds | Pooled **23.5%** dead steps vs a 30% gate; one failure word at exactly **0.0%** dead | +| 21 | **Memo-key caching on the frozen `Word`** | Keys are built ~28x per distinct state, which looked like a cheap win. | High count did not imply high cost; caching would add state and invalidation complexity around negligible work. | **CLOSED:** all `AnalysisStateKey` construction and hashing on Amharic measured **3 ms of 81,707 ms = 0.0%**. | +| 12 | Pool small short-lived collections | Fewer allocations, faster | `HashSet/Dictionary.Clear()` is O(capacity); Gen0 already beats pooling at this size | -15–17% bytes but **+8.6% wall**, +12% on the parallel battery. Reverted | +| 23 | **`(State, AnnotationIndex)` traversal dedupe** (PR #511) | `Advance` forks per `Optional` annotation, so instances are exponential in optional count and independent of `|States|`; keying on state+position bounds it at `N * |States|` | The key omits what filters the traversal. Variable bindings gate later arcs; registers carry the captures `AllSubmatches` consumers read | **112 of 20,000 differential-fuzz cases change `Match()`** (108 bindings, 4 shortened ranges) while `AllMatches().First()` is identical in all 20,000. Narrowed to the deterministic method with no capture groups: **0 divergences**, full bound preserved. Extension censuses: environment matchers are **0.20%/1.18%** of instances (Amharic/Mbugwe); the analysis rules' apparent **93.95%** collapse is **99.7% alternate captures**. See `rejected-optimizations/state-position-traversal-dedup.md` | + +## Closed before building — motivation refuted by measurement + +| # | Optimization | Expected | Why it died | +| --- | --- | --- | --- | +| 13 | Surface-length pruning inside the synthesis fold (P2) | "The only lever on Amharic's ~160 ms per synthesis run" | Amharic's forward synthesis is **14 ms**. The 160 ms was arithmetic on an unmeasured denominator — dividing 30 s by 186 synthesis inputs and assuming the time was there | +| 14 | Nogood lattice subsumption (P4) | Nogood hits dominate positive replays **434,628 to 25,102** — the engine mostly proves subtrees *empty*, and generalising "empty" over the feature lattice would turn nogoods from points into regions | **CLOSED on direct evidence** (it was previously closed on adjacent evidence, which is why it was reopened). Two heavy Sena probes produced 16,176 expansions -> 16,176 distinct keys -> **16,176 singleton buckets, 0 subsumable (0.0%)**, and 12,595 -> 12,595 -> **12,595 singletons, 0 subsumable**. Not one pair of nogood entries even shares a `(shape, stratum, counts, non-head)` bucket, so there is nothing for a subsumption ordering to act on. Breadth across all 33 conformance fixtures agrees: 7,493 empty expansions, 7,398 distinct keys, **0 subsumable (0.0%)** pooled, every non-empty bucket a singleton. Direction checked explicitly as `earlier.Subsumes(later)` (E subsumes Q, general over specific); the reverse would be unsound. Independently reconfirms row 4 from a different angle: the keys are maximally discriminating | +| 15 | Move constraints into the analysis key (Maxwell & Kaplan's category-splitting) | Their biggest measured win: make the chart prune what the constraint solver otherwise would | Everything synthesis rejects on is root- or realization-dependent, and the root is unknown until `LexicalLookup`. Also a Sena analysis state costs ~0.73 ms while a synthesis input costs at most ~0.12 ms — **states are dearer than synthesis inputs** | + +## Reopened this round, and their outcomes + +| # | Optimization | Expected / observation | Outcome | +| --- | --- | --- | --- | +| 22 | **Analysis-side rule prefilter** | 94.4% of Sena allomorph pattern matches failed, so a cheap necessary-condition check had a large direct-work ceiling. | **RETAINED after the earlier self-time cap was refuted by direct work counts.** The original ~10% ceiling counted matcher self-time but omitted traversal allocation, feature unification, and GC booked elsewhere. The implemented edge-segment prefilter removed **87% of FST traversals**, 54% of arc checks, and 85% of traversal instances on 45 Sena words, with `WordAnalysisSignature` parity on those words (a morph-count comparison on Sena; see `hermitcrab-perf-2026-09-disposition.md`) and all 464 conformance-fixture words. See `docs/optimizations/edge-segment-prefilter.md`. | +| 16 | **`ExpandAlternatives` dedupe** | 20.7% of Sena wall, the largest bucket nobody had instrumented. Apparent census: 395,026 alternatives -> **61** distinct (0.0%), 85% of duplicates same-analysis-word | **CLOSED on the sound census.** With the pending-trail content added to the key, 395,026 -> **351,414 distinct (89.0%)** -- only **11%** are duplicates at all, and all 43,612 of them are same-analysis-word. Sound ceiling = 20.7% x 0.110 = **2.28% of Sena wall**, against a gate of >=10% to build and <5% to close. **The fourth apparent redundancy to evaporate under a complete key** | +| 17 | **Per-node cost in the analysis cascade** | 84% of the Amharic cascade is not linguistic computation; clone is 46.7% and 37.4% was unattributed | **Partly settled, remainder still open.** (a) The remainder split **MISSES its <15% gate by ~2.5x** — still ~38% on Amharic `anCascade`. Memo, Trail and **Equality each measured <0.2%**, so `Word.ValueEquals` behind the cascade's `HashSet` is *not* the missing mass, contrary to the standing hypothesis. **Two caveats the first write-up missed.** (i) The ~38% figure comes from a *contended* run — three timing probes were live — and descheduled wall time lands preferentially in the remainder while deflating every self-time share, so **CPU starvation is itself an unbracketed candidate biasing in exactly the direction observed**. (ii) The claim that GC pause time is unmeasurable is **wrong at the measurement point**: only the library assemblies are netstandard2.0; the probe harness lives in `tests/SIL.Machine.Morphology.HermitCrab.Tests`, which targets **net10.0**, so `GC.GetTotalPauseDuration()` and `GC.GetGCMemoryInfo().PauseTimePercentage` are directly callable. Next probe, gated: re-run serial — remainder <20% means contention was the missing mass and 17(a) recloses on the cheaper-clone target; still >=30% then bracket with `GetTotalPauseDuration()` deltas — ΔPause >=50% of the remainder's absolute ms confirms GC (build target converges with clone/freeze at 46.7%), <=15% refutes it and the next split is enumerator/LINQ orchestration (the 2026-07 alloc audit already flagged `ExecuteCommands` enumerator boxing); a collection census over 24 Amharic words gives gen0 8,217 / gen1 2,729 / gen2 301, an unusually high gen1+gen2 ratio consistent with — not proof of — GC as the remainder. (b) **Clone-discard fraction = 66.2%** (created 2,505, discarded 1,659), a count and therefore contention-immune, against a >=70% gate: **misses by ~4 points, so the build is "make a clone cheaper", not "defer cloning"**. Scoped to `AnalysisAffixTemplateRule`'s eager per-template clone, the one clone-before-verification site; the other four analysis-side sites clone only after a successful match and were excluded rather than blended in | +| 18 | Corpus-scope memoization | `AnalysisScope` dies per word, but `AnalysisStateKey` is word-independent by construction, so entries are reusable across a corpus. Nogood hits dominate positive replays 10-17x, making cross-word nogood reuse the cheapest available form of reuse | **CLOSED, and worse than a no-op.** Indonesian, 121 words, **serial on a clean machine: 1.015x, 1.4% wall-clock reduction** against a >=20% build gate and <5% close gate. Costs **68 MB** added peak managed memory. Cross-word reuse is real but small: 39 memo, 73 nogood, 237 template-nogood hits. Token-vs-type: **121 tokens, 121 distinct**, so no type-level dedupe is available either. **A first, contended run of the same harness reported 2.385x / 58.1%** with a 210.9% spread on the shared arm against 13.7% on the baseline; two testhosts were live. The counts were byte-identical across both runs — only the timing was contaminated, and min-of-N on a high-spread arm picks the luckiest sample. Analysis-set equality held throughout, so the key genuinely is word-independent. **Sena, first 300 words, is the decisive case: the shared scope is 28-32% *slower* than per-word scopes** (two clean serial runs, no contention: 282.7s vs 372.2s and 308.9s vs 396.7s baseline-vs-shared; deterministic counters byte-identical across both runs), and costs **~29.3 GB** added peak managed memory against a 2 GB budget. Root cause: `AnalysisScope.MaxMemoEntries` (100,000) is a *per-scope*, not per-word, budget. Two heavy early words (#74, #76 of 300) alone fill the shared `Memo` table to the cap and `TemplateMemo` to 55,071; every word after that loses the free within-word memoization a fresh per-word scope would have given it for nothing, because `Store` silently no-ops past the cap (see its comment) with no eviction. Cross-word hits (shared minus baseline) are **negative** — memo -77,648, nogood -747,607 — i.e. sharing produced *fewer* total hits than 300 independent fresh-table runs would have on their own; the small `+820` template-hit / `+9` template-nogood gain does not come close to covering it. Amharic, capped at 28 words (word 29 hangs, excluded per standing guidance): **0.9% wall-clock reduction**, table nowhere near the cap (Memo 1,251, TemplateMemo 652 of 100,000), consistent with row 17's finding that Amharic's cost is dominated by clone/allocation per node rather than distinct-state count, so avoiding a handful of cross-word re-derivations barely moves wall time. Token-vs-type on all three corpora, at the runs' own sizes and on each corpus's full word list: **zero duplicate tokens** — whole-word caching is not available as a cheaper alternative anywhere here. No analysis-set divergence on any corpus, at any scope regime: the stop condition that would have put rows 1-3 in question did not fire. **A shared-but-capped memo table is not "the same optimization, smaller ceiling" — past the cap it actively takes away the free per-word memoization the product already has, and retains every live entry for the run's full duration instead of per-word, which is where the 29 GB comes from.** Any future attempt needs real eviction (LRU, generational, or per-word sub-budgets) before this is worth re-measuring, and even then the >=20% gate looks unreachable given Indonesian's ceiling under an uncapped-in-practice regime was already only 1.4% | +| 19 | Contexted constraints / abstract feature-only replay | Pack without merging states by replaying only shape-independent constraints. | **CLOSED.** Cost-weighted P1b die-point histograms put shape-free work at **6.0%** and **0.0%** on two heavy Sena probes and **0.4%** on 28 Amharic words; all fire the <30% close gate. All 33 conformance fixtures were measured for breadth: 24 close, 3 nominally fire the build gate, 3 are undecided, 2 have zero rejections, and 1 has zero words. The three build-gate fixtures are all sub-50 ms and cannot carry timing claims. One reliable fixture confirms that count and cost can diverge, but die-point rejection cost itself is only ~0.1–0.2% of wall on the real corpora. Abstract replay would optimize a phase that costs approximately nothing on the corpora treated as ground truth. Not built. | +| 20 | **Row 5's family on `GenerateWords`** (generation, ~100% synthesis share vs parsing's ~5%/~0.3%) | Row 20 was unmeasured; generation's share is ~100% so row 5's fold ratios would apply at face value if they transferred. `UseSynthesisFoldMemo` had never been wired into `GenerateWords` (only into the parse-then-synthesize path) — wired in under the same untraced/`MaxDegreeOfParallelism==1` restriction to make this measurable at all | Hits are no longer uniformly zero: 3 of 23 lookup-reaching fixtures clear the 5% gate (diacritic-segments 33.3%, suffixing-vowel-harmony 28.6%, suffixing-evidential-adjacency-chain 37.7%; a 4th sits just under it at 4.8%), reopening row 5 narrowly for generation. But the reopened case recloses immediately: the A/B on the best fixture shows the wall-clock "win" sitting inside its own noise floor, and the pooled hit rate is dominated by `deep-optional-affix-nesting` (925 of 1,613 lookups) at hits=0, reproducing row 5's exact 4.5%-slower key-construction cost on the identical fixture. Parity: 0 divergences, 1,236 generations, 32/33 fixtures (33rd throws identically both sides). Best case: 130 hits / 5 reps (37.7% hit rate), **7.8%** wall-clock reduction inside a **22.2%** off-arm spread — not a result. 19/23 lookup-reaching fixtures at exactly **0%**; pooled hit rate **2.67%** (43/1,613), skewed by the one large zero-hit fixture. | + +## 2026-10-21 search-cost round + +Full record: [hermitcrab-2026-10-21-search-cost-round.md](hermitcrab-2026-10-21-search-cost-round.md). +The unit is HC-Rust parser work (summed per-word `work_steps`, deterministic), so "N% less work to +parse the words". Steps are secondary, and ms are labelled. C# figures are in C#'s own counters. +Rows 24 and 25 are **HC-Rust port bugs: C# was already exact, and no C# change follows from them.** + +| # | Optimization | Expected | Outcome | Number | +| --- | --- | --- | --- | --- | +| 24 | **Analysis-side affix literals keep `StrRep` identity on wide featureless tables** (HC-Rust only; PanGloss divergence 090 follow-up) | Running HC-Rust on C#'s own export of a private featureless project made 18 lookups on its slowest word, against 1,655 on HC-Rust's own snapshot (C#: 49). That points at a loading difference, not search | **KEPT in HC-Rust; no C# change.** Above 64 character definitions HC-Rust lost the literal's identity, so an enclitic matched any final segment. C# gives featureless segments a `StrRep` `EqualTo` constraint (`CharacterDefinitionTable.cs:68`) | 7 spot words: **98.5% less work** (2,365,602 -> 35,913); worst variant **99.8% less** (50,778,583 -> 83,677). Steps on full lists fell 46-55x. Parses IDENTICAL on three private lists and on Amharic, Sena and Indonesian | +| 25 | **Re-inserted segment-class node keeps its members on featureless grammars** (HC-Rust only; PanGloss entry 103, `414ac6b5`) | A deletion-rule variant of the same project hit 1 s timeouts. A profile showed analysis at 0.1-0.3% of work, synthesis at ~90%, and zero lexically dead branches | **KEPT in HC-Rust; no C# change.** HC-Rust's re-inserted node matched any segment, and its root trie ignored the member set. C# unifies against the class's `StrRep` union | Held list (7,206 words): **94.7% less work** (303,966,311 -> 16,220,190). The 16 words at the 1 s limit: **99.9% less**. IDENTICAL everywhere; byte-identical work where it cannot fire | +| 26 | **Shared trailing slot trie in the template battery** (research candidate 2; PanGloss 095; Machine **PR #546**) | Templates with identical trailing slots (rules, order, optionality) run the same slot batch on the same word. Sena's trie-exact ceiling is 448,573 batches, 11.3% of work | **KEPT and ported.** Control factoring within one battery call, with no cache. In C# it covers sequential, untraced walks without an alternative limit. It composes with row 2 | HC-Rust: Sena **11.5% less work** (31.6% fewer steps, 448,573 batches avoided = ceiling); Mbugwe 18.1% less (1,364-word partial); Aweti 0.35%; 0 on one-template grammars. C# batches: Sena sample 47,072 -> 32,287 (31.4%); Mbugwe sample 72,467 -> 58,486 (19.3%). C# tests: a 2,904-combination differential, 126/126 HermitCrab tests, conformance 58/60 equal to the unchanged base `aab237b2` with memo on | +| 27 | **Prune analysis branches that cannot reach a root** (research candidate 1; census `d1690c13`) | The hindsight lexically-dead ceiling is 16-75% of parser work | **CLOSED on the prunable share.** Most dead work is leaves, which cost as much to prove dead as to run, or nodes that may still compound (row 10's disqualifier, 36-46% on Amharic, Sena and Mbugwe). Root-reachable synthesis failure is untouched: 86-98% of synthesis work where measured | The plain dead-internal bucket, the only one a sound summary could remove at a profit, is **0-5% on 6 of 8 lists**. The exceptions are v1 at 12.2% and Aweti at 39.8%. **Reopening condition:** a shadow summary on Aweti/v1 deciding a meaningful share of that bucket | +| 28 | **Per-parse distinct-shape lookup dedupe / fixed-cap root-result cache** (research reports R4, R5) | "97,575 lookups over 56 shapes" | **CLOSED.** That figure was trace mode, which disables merging. It is a memo regardless | Untraced, 1.13-1.49 calls per distinct (stratum, shape). Repeat-lookup work is **0.5-3.3%** of parser work (10.4% only on row 25's pre-fix bug). Dead-candidate lookups are 30-75% of lookup work, but lookup is 1-12% of parser work | +| 29 | **Rule-level memo per battery call** | Repeated slot-rule applications inside one battery | **CLOSED.** It is no bigger than row 26 and it is a memo | Sena repeats: 33.7M work units (rule-level) vs 32.9M (slot-level) | +| 30 | **Hoist `segs_of` / borrow segments in `Transduce::new`; bitset for the root-trie active set** (HC-Rust) | Fewer allocations and a less quadratic trie walk | **NOT BUILT.** Time-only, with 0 work units by construction, so outside the owner metric | ~5.1M avoidable segment-vector builds per private-v1 held run (5.48M slot-rule attempts vs 365K slot batches). `Vec::contains` on the trie active set is quadratic | + +**Lesson of the round:** two of its three large reductions (rows 24 and 25) were port bugs, found +by running the port on the oracle's own grammar export and comparing lookup counts. Do that before +optimizing the algorithm. + +--- + +## The lesson that generalises + +Four independent measurements, four boundaries, same collapse: + +| measurement | key used | apparent | sound | +| --- | --- | --- | --- | +| synthesis-input dedupe (#6) | order-insensitive | 9,774x | 15–40% | +| fold-entry census | trail position only | 6,476x | not established | +| fold-step sharing (#5) | trail position only | 3.22x / 8.10x | **hits = 0** | +| FSA traversal dedupe (#23) | state + position | 93.95% collapsible | **0.28%** register-identical | + +**The redundancy in HermitCrab's synthesis is apparent, not real. The trail is what makes each +step distinct, and every measurement showing large shareable work is measuring a key that omits +it.** The thing that makes sharing visible is the thing that makes sharing wrong. + +That closes a *family*: packed parse forests **of the merge-by-key kind**, fold-step sharing and +synthesis-input dedupe all require distinct derivations to converge on a genuinely identical +state. In this engine they do not converge -- the same fact as the rules being non-order-invariant, +seen from the other side. This argument did not itself close row 19 because a contexted-constraint forest +packs without merging states; row 19 was later measured independently and closed by its die-point costs. + +## Where the time actually is + +Measured with eight exclusive buckets summing to wall. Amharic is a single run (`unaccounted` +0.1%). **Sena is spliced from two runs -- see the note in its row.** + +| grammar | breakdown | +| --- | --- | +| **Sena** heavy words | battery **51.4%**, cascade 18.4%, all synthesis ~5% -- *pre-`ExpandAlternatives` run (143,303 ms, unaccounted 20.1%)*; `ExpandAlternatives` **20.3%**, unaccounted 1.5% -- *later run (112,713 ms)*. **Two runs: not co-summable.** One re-run would put all Sena shares on one denominator | +| **Amharic** | analysis cascade **~95%** of analysis, which is 99.5% of wall; **all synthesis 0.3%** | + +**Every optimization in this ledger that worked reduced how many nodes are visited. None touched +what a node costs.** Amharic spends **~170 ms per analysis state**. (The *measured* state-count floor -- 2,555 +expansions against 2,546 states -- is a Sena heavy probe. For Amharic, near-floor behaviour is +**inferred** from R ~ 1, which compares the full key against the narrowed key, not against a +semantic floor.) + +### Inside one node (Amharic, 28 words, both probe layers live) + +| bucket | `anCascade` 81,707 ms | `anBattery` 7,368 ms | +| --- | --- | --- | +| clone / freeze / allocation | **46.7%** | 41.9% | +| **remainder (unattributed)** | **37.4%** | 15.4% | +| matcher traversal | 12.5% | **39.3%** | +| FeatureStruct algebra | 3.4% | 3.5% | +| memo-key construction | 0.0% | 0.0% | + +Censuses: 27,385 rule attempts — guards reject 19.5% (application-count 6.3%, unifiability 13.2%), +**80.5% reach matching**. Of 25,178 allomorph pattern matches, **1,938 succeed (7.7%)**, 23,240 +fail (**92.3%**). + +**The headline: only ~16% of the Amharic analysis cascade is *identified* as linguistic +computation** -- matching 12.5% plus feature unification 3.4%. Clone/freeze is **46.7%** and +**37.4% is unattributed**, which is unknown rather than known-non-linguistic. The defensible +statement is that at most a sixth of the cascade has been shown to decide linguistic questions; +of the rest, half is measured (clone) and half is still open. + +**Gate outcomes, written before the run.** "Any bucket >=50% becomes the sole build target" — clone +is 46.7%, does not fire. "Pattern matching dominates *and* >=80% of attempts fail -> build a +prefilter" — 92.3% do fail, but matching is 12.5%, so it does not dominate; does not fire. +**"Remainder >=30% means the decomposition is incomplete and that is the result" — 37.4%, FIRES.** + +So there is no declared build target yet. The next probe splits the 37.4%: `RuleCascade` / +enumerator orchestration, trail dictionary operations, LINQ allocation, memo lookups, GC. + +**A caution on the 46.7% before anyone acts on it.** `Word.Clone` being half the cascade does not +by itself mean cloning is waste — the cascade forks a `Word` per rule application precisely so it +can explore alternatives without mutating shared state. The deciding question is what fraction of +clones are *discarded unused*: cloned, rejected by a guard or a failed match, thrown away. Given +92.3% of pattern matches fail, that fraction has now been **measured at 66.2%** (row 17b), which misses the >=70% gate and points at the cheaper-clone build rather than the defer-clone one. If most +clones are discarded, the lever is **not cloning until a rule commits**. If most survive, the lever +is **making a clone cheaper** (copy-on-write shape, which the RUSTIFY branch already explored). +Different builds; measure first. **Gate: probe the clone-discard fraction; >=70% discarded means +the defer-clone build, below that the cheaper-clone build.** + +## Two durable code constraints + +Neither is expressible in code, and both bite silently. + +1. **`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and + `_nonHeadApps` only. This is safe *solely* because `AnalysisStateKey` includes the per-rule + count multiset, which guarantees a memo hit has identical arrival and stored-arrival counts. + **Anyone narrowing that key breaks correctness with no test failure.** Fix, if the key ever + changes: store arrival counts on `MemoEntry` and compute `stored - storedArrival + query` — a + no-op under today's key, which is also how to test it. This is the one code change from this + work worth making on its own merits, as defensive hardening. +2. **HermitCrab's morphological rules are not order-invariant.** Same pending-rule multiset, + different application order, different synthesis output — 2 cases on Sena, 12 independently on + Indonesian. This bounds every packed-readout scheme. It is the assumption + Maxwell and Kaplan (CL 19(4):571–590) rely on and this engine does not satisfy. + +## Calibration record — predictions vs outcomes + +Written before the measurements, scored after. Kept because the misses are informative. + +| prediction | outcome | +| --- | --- | +| Sena key collapse 2–4x | **1.12x** — wrong, and it was the number that decided the project | +| Indonesian key collapse 1.0–1.3x | 1.17x — right | +| Amharic "most likely to disappoint" | best static classification of the three (31 of 36 rules) — wrong | +| Fold sharing ~50% wall ceiling | **0.96x realised** — wrong; the ceiling was measured with a key that would lose parses | +| Amharic synthesis-bound at ~160 ms/run | forward synthesis is **14 ms** — wrong; arithmetic on an unmeasured denominator | + +## Method rules earned here + +1. **Counting is not timing.** Three published findings in this work were retracted, all arithmetic + on an unmeasured denominator: "synthesis is the bottleneck" (candidate counts), "Amharic is + synthesis-bound" (30 s / 186 inputs), "fixtures only look synthesis-heavy because they are small" + (divided by `synForward`, which excludes the buckets the work runs in). +2. **A ratio measured with an incomplete key is not a ratio.** See the table above, three times. +3. **Warm-up dominates small fixtures.** A first A/B here showed 1.53x purely from JIT; the off-arm + alone varied 42.7% between its own first and second sample. Discard warm-up, take min of N, + interleave arms, and print the off-arm spread as a noise floor beside every speedup. +4. **Sub-50 ms fixtures cannot carry a timing claim.** One 10 ms fixture's share moved 1.3% -> 14.5% + between two runs of identical code. +5. **Three grammars minimum, reported unpooled.** Maxwell & Kaplan measured a 100x swing between two + variants of one grammar. A pooled average across fixtures of different size has had to be + retracted twice in this project. +6. **Min-of-N is only valid when the arm's spread is small relative to the effect.** Row 18's + first run read 2.385x with a 210.9% spread on one arm and 13.7% on the other; serial and clean + it is **1.015x**. Min-of-N picks each arm's luckiest sample independently, so a high-spread arm + wins spuriously. Print the spread beside every speedup and refuse a number whose spread exceeds + it. +7. **Never dispatch timing-sensitive probes concurrently.** Three at once put four testhosts on the + machine and invalidated the bucket percentages in rows 17 and 22 as well as row 18's headline. + Counts survive contention; shares and speedups do not. +8. **A shared bounded cache without eviction is not a smaller version of the optimization** — it + competes with itself for a fixed budget and can land strictly worse than no sharing. Row 18's + shared scope produced *negative* cross-word hits. +9. **Never aggregate the test logger's console output.** It prints every line twice, the second + copy space-prefixed; a `^\s*` grep doubles every total. This produced a 2x error in row 20's + first write-up. +10. **Search completeness is never traded for speed.** HermitCrab is the permanent fallback engine + behind the FST work. +7. **A bucket reporting exactly `0.00` is broken until proven otherwise.** `NodeCostProbe.Enabled` + was declared and read in `Matcher.cs` but set by nothing — two gates in two assemblies, one + wired. Both lower-layer buckets read `0.00ms`, which looks like "measured and negligible" + rather than "never ran", and their time silently inflated the remainder. Wire cross-assembly + probe gates from a single place. +8. **Always carry an explicit remainder column.** It caught the wall-split misattribution and the + dead probe gate. Four of the five retractions in this work would have been caught at the point + of measurement by a remainder that did not sum. + +--- + +## Addendum, September 2026: the round that followed, measured by counts + +Branch `perf/hc-defer-template-clone` (on upstream master, after the merged memoization PR 456) re-ran +this ledger's open questions with deterministic step counters instead of wall time. Its records live +with the code: `docs/hermitcrab-perf-2026-09-counts.md` and one file per change under +`docs/optimizations/`. Corrections to this ledger that fell out of it: + +- **Row 3 ("shape sharing shipped") was not true on master.** No reachable commit carried it; master's + `Word.Clone` deep-copied the Shape and every node feature structure. The follow-up independently implements + a different design: internal `CloneForEngine` copy-on-write with un-share at mutation sites, while public + `Word.Clone` remains a deep mutable clone. A shared shape is frozen, so a missed mutation site throws. +- **Row 22's ~10% cap for an analysis-rule prefilter was wrong by an order of magnitude.** It was computed + from matcher *self-time*. Counted directly, an edge-segment prefilter (patterns are anchored at both ends + and matched by unification, so the first/last literal segment must unify with the shape's first/last + segment) removes **87% of FST traversals** and 54% of arc checks on 45 Sena words, with identical analysis + signatures on every Sena word (morph counts only on Sena; see the disposition doc) and all 464 + conformance-fixture words. +- **"memo-key construction 0.0%" (row 21) was wrong at the profile level.** Hashing each new word's + syntactic feature structure for `AnalysisStateKey` is ~15% of CPU on the heaviest Sena word; the key + itself is cheap but freezing a fresh clone of a nested feature structure is not. +- **Row 16's complete-key result still stands.** An experimental `ExpandAlternatives` memo/hoist did not + reduce clones and was removed. The final branch has no alternatives-expansion change relative to the + pinned production baseline; the intermediate 75% counter reduction is not a shipped result. +- **The ledger never cited John Maxwell's work**: PR 491 (final-template prune in analysis) and the + 2025 `investigate_linear_traverse` branch. Both were tested. The prune was inert on every local corpus + because each has a partial morpheme, and the traversal measured negative by counts (instances used 2.3x) + because the current traverser already pools instances. Both implementations and their toggles were removed; + their rejected-lead records remain. +- **Wall time on the measurement machine moved 2x between windows with identical binaries** (a session + transition in between). Every verdict in this ledger that rests on a wall-time delta under ~20% should + be read with that in mind; the round's method note is to count steps on the sequential path. diff --git a/docs/hermitcrab-perf-2026-09-counts.md b/docs/hermitcrab-perf-2026-09-counts.md new file mode 100644 index 000000000..16cc86354 --- /dev/null +++ b/docs/hermitcrab-perf-2026-09-counts.md @@ -0,0 +1,93 @@ +# HermitCrab optimization round, September 2026: final disposition + +This is the final record for the count-driven round that followed +`hermitcrab-optimization-ledger.md`. Temporary counters were used to choose targets and were then removed; +the final branch carries only product changes, focused tests, the parity/timing harness, and these records. + +## Measurement scope + +The exploration used sequential parsing, one warm-up followed by repeated samples, and canonical analysis +signature sets as the parity gate. Deterministic work counts came from 45 Sena words (five known-heavy probes +plus the first 40 list entries) and 464 words from 33 conformance fixtures. Timing was never run concurrently. +The final branch-wide sample compares production commit `d7347226` (harness-only baseline tip `4f677735`) +with candidate code tip `7370e2dd`. + +The probe counters are intentionally absent from the final diff. They put volatile gates in hot code across +two assemblies and would create permanent maintenance cost for a completed experiment. The reconstruction +design remains in `hermitcrab-probe-design.md`. + +## Retained optimizations + +Each retained mechanism has one record with its invariant, implementation, evidence, parity coverage, +portability, and limits: + +| Optimization | Principal evidence | Record | +| --- | --- | --- | +| Deferred template clone | 1.054x on the 45-word Sena battery; exact parity | `optimizations/deferred-template-clone.md` | +| Feature-structure traversal-state reuse | equality/freezing allocation left the hot profile; not isolated | `optimizations/feature-struct-visited-state-reuse.md` | +| Frozen flat-tree equality | removes visited sets from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-equality.md` | +| Frozen flat-tree clone | removes the copies map from the proved tree case; not isolated | `optimizations/feature-struct-frozen-tree-clone.md` | +| Single-pass `Shape.CopyTo` mapping | removes a second enumeration/dictionary build; not isolated | `optimizations/shape-copy-single-pass-map.md` | +| Copy-on-write shape sharing | clone 36% → 9% of profile; peak working set 11 GB → 6 GB | `optimizations/shape-sharing-on-clone.md` | +| Flat FST register arrays | `CreateInstanceMDArray` 11.5% → 0.2% of profile | `optimizations/flat-register-arrays.md` | +| Analysis edge-segment prefilter | 87% fewer FST traversals and 54% fewer arc checks on 45 Sena words | `optimizations/edge-segment-prefilter.md` | +| Copy-on-write syntactic feature structures | removes repeated clone/freeze/hash work measured at ~15% of a heavy Sena profile | `optimizations/syntactic-feature-structure-sharing.md` | + +Only the deferred clone was timed alone. Profile movement and deterministic work removal justify the other +retained mechanisms; the five-language result below measures the branch as a package and must not be divided +among them. + +## Rejected or removed experiments + +| Experiment | Deciding evidence | Final state | +| --- | --- | --- | +| Two-pass nondeterministic traversal | 2.3x more traversal instances, only 1.6% fewer arc checks, 27% more clones | removed; `rejected-optimizations/two-pass-nondeterministic-traversal.md` | +| Final-template interleaving prune | zero prunes on Sena, Amharic, and Mbugwe because partial morphemes disable it | removed; `rejected-optimizations/final-template-prune.md` | +| `ExpandAlternatives` memo/hoist | word clones unchanged at 4,747,013; no retained difference from production ordering | removed; `rejected-optimizations/alternatives-memo-and-hoist.md` | +| Compounding split prefilter | only ~2% of clones and ~9% of FST traversals were in scope | not built; `rejected-optimizations/compounding-split-prefilter.md` | + +All associated production switches, alternate traversal code, and research counters were deleted. The two +retained internal toggles exist solely for correctness/performance A/B of the edge prefilter and syntactic +feature-structure sharing. + +## Five-language branch-wide sample + +Each row used three fixed word-list indices and two reverse-ordered process rounds (baseline/candidate, then +candidate/baseline). Every process parsed sequentially, discarded one warm-up, and recorded three repetitions, +for six measured samples per arm. Times are the sum of each word's minimum sample across both rounds. Baseline +and candidate used identical grammar names and word keys; `scripts/hc-bench-compare.ps1` compared error +outcomes and rejected missing keys or signature differences. All ten arm comparisons passed the legacy +`WordAnalysisSignature` check, which on these FLEx-exported grammars compares morph counts only (`Morpheme.Id` +is empty; see `hermitcrab-perf-2026-09-disposition.md`), and the saved outputs contained zero parse or load +errors. + +| Language | Word-list indices | Baseline ms | Candidate ms | Ratio | Interpretation | +| --- | --- | ---: | ---: | ---: | --- | +| Sena | 49, 52, 74 | 1,000.7 | 448.3 | 2.23x | directional: 0/3 probes clear both the 50 ms and spread rules | +| Amharic | 8, 9, 267 | 6,410.0 | 3,646.3 | **1.76x** | stable: 3/3 probes clear both rules | +| Indonesian | 0, 9, 44 | 99.0 | 56.0 | 1.77x | exploratory: 0/3 probes clear both rules | +| Mbugwe | 0, 65, 1272 | 8,979.4 | 4,901.6 | **1.83x** | stable: 3/3 probes clear both rules | +| Aweti | 2, 3, 27 | 134.2 | 54.4 | 2.47x | exploratory: 0/3 probes clear both rules | + +The ratios are deliberately unpooled and are not extrapolated to whole corpora. An initially selected heavy +Sena trio did not complete the baseline within five minutes, and an Aweti sample containing index 0 crossed +two minutes; both whole testhosts were terminated and replaced symmetrically before any comparison. This is +sampling evidence, not a claim about those excluded words. + +## Correctness gate + +The minimized code tip was merged into the conformance integration branch and run through the complete +HermitCrab conformance-inclusive project: + +- NUnit discovered and ran 604 non-explicit tests: 603 passed, zero failed. +- One symlink-security test was skipped because the Windows host lacks symlink privilege. +- Fixture manifests, DTD/schema checks, source/graph audits, 452 fixture rows, and 348 coverage items passed. +- The ordinary HermitCrab suite also passed with syntactic feature-structure sharing disabled. + +## What remains + +The edge prefilter leaves roughly 44% of remaining traversals successful, so another edge check has little +headroom. Compounding construction is too small on the measured heavy word. Alternatives and packed-readout +schemes remain bounded by the ordered pending trail. The next credible large change is a persistent/shared +trail representation that can avoid memo-replay and alternative-materialization clones without merging +linguistically distinct derivations; it is a data-structure redesign, not a safe incremental tweak. diff --git a/docs/hermitcrab-perf-2026-09-disposition.md b/docs/hermitcrab-perf-2026-09-disposition.md new file mode 100644 index 000000000..596c4e7cf --- /dev/null +++ b/docs/hermitcrab-perf-2026-09-disposition.md @@ -0,0 +1,135 @@ +# HermitCrab optimization round, September 2026: final disposition of the branch + +**Date:** 2026-09-03. **Branch:** `perf/hc-optimization-pr` @ `179708d2`, renamed `perf/hc-optimization-archive`. +**Decision:** do not merge the branch. Isolate one mechanism onto master as its own small PR; archive the rest with +its records. Re-focus on sillsdev/machine PR #491 and on grammar hygiene, which are where the large returns are. + +## Why the package is not merged + +- 896 source lines across 28 files; about a third inside `SIL.Machine.FeatureModel.FeatureStruct`, the rest in FST + traversal internals, `Word`, `Shape`, and the analysis rules. Not reviewable as one change. +- Only one mechanism (deferred template clone, 1.054x) was ever timed alone. The package figure (1.76x Amharic, + 1.83x Mbugwe) cannot be attributed to its parts (`hermitcrab-perf-2026-09-counts.md`). +- The parity gate that blessed the package compared morph counts, not analyses: `MorpherTests.WordAnalysisSignature` + joins `Morpheme.Id`, which comes from `` and is empty for every FLEx export used. On Sena, Mbugwe + and Amharic the signature therefore degenerates to a morph count per analysis, so every earlier "exact parity" + claim on those grammars verified counts, not analyses. Anything re-measured on them needs a + gloss + allomorph-index + syntactic-feature-structure signature instead. +- Meanwhile PR #491's mechanism, measured with the guard lifted (`AlwaysEnforceFinalTemplates = true`), is + 8–33x on Mbugwe and Sena with identical analysis sets, and 298x on the single heaviest Sena word; Amharic + barely moves because its per-word cost is not in the template interleaving. The branch is worth keeping only + where #491 cannot fire (Amharic-shaped grammars), and only in its most reviewable piece. + +## Disposition per mechanism + +| Mechanism | Lines / files | Evidence on the branch | Disposition | +| --- | --- | --- | --- | +| Analysis edge-segment prefilter | ~120 in 2 rule files + 9 in `Morpher` + tests | 87% fewer FST traversals on 45 Sena words (deterministic counts); one invariant; internal toggle | **Isolated onto master** as `perf/hc-edge-prefilter` (`be7f241d`), see below — **that standalone branch and its own archive tag were later deleted, but the mechanism is preserved in the whole-round archive branch `perf/hc-optimization-archive`** | +| Copy-on-write `Shape` sharing | `Word`, `Shape`, every clone-then-mutate site | clone 36% → 9% of profile; peak working set 11 GB → 6 GB | Archived. Aliasing rules are subtle; re-open only if memory becomes the constraint. Commit `dfd1be94`, `e18ca86d` | +| Copy-on-write syntactic `FeatureStruct` sharing | `Word`, `Morpher`, rules | ~15% of a heavy Sena profile; never isolated | Archived. `4a6a47ed`, `7f9fdef3`, `da35ad4e`, `f08fefa7` | +| `FeatureStruct` equality/clone/freeze fast paths, visited-set pooling | 312 lines in core FeatureModel | "not isolated" | Archived. `325f8419`, `424a3c23`, `b23a7c99`, `00ece2ba`, `2f4dabe0` | +| Flat FST register arrays | FST internals | `CreateInstanceMDArray` 11.5% → 0.2% of profile | Archived. `b1692a7a` | +| Deferred template clone | `AnalysisAffixTemplateRule` | 1.054x alone | Archived. `57703d6b`, `24d75c82` | +| `ExpandAlternatives` memo/hoist | `Morpher` | no retained difference | Already removed on the branch. `d8886b0c`, `f4195be1` | +| Two-pass nondeterministic traversal | FST | negative | Already removed. `c3366fcc` | +| Final-template prune (own version) | analysis rules | zero prunes under the guard | Already removed; superseded by #491 work. `7e6ec59d` | +| PerfBench / compare script / ParseCounters | tests, scripts | harness | Archived. Its signature is degenerate on FLEx exports; any revival must switch to a gloss/allomorph/feature-structure signature first | +| Research records (`docs/optimizations/*`, `docs/rejected-optimizations/*`, ledger, counts, probe design, this file, the #491 re-review) | docs | — | Kept on the archive branch as the record of what was tried and why | + +## The isolated prefilter + +**Status update:** `perf/hc-edge-prefilter` was deleted after this round closed, and its own archive tag was +later deleted too, but the mechanism is not lost: it remains on the whole-round archive +branch `perf/hc-optimization-archive`, which still carries +`Morpher.EdgePrefilterEnabled`, the edge constraints in `AnalysisAffixProcessRule`, and +`AnalysisEdgePrefilterTests`. The description below is kept as a standalone record of the isolated, reviewable +form, in case that smaller diff is preferred over pulling the mechanism out of the full archive again. + +Branch `perf/hc-edge-prefilter`, worktree `.worktrees/prefilter`, one commit `be7f241d` on `origin/master` +(`d7347226`). Content: `AnalysisAffixProcessAllomorphRuleSpec` captures the left/right top-level `Constraint` +feature structures at compile time; `AnalysisAffixProcessRule.Apply` rejects an allomorph before its matcher runs +when the shape's first/last Segment node provably cannot unify with them, using the matcher's own `IsUnifiable` +call and `UseDefaults`; optional edge nodes and non-Constraint edges always pass; tracing emits the same +`MorphologicalRuleNotUnapplied` the matcher would have. `Morpher.EdgePrefilterEnabled` (internal, default true) is +the same-binary A/B switch. The research counters were not carried over. `AnalysisEdgePrefilterTests` (final branch +version, no counter dependency) came with it. HermitCrab suite: 104/104. + +### Standalone measurement (same binary, toggle off = master behavior, toggle on; sequential, memo on) + +Signature: gloss + allomorph index + syntactic feature structure, compared as multisets. Counters: throwaway +increments at the two nondeterministic `Traverse` entries and at `CheckInputMatch` (not committed). + +| Grammar | Words | Wall ms off → on | FST traversals off → on | Arc checks off → on | Parity | +| --- | ---: | ---: | ---: | ---: | --- | +| Sena | 42 (49,52,74,0–39; one invalid line skipped) | 71,296 → 69,900 (−2%) | 8,760,382 → 1,270,122 (−85%) | 12,618,566 → 6,004,919 (−52%) | identical 42/42 | +| Mbugwe | 22 (0,65,1272,1–19) | 110,609 → 96,293 (−13%) | 28,054,872 → 11,419,808 (−59%) | 36,811,510 → 25,960,854 (−29%) | identical 22/22 | +| Amharic | 10 (8,9,267,5,10,12,15,20,25,30) | 111,617 → 103,403 (−7%) | 145,050 → 129,802 (−10%) | 16,705,704 → 15,777,072 (−6%) | identical 10/10 | + +**Reading.** The count reduction reproduces (Sena −85% traversals, matching the branch's −87%), and parity holds +under a real signature. But the traversals removed are the cheap ones: they would have died at their first arc +check, so wall time moves only 2–13%. The branch's package figure (1.76x Amharic, 1.83x Mbugwe) therefore came +mostly from the allocation, clone, and GC work, not from this mechanism. Standalone, the prefilter is correct, +small, and reviewable, but it is **not a large return**. Recommendation at the time: keep `perf/hc-edge-prefilter` +ready as an optional small PR (7–13% on Bantu grammars, where it stacks with #491's prune on the surviving +branches), and do not spend review budget on it ahead of #491. The standalone single-commit branch is deleted +(see status note above); a PR now starts from the mechanism preserved in the whole-round archive +(branch `perf/hc-optimization-archive`) rather than resuming +that branch. + +Raw logs: session scratchpad `pf-sena.log`, `pf-mbugwe.log`, `pf-amharic.log`. + +## Branch and worktree inventory for the round (2026-08-20 to 2026-09-03) + +Every branch below is local unless marked. Worktrees for archived branches were removed on 2026-09-03; the +branches remain. + +| Branch | Tip | What it is | State | +| --- | --- | --- | --- | +| `perf/hc-optimization-archive` (was `perf/hc-optimization-pr`) | `b58b1642`+ | the whole round, code + records | archived, unmerged; worktree `.worktrees/ledger` | +| `perf/hc-defer-template-clone` | `4a6a47ed` | working branch of the round | fully contained in the archive; worktree removed | +| `perf/hc-engine-alloc` | `f22f88a8` | allocation/two-pass side branch | fully contained in the archive; worktree removed | +| `perf/hc-conformance-check` | `0799a977` | archive merged into the conformance integration branch for the 603-test gate | verification only; worktree `.worktrees/hc-conf` can go | +| `perf/hc-edge-prefilter` | `be7f241d` | the one isolated mechanism on `origin/master` | optional small PR at the time; **branch and its own archive tag have both since been deleted, but the mechanism is preserved on branch `perf/hc-optimization-archive`** | +| `pr-491` | `96c97f76` | jtmaxwell3's PR head | worktree `.worktrees/pr491` holds uncommitted throwaway counters + `FinalTemplateUpperBound` harness | +| `docs/hc-optimization-ledger` (remote, PR #490) | `cc806613` | the ledger, now 23 rows | open docs PR | +| `feature/forest-memo`, `feature/synthesis-fold-probes`, `feature/synthesis-fold-sharing`, `feature/per-node-cost` | — | evidence branches of the ledger round | see `hermitcrab-optimization-ledger.md` | +| `investigate-memoization-rule-order` | `4f5aeeb4` | despite the name, conformance-suite commits | belongs to the conformance line, not this round | + +Records in this directory: `hermitcrab-optimization-ledger.md` (23 attempts), `hermitcrab-perf-2026-09-counts.md` +(count round), `hermitcrab-probe-design.md`, `optimizations/*` and `rejected-optimizations/*` (one record per +mechanism), and this file. The round's execution checklist has been folded into this record; it is not kept as +a separate plan document. + +Four further records of the round were deliberately left out of this set, because they concern another author's +open PR and a sibling project rather than this round's own mechanisms. They are described here so that nothing +depends on reading them, and they remain on the `perf/hc-optimization-archive` branch: + +- `pr491-final-template-prune-upper-bound.md` — the measurement record behind the 8–33x figures above: PR #491's + prune with its partiality guard lifted, per-grammar wall time, affix-rule applications and template entries, + plus the analysis-set parity check and the signature caveat quoted earlier. It also proves the guard + over-broad, since partial *lexical entries* can never rescue a pruned branch (only partial rules can, and only + at or below the template's stratum). +- `pr491-review-draft.md` — the review comment drafted from that record: the win with the guard lifted, a + duplicate-parse bug when enforcement is active (an all-slots-skipped clone differing from its input only by + `FinalTemplateState` survives the `ApplyTemplates` equality filter), the rules-only per-stratum guard, and + state transitions that run even while the prune is disabled and cost memo hits. +- `pangloss-final-template-prune-handoff.md` — the same prune handed to PanGloss (the Rust port) for + implementation there. +- `pangloss-hc-rust-speedups-handoff-2026-09-02.md` — speedup leads handed the other way, from PanGloss into + this round. Archived; its section 4 is superseded. + +Privacy: word forms and glosses from the private grammars appear only as word-list indices and rule ids in these +records. The raw A/B logs (which contain word forms) live only in the session scratchpad and are not to be +committed. The main checkout's `.git/info/exclude` now lists the local grammar backups, `Mbugwe.xml`, the Aweti +word list, and the rulestats HTML report so they cannot be staged by accident. + +## What to do next, in order + +1. Fix and merge PR #491 (duplicate parses, memo splitting while disabled, entry-counting guard). Draft comment in the + re-review record. +2. Classify the unclassified affixes in the FLEx projects (Mbugwe 1, Amharic 1, Sena 6); that turns #491's default + prune on. Add the finding to the grammar advisor. +3. ~~Open the prefilter PR from `perf/hc-edge-prefilter`~~ — that branch and its own archive tag are both + deleted; a PR now starts from the mechanism preserved on branch + `perf/hc-optimization-archive` rather than resuming the deleted branch. +4. Port the prune to PanGloss per the handoff (`HANDOFF-pangloss-final-template-prune.md`, session scratchpad). diff --git a/docs/hermitcrab-probe-design.md b/docs/hermitcrab-probe-design.md new file mode 100644 index 000000000..eb6275d85 --- /dev/null +++ b/docs/hermitcrab-probe-design.md @@ -0,0 +1,238 @@ +# HermitCrab probe design — enough to rebuild it + +Companion to `hermitcrab-optimization-ledger.md`. That document records *what was tried and why it +failed*. This one records *how it was measured*, in enough detail to reconstruct the instrumentation +and the keys from scratch. + +**Why this exists:** the seven probe branches that produced the ledger's numbers were deleted after +the round closed. Research scaffolding in two product assemblies is not worth maintaining — the +probes gate on `volatile bool` reads inside `Matcher.cs`, a hot inner loop in `SIL.Machine` used by +every consumer. So the code is gone and this is the design record. Anything here can be rebuilt in a +day; the expensive part was learning what to measure and which measurements lie. + +--- + +## 1. Instrumentation architecture + +Two probe classes, in two assemblies, with **two independent gates**. + +`SynthesisProbe` in `SIL.Machine.Morphology.HermitCrab`, and `NodeCostProbe` in `SIL.Machine`. Each +has its own `internal static volatile bool Enabled`, default `false`. **Both must be set.** Setting +only the HermitCrab one makes every lower-layer bucket read exactly `0.00ms`, which looks like +"measured and negligible" rather than "never ran" — this cost a round (ledger method rule: a bucket +reporting exactly zero is broken until proven otherwise). Wire both from one place in the harness. + +Everything is insertion-only: a `RecordDie(...)` or `Add*Ticks(...)` call placed immediately before +an existing `return` / `continue`, never a change to control flow. Where a site read +`else if (IsTracing) { trace }`, it becomes `else { if (IsTracing) { trace } Record(...) }` — +behaviour-preserving. Counters are `Interlocked`; timers are `Stopwatch.GetTimestamp()` deltas +accumulated with `Interlocked.Add`, with `Reset*` methods so a harness reads per-word deltas. + +### The wall-time split + +**Eight mutually exclusive top-level slices that sum to wall**, measured around +`Morpher.ParseWord`: + +`lookup` (brackets `LexicalLookup`, materialised with `.ToList()` so the bracket covers only the +lookup itself) · `synCascade` · `synBattery` · `synForward` · `synExpand` (brackets +`Word.ExpandAlternatives()` in `SynthesizeSequential`) · `anTotal` (brackets the whole +`_analysisRule.Apply(input)` call) · `unaccounted` (defined as the remainder, never tracked). + +`synForward` is recorded **net of** `synCascade` and `synBattery`: capture both counters before the +`_synthesisRule.Apply` call and subtract their deltas after. This matters enormously and caused a +25x published error — the work a synthesis fold memo would share runs *inside* the cascade and +battery brackets, so the share to divide by is `synCascade + synBattery + synForward`, not +`synForward` alone. Template slot rules compile to the same `SynthesisAffixProcessRule` classes via +`RuleBatch` (`SynthesisAffixTemplateRule` ctor), which is why they land in `synBattery`. + +Nested *inside* `anTotal`, as a breakdown rather than additional slices: `anCascade` (the +`_mrulesRule.Apply` call in `AnalysisStratumRule.ApplyMorphologicalRules`), `anBattery` +(`ApplyTemplateBattery`), `anPhono` (`_prulesRule.Apply`). + +**Always print the remainder as its own column.** It caught the misattributed split, the dead probe +gate, and would have caught four of five retractions at the point of measurement. + +### The per-node decomposition + +Inside `anCascade` and `anBattery`, sub-buckets: `match` · `feature` · `clone` · `key` · `memo` · +`trail` · `equality` · explicit remainder. + +`match` and `feature` come from the `SIL.Machine` layer. To avoid double counting, the HermitCrab +side reads `NodeCostProbe.MatcherTicks` / `FeatureTicks` **before and after** the cascade call and +records the delta — so the lower-layer buckets are slices *inside* the HermitCrab bucket, not +alongside it. Matcher self-time has nested work subtracted at the source (inside `Matcher.cs`); +feature work needs a re-entrancy depth guard or recursive `Unify` calls double-count. + +Measured outcome, so a rebuild can sanity-check itself: on Amharic `anCascade`, clone **46.7%**, +remainder **37.4%**, match 12.5%, feature 3.4%, key **0.0%**, and memo / trail / equality each +**<0.2%**. + +### Censuses + +Rule attempts, split by which guard rejected them (`guardSelector`, +`guardApplicationCount`, `guardUnifiability`, `reachedMatching`); allomorph pattern-match +attempts vs matches; clone created vs discarded vs kept. + +Die points, as an enum with a counter array **and a parallel tick array** — timing each individual +guard check, not the enclosing `Apply()`. Count and cost diverge: `RuleNotApplicableOrPatternMismatch` +was **100.0% of events** and about **0.2% of time**, because each rejection is one list index plus one +reference compare (~29 ns). Never report a die-point histogram by count alone. + +Die points, classified for the abstract-replay question — **shape-free**: synthesis-side +`MaxApplicationCount`, syntactic-feature unification, MPR features, realizational +subsumption/blocking, co-occurrence and obligatory features in `IsWordValid`. **Shape-dependent**: +allomorph pattern match / rule-not-applicable, allomorph environments, final `IsMatch`. +`LexicalLookupMiss` is neither — it precedes any synthesis step. + +--- + +## 2. The keys + +### `AnalysisStateKey` (pre-existing) + +`(Shape, Stratum, SyntacticFeatureStruct, RealizationalFeatureStruct, NonHeadCount, per-rule +un-application count multiset)`. Order-independent by design — no analysis-side rule reads trail +order, which is the redundancy it collapses. The multiset is XOR-combined, not `*31`-rolled, so it +hashes commutatively. + +**Freeze-on-read in the constructor is mandatory.** `AnalysisAffixTemplateRule.Apply` reassigns +`SyntacticFeatureStruct` to a fresh unfrozen clone *after* the owning `Word` is frozen, and that +setter has no `CheckFrozen()` guard. + +### The narrowed variant (row 4) + +Filter the count multiset to rules that are **not** provably shrinking. Two requirements learned the +hard way: build the retain-map **once per grammar**, not per key (key construction is on the hottest +path); and filter during hashing and equality rather than by materialising a filtered dictionary, so +the narrow key allocates no more than the full one. **Absent from the map means retain** — a rule the +classifier walk does not cover must be counted, never silently dropped. + +A `retainedCount` field is needed so equality can compare sizes without enumerating. + +### `SynthesisStateKey` (row 5) — and the trap + +Everything a synthesis step reads: shape+annotations, syntactic FS, realizational FS, MPR set, root +allomorph, disjunctive allomorph indices, applied-rule counts, `IsPartial`, +`IsLastAppliedRuleFinal`, stratum, **and the ordered remaining trail** — +`Word.MorphologicalRuleTrail` sliced to `[0, PendingTrailPosition]`. + +**The trail content is the whole game.** A key with `PendingTrailPosition` (an integer index) and no +trail content is fine for *measuring* and catastrophic for *skipping work*: two candidates at the +same index with different pending sequences compare equal, and merging them discards one +derivation's continuation. Every large apparent redundancy in this engine was measured with such a +key, and each collapsed when the content was added — 9,774x → 15–40%, 61-of-395,026 → 351,414-of-395,026, +3.22x/8.10x → `hits = 0`. + +**`Word.ValueEquals` is not a starting point.** It omits `_syntacticFS`, MPR features and the +disjunctive allomorph indices, all of which synthesis reads. Inheriting from it produces a silently +inflated ratio. + +Two more properties a real memo needs: stored results are **sets**, not values (realizational rules +are trail-exempt — `SynthesisRealizationalAffixProcessRule` has no `IsMorphologicalRuleApplicable` +gate — and several allomorphs of one rule can legitimately match one input before the disjunctive +break); and stored outputs embed the producing candidate's trail, so replay must **re-anchor**, the +mirror of `Word.ReplayOnto`. + +### `RuleLengthClassifier` (row 4's static half) + +Verdicts: `Shrinking` / `NonShrinking` / `Unknown`, where `Unknown` is treated identically to +`NonShrinking` by the key and kept distinct only so a census can tell "this grammar has zero +morphemes" from "this grammar has constructs we cannot analyse". + +An `AffixProcessAllomorph` shrinks iff **every** allomorph (a) inserts at least one *segment* +(`InsertSegments` counting only `HCFeatureSystem.Segment` nodes — a boundary-only insertion is +conservatively NonShrinking, since `InsertSegments.GenerateAnalysisLhs` omits boundaries from the +analysis pattern; `InsertSimpleContext` counts as 1), (b) has **every Lhs part captured** by a Rhs +`CopyFromInput` or `ModifyFromInput`, and (c) copies no part twice. + +The direction trap, which is easy to invert: on the analysis side rules are *un*-applied, so a rule +that inserts on synthesis **removes** on analysis, and a rule that truncates on synthesis +**untruncates** — an Lhs part with no Rhs copy makes the un-applied word **grow**. The authority is +`AnalysisMorphologicalTransform.GenerateShape`, which walks the Lhs parts and either copies the +captured span or calls `Untruncate` to regenerate the part from its pattern. Reduplication (a part +copied twice) and `CompoundingRule` are `Unknown`; conservatism only costs pruning opportunity. + +Measured census, for a rebuild to check against: Sena 19 of 27 rules shrinking, Indonesian 10 of 15, +Amharic 31 of 36. + +--- + +## 3. Harness design + +`[TestFixture]`, `[Explicit]`, environment-driven, writing only `TestContext` lines and no files — +this repo never commits real grammars, and derived corpus data must not leak into a committed path. +**Never print rule, stratum, morpheme or lexical-entry names, or word lists.** Counts, ratios and +timings only. Conformance fixture IDs are committed and safe. + +Env: `HC_MEMO_GRAMMAR`, `HC_MEMO_WORDS`, `HC_MEMO_MAX_WORDS`, `HC_PROBE_WORDS` (explicit +comma-separated list), `HC_PROBE_FIXTURES_ROOT`, `HC_MEMO_TIMEOUT_MS`. + +Breadth comes from `SIL.Machine.Morphology.HermitCrab.Conformance.Fixture.DiscoverAll(root)` over +`conformance/{languages,edge-cases}` — 33 grammars, each `grammar.xml` + `words.yaml` with +hand-derived expectations. Depth comes from the three real corpora. **Amharic: never exceed the first +28 words** — zero-based index 28 is pathological and burns hours. + +Run the memoized path: `new Morpher(traceManager, language, maxDegreeOfParallelism: 1)`. Tracing +disables the memo, so a traced run measures nothing. + +### A/B methodology, all of it earned by being wrong first + +Parity is the gate and speed is only reported. Acceptance is **analysis-set equality on canonical +morpheme-signature sets** (`MorpherTests.WordAnalysisSignature`, sorted) — never object or byte +equality, because a replayed `Word` is legitimately not field-for-field identical. + +Timing: **discard a warm-up rep per arm, then take min of N interleaved samples, and print the +off-arm spread beside every speedup.** Refuse any speedup smaller than its own spread. Three +failures forced each part of this — a first A/B read 1.53x that was entirely JIT; another read +2.385x with a 210.9% spread on one arm that was 1.015x serial; and adding one arm's warm-up while +discarding the other's manufactured a uniform ~2.4x across every fixture including ones with zero +shareable steps. + +**Never dispatch timing-sensitive probes concurrently.** Three at once put four testhosts on the +machine and invalidated three rows' bucket percentages. Check `tasklist | grep -c testhost` before +any timing run and report the count alongside the numbers. Counts survive contention; shares and +speedups do not. + +**Never aggregate the test logger's console output.** It prints every line twice, the second copy +space-prefixed, so a `^\s*` grep doubles every total — this published a 2x error. + +Sub-50 ms fixtures cannot carry a timing claim: one 10 ms fixture's share moved 1.3% → 14.5% +between two runs of identical code. Flag them (`reliable = wallMs >= 50`) rather than filtering them +out silently. + +### One concurrency hazard worth knowing + +A per-word timeout wrapper built on `Task` has no cooperative cancellation, so an abandoned word's +task keeps mutating engine state after the harness moves on. Harmless with a per-word +`AnalysisScope`; under a **shared** scope it collided with the next word on the same +`Memo`/`InProgress` collections and produced `IndexOutOfRangeException` inside +`MemoizedCombinationRuleCascade`. A shared-arm timeout must abort the run, not continue. Also note +`Task.Result` throws `AggregateException`, not the bare exception type. + +### Final reusable harness + +The retained `PerfBench` is narrower than the historical probes above. It accepts `HC_BENCH_GRAMMAR`, +`HC_BENCH_WORD_LIST`, `HC_BENCH_REPS`, `HC_BENCH_OUT`, and `HC_BENCH_LABEL`; it records timings, errors, +resource observations, memo diagnostics, and canonical result signatures. It does not retain the completed +round's hot-loop counters. + +Timeouts are now the caller's responsibility: start one `dotnet test` process for an arm and terminate the +whole process if its budget expires. Do not restore an in-process `Task` timeout—the abandoned parse continues +running, contaminates later samples, and can race shared engine state. `scripts/hc-bench-compare.ps1` compares +grammar and word keys symmetrically before comparing errors, signatures, timing, or spread. + +--- + +## 4. Two durable engine facts, restated here so they survive independently + +**`Word.ReplayOnto` does not splice `_mrulesUnapplied`.** It splices `_mruleApps` and `_nonHeadApps` +only. This is safe *solely* because `AnalysisStateKey` includes the count multiset, which guarantees +a memo hit has identical arrival and stored-arrival counts. **Narrowing that key breaks correctness +with no test failure.** Fix, if the key ever changes: store arrival counts on `MemoEntry` and compute +`stored - storedArrival + query` — a no-op under today's key, which is also how to test it. + +**`AnalysisScope.MaxMemoEntries` is a per-scope cap with no eviction.** Sharing one scope across a +corpus therefore makes words compete for a fixed budget: two heavy Sena words exhausted it at index +74 and 76, and every later word lost the within-word memoization a fresh scope would have given it. +Cross-word hits went *negative* and the corpus ran 28–32% slower. A shared bounded cache without +eviction is not a smaller version of the optimization. diff --git a/docs/hermitcrab-search-asymptotics-research.md b/docs/hermitcrab-search-asymptotics-research.md new file mode 100644 index 000000000..6c1874377 --- /dev/null +++ b/docs/hermitcrab-search-asymptotics-research.md @@ -0,0 +1,370 @@ +# HermitCrab analysis search — asymptotic research + +**Disposition: count ceilings before building.** The first two candidates below fit the two open work +items without a search-sized memo. Partial-order reduction has a narrow correctness route and poor prior +coverage. Exact finite-state compilation is a larger, restricted-subgrammar project. None is a measured +optimization, and none establishes a polynomial parser for arbitrary FieldWorks grammars. + +This integrates the six supplied reports: **R1** partial-order reduction and rewriting; **R2** packed forests +and subset DP; **R3** finite-state methods; **R4** search and symbolic sets; **R5** enumeration and graphs; +**R6** abstract interpretation and morphology prior art. The reports are research inputs, not independent +measurements. The [ledger](hermitcrab-optimization-ledger.md), its +[September disposition](hermitcrab-perf-2026-09-disposition.md), and the +[probe design](hermitcrab-probe-design.md) remain the measurement record. + +## The problem in one page + +The motivating case is **a grammar with 16 extra affix processes in one Unordered stratum**. Analysis +starts at the surface, unapplies phonology, then interleaves morphological-rule unapplication with affix +templates. Every candidate can stop for lexical lookup; candidates from intermediate strata also enter +the lookup/synthesis pipeline. A found lexical entry supplies its root allomorphs, each candidate's +alternatives are expanded, and forward synthesis must both pass validity checks and regenerate the surface +form [C1, C4, C5]. + +There are three different sources of growth: + +- **Rule orders.** The combination cascade restarts its rule loop at zero after every successful + unapplication. With `k` enabled, single-use rules, there can be `k!` full orders and + `sum(j=0..k) k!/(k-j)!` ordered prefixes. Multiple application is enabled; ordinary affix/compound + unapplication checks each rule's limit rather than a cascade-wide used-rule mask. C# realizational + unapplication has no such count guard, so a finite count-vector bound needs an eligibility proof for + the rule kinds included [C1, C2]. Allomorph branches and compounding add further choices. The 16 added + processes are not necessarily the total `k`. +- **Optional sites.** Inverse deletion inserts optional nodes into one phonological shape; it does not + immediately allocate every subset shape. Later matching still has consume/skip choices. The Rust root + trie already merges equal active trie positions. General morphological matching also returns captures + and variable bindings, so its branches cannot be merged on position alone [C6, C7]. Optionality can + therefore multiply matcher work even while the shape representation stays compact. +- **Alternatives.** An analysis-state key deliberately omits trail order. The stratum merge keeps + alternatives, and expansion grafts subsequent changes onto their source histories. Forward synthesis + reads the pending rule in order. A shared shape or rule multiset is not a shared synthesis continuation + [C3–C5]. One final analysis does not imply that only one trail needs checking. + +All numbers here are **supplied by the common brief, not reproduced in this job**: + +| Aggregate | Reported value | +| --- | --- | +| Baseline parse time | approximately 15–60 ms per input in both implementations | +| Time after the 16 additions | 17–22 s for one expensive input in Rust | +| Lexical lookups / distinct shapes | 97,575 / 56, from trace mode | +| Distinct canonical states | 10,079; they differ in shape or per-rule counts | +| Template-battery runs / distinct input states | 43,641 / 4,988; approximately 2.9 s | +| Lookup + alternative expansion + synthesis | approximately 6.5 s | +| Alternatives synthesized / distinct final analyses | 367,278 / 1 | +| Compounding bound in the case | `MaxStemCount = 2` | +| Deletion-rule variants | 200–1,600 inputs reach a 1 s per-input timeout | + +These are ceilings to investigate, not interchangeable denominators. Tracing disables the ordinary +analysis merge and C# memoization [C3, C4]; the trace lookup count cannot establish an untraced duplicate +rate. Nor do 56 shapes establish 56 lookup keys across strata. The apparent battery repetition is at most +38,653 calls (88.6%) under the brief's identity, before checking that identity and the cost of avoiding it. +The 367,278-to-one ratio proves neither order invariance nor a synthesis-sharing opportunity. + +## Verified code anchors + +Line numbers below were read directly, not copied from the reports. **C#** means the requested main +checkout, commit `a4b29742b6274a01c7398ca3e800a19fa6d2c9aa`; paths are relative to +`src/SIL.Machine.Morphology.HermitCrab/` unless qualified. The six central files (`AnalysisStratumRule`, +`AnalysisStateKey`, `MemoizedCombinationRuleCascade`, `AnalysisAffixTemplateRule`, `Morpher`, `Word`) are +text-identical in this PR worktree's starting commit `5bbfa018bdbd5228c3f1aa73db951a4f01d167d3`, ignoring +line endings. The C# anchors are a snapshot at `a4b29742`, before #491 (`b240b7b5`) merged; they were not +re-read at current master. **Rust** paths are relative to the supplied checkout's `rust/crates/`, with +branch tip `fa3bfb186852baaaf1ed28fcc0ba4602b2c19e96` at inspection. This identifies inspected files; it is +not a claim that either external checkout was clean. + +| Anchor / fact | C# file:line | Rust file:line | +| --- | --- | --- | +| **C1 — cascade and template interleaving** | `AnalysisStratumRule.cs:24–61,128–137,174–244`; base combination loop: `src/SIL.Machine/Rules/CombinationRuleCascade.cs:25–51` | `pg-rules/src/cascade.rs:219–265`; `pg-rules/src/stratum.rs:811–888`; strata/results: `pg-parse/src/morpher.rs:701–765` | +| **C2 — application/compound guards and feature effects** | `MorphologicalRules/AnalysisAffixProcessRule.cs:39–63`; `MorphologicalRules/AnalysisCompoundingRule.cs:39–77,129–133`; realizational exception: `MorphologicalRules/AnalysisRealizationalAffixProcessRule.cs:40–58` | `pg-rules/src/stratum.rs:707–724,792–805` | +| **C3 — existing sharing and actual keys** | `AnalysisStateKey.cs:31–36,60–85,94–104`; merge: `AnalysisStratumRule.cs:130–167`; inner memo: `MemoizedCombinationRuleCascade.cs:39–94`; battery memo: `AnalysisStratumRule.cs:196–218` | key fields: `pg-rules/src/analysis_state_key.rs:22–47`; key construction: `pg-rules/src/stratum.rs:664–694`; ordered output key: `pg-rules/src/word.rs:309–321,410–433`; stratum fold: `pg-rules/src/stratum.rs:1180–1239` | +| **C4 — lexical lookup, expansion, acceptance, fallback** | `Morpher.cs:167–201,346–367,417–445,634–665`; expansion: `Word.cs:445–486` | `pg-parse/src/morpher.rs:459–499,765–826,933–997`; expansion: `pg-rules/src/word.rs:487–548` | +| **C5 — ordered trails are observable** | `Word.cs:202–215,253–259,339–347,371–382,627–646` | `pg-rules/src/word.rs:318–319`; pending-rule test and consumption: `pg-rules/src/stratum.rs:1546–1571,1606–1609` | +| **C6 — optional nodes and root-trie sharing** | `PhonologicalRules/NarrowAnalysisRewriteRuleSpec.cs:40–59`; optional insertion argument: `src/SIL.Machine/Annotations/Shape.cs:160–174` | deletion sites/insertion: `pg-rules/src/rewrite.rs:2536–2581`; root trie consume/skip/dedup: `pg-parse/src/root_trie.rs:227–277` | +| **C7 — general matcher work carries captures** | `MorphologicalRules/AnalysisAffixProcessRule.cs:24–33`; capture-dependent reconstruction: `MorphologicalRules/AnalysisMorphologicalTransform.cs:61–100`; optional forks/registers/bindings: `src/SIL.Machine/FiniteState/TraversalMethodBase.cs:248–308` | template slot results and their semantic deduplication: `pg-rules/src/stratum.rs:1057–1075` | +| **C8 — template admission and slot semantics** | `AnalysisAffixTemplateRule.cs:18–73`; non-disjunctive union: `src/SIL.Machine/Rules/RuleBatch.cs:52–61`, configured at `AnalysisStratumRule.cs:27–30` | `pg-rules/src/stratum.rs:894–962,990–1075` | +| **C9 — proposer is followed by exact-engine confirmation** | exact confirmation target: `Morpher.cs:346–367` | current emitter: `pg-foma/src/emit.rs:2615–2675,2704–2745,2768–2808`; confirmation filters/reparse: `pg-foma-runtime/src/confirm.rs:498–517` | + +In particular, at `a4b29742` the C# key contains shape, stratum, syntactic and realizational features, +non-head count, and exact per-rule unapplication counts. It contains neither final-template state nor a +separate morphology history; #491 later added final-template state to the key, so current master lacks only +the morphology history. Rust adds both, saturates counts at each rule's maximum, and clears its analysis +final-template state at the stratum fold (`pg-rules/src/stratum.rs:1196–1200`). The same key name does not +imply the same equivalence relation [C3]. Neither key is a synthesis key. This inspection verifies these +fields and call sites; it does not certify a new key as complete for every continuation. + +There is a second relevant implementation difference: Rust's common dispatch applies the count gate and +pushes a pending-rule entry for every morphological rule kind. C# realizational unapplication has no count +gate and records counts without adding a pending-trail entry [C2, C5]. These are verified bookkeeping +differences, not a demonstrated output divergence here. Any proof using bounded counts or trail identity +must address the C# semantics explicitly; copying Rust's control vector is insufficient. + +## Ranked candidates + +Rank is the order in which to spend research effort under the memory constraint, not a promised speedup. +The reference labels point to the bibliography below; **every literature citation is from memory and was +not checked against a publication or online**. + +| Rank / technique | Home field + reference | Mapping in C# / Rust | Exact-parity condition; check and fallback | Expected asymptotic change | Cheapest ceiling experiment | Difference from closed rows | +| --- | --- | --- | --- | --- | --- | --- | +| **1. Grammar-compiled residual viability**; includes lexical goal propagation, abstract interpretation, and residual lexical coaccessibility | Program analysis / demand-driven databases / automata; [L1, L2, L5] | Before recursive expansion in C1/C2, with summaries of templates C8, optional nodes C6, and lexical targets C4; Rust `combination_rec` and `apply_one_mrule` | **Static:** sound transfer summaries for all remaining operations and reachable strata/roots, including compounds. **Runtime:** prune only a proved-empty abstract continuation. Unsupported operation, budget exhaustion, guessing/overlay not covered: unknown, ordinary search | Fewer explored prefixes; a certified precedence restriction can change `k!` orders to `L` viable linear extensions. No universal worst-case improvement | Shadow checks at C1/C2: count provably dead branches and descendant rule/template/lookup work, unpooled; count summary size and query work | Rows 10/11 reopen only with residual transformations **and compounding** covered; root-reachable synthesis failures remain. This is neither row 8's bare-root length bound nor row 19's feature-only synthesis replay | +| **2. Compile the template battery into a shared-prefix control trie** | Grammar factoring / parsing / finite-state compilation; [L3, L5]. A specialization of the reports' compiled-control ideas, not a reported measurement | C8: replace repeated reverse slot walks inside C1's battery. Rust `run_template_batch`, `template_unapply_slots`, `apply_slot_batch` | **Static:** share only identical slot-batch operations and optionality, with per-template admission and terminal labels retained. **Runtime:** identical input to a shared prefix; preserve all branch outcomes and existing dedup semantics. Trace, unsupported effects, or control-size budget: original battery | For `t` templates sharing prefix work `W_p`: `t·W_p` becomes `W_p`, plus divergent branches and terminal bookkeeping. For a deterministic `p`-slot prefix, this is `t·p` to `p`. No reduction of unrelated rule permutations | Compile-only census of distinct reversed slot prefixes; at C8 count attempts/outcomes at common prefixes, admitted templates, skip-only terminals, clones, and maximum DFS depth. Separate per-battery savings from avoiding battery calls | No `(Word,state)` result cache: sharing is in the grammar's control trie. Does not repeat rows 1/2/18 or the alternatives memo/hoist. Deferred cloning and the final-template filter already have separate records | +| **3. Trace-class POR with complete ordering recovery**; critical-pair analysis supplies certificates | Concurrency / term rewriting; [L4] | C1 unordered recursion; class readout at C4/C5. Same boundaries in Rust `combination_rec` and the expansion/synthesis consumer | **Static:** strong independence with enabledness and branch/effect correspondence, not just equal final shapes. **Runtime:** stack-local sleep sets; retain every class's ordered histories and all stopping/template points. Uncertified pair is dependent; unhandled rule kind uses ordinary search | Ordered-prefix exploration becomes trace-class exploration; fully independent single-use, single-outcome rules give `2^k` classes and `O(k·2^k)` transition attempts. Recovery/synthesis can still require `k!` trails | Count certified pairs **where enabled**, classes, and prefix/template work avoided; separately count recovered trails and synthesis. Earlier eligibility was only 0.00–0.05% of attempts in the supplied research context | Recovers the orders discarded by the rejected commuting-order canonicalization. Skipping class members without a full synthesis/output-invariance proof repeats row 6; merging synthesis steps repeats row 5 | +| **4. Exact finite-state compilation of a verified subgrammar** | Computational morphology / transducers; [L5] | Compile C1/C2/C6/C8 and the lexicon/forward checks C4 into a labelled relation; Rust emitter C9 is a starting point for coverage inspection, not an exact replacement | **Static:** exact regular encoding of features, annotations, root/non-head identity, all ordered outputs, copying and rule limits; unsupported construct or compile-size overflow rejects eligibility. **Runtime:** budget checked before allocating input-product storage; ordinary parser on refusal | Control alone: `prod(i)(m_i+1)` states, at most `k` outgoing choices per state; for single use `2^k` states / `O(k·2^k)` arcs. Full relation and output enumeration may still be exponential/factorial | Coverage audit and compiled states/arcs first; then accepted labelled paths, confirmation work actually removed, distinct complete analyses, and bytes. Compare against 367,278 trails, not just one output | Shares a **grammar-compiled** relation without erasing order or adding a per-search chart. A permissive proposer that still reparses has not removed the cascade | + +## 1. Residual viability: prune only a proved-empty continuation + +Let `S` be a concrete HermitCrab internal `Word` plus cascade/stratum control. Let `Reach(S)` include +every possible stopping candidate after the remaining inverse operations, including lookup in intermediate +strata and every permitted compound split. Choose a finite abstract domain, an abstraction `alpha`, and +transfers satisfying `alpha(T(S)) <= T#(alpha(S))` for **sets** of results. Compute a conservative closure +of the remaining transfers. If that closure has empty intersection with an over-approximation of all +compatible lexical targets, no concrete continuation can find a root. Removing that branch therefore +removes no successful synthesis and no reported analysis. This follows by induction over each concrete +continuation; it needs no assumption that rule order commutes. + +The target set must include everything ordinary lookup could supply. Compound non-head lookup is already +inside unapplication [C2]; it cannot be ignored or treated as a single final root. Root guessing must +either have its own sound target abstraction or retain the original candidates for the fallback branch +[C4]. Rust supplied-root overlays also affect targets (`pg-parse/src/morpher.rs:459–499`). A finite list of +ordinary root spellings is insufficient in either situation. + +Start with a grammar-compiled coarse domain: mandatory/optional segment-length intervals, safe segment +classes, and only feature constraints whose transfer is proved. Inverse truncation can **grow** a shape: +`MorphologicalRules/AnalysisMorphologicalTransform.cs:54–80,108–132` reconstructs uncaptured input parts. +Optional nodes require both skip and consume possibilities [C6]. An unsupported transform maps to top, +never to failure. Staged +refinement is useful only if every stage remains an over-approximation; an inconclusive expensive stage +returns unknown. This is the safe part of R6's abstract interpretation, R4's magic sets, R3's lazy product, +and R5's extension-oracle proposals. They are one pruning family, not four independent wins. + +R4's precedence idea is a special case with a stricter obligation. To install `r before s`, prove that +**every successful complete path containing both occurrences** respects that order, including paths with +intervening enabling rules/templates. Failure of the adjacent two-step sequence `s;r` is insufficient. +Rules are optional: `s` without `r` must remain allowed. A runtime implementation can forbid a later `r` +after `s` only under that complete-path certificate; it cannot require `r` before admitting `s` altogether. +Begin with single-use rules, no certified edges for unknown constructs, and no inference from prefix/suffix +direction alone. A cycle says that the affected combination is impossible, not that individual rules +should be deleted. + +This must remain grammar-sized: use finite transfer tables and stack-local query work, not a table keyed +by every visited `S`. A lazy lexical product that needs an unbounded visited-state set fails the memory +constraint; use a statically bounded abstraction or return unknown. Count dead **subtrees and their work** +without double-counting descendants. The ledger's 23.5% pooled dead-step result and zero on one expensive +input are reasons to stop if the stronger oracle still mostly accepts. No lexical oracle can reject a +root-reachable path merely because later synthesis will fail. + +## 2. Make the template battery cheaper by factoring its control + +The code provides a useful, smaller target than a whole-search chart. C# builds a non-disjunctive battery +and non-disjunctive batches for individual slots. Each template walks slots from last to first; an optional +slot permits the unchanged input to continue, while a required slot stops that skip path [C8]. Templates +are therefore finite control programs over **set-valued** morphological transformations. + +Compile a trie of those reversed control programs. Share a prefix only when its slot operation has the +same ordered rule identities, application role, optionality, and relevant compiled settings. Keep terminal +template identities and their entry predicates. At each battery invocation, evaluate each template's +selector/required-feature predicate on the **original input**, as today; a shared edge is eligible only +for the admitted descendant templates. Preserve per-template finality policy in Rust. Do not move entry +predicates to a transformed intermediate input [C8]. + +For an admitted template, the original slot path has an identical path through the trie. Conversely, +every trie terminal has an admitted original template. At a shared edge the inputs and operations are +identical, so applying the operation once yields exactly the result set each original execution would +have produced. Induction over prefix length preserves each template's ordered rule trail and complete +internal `Word` effects. Branches split at differing slots; required slots retain no skip edge, optional +slots retain both routes, and all-skipped terminals remain. The union of terminal results is consequently +the original battery result set. This is control factoring, with no identification of two independently +arrived search states and no commutation assumption. + +Two implementation boundaries matter. First, retain the existing slot/battery deduplication and representative +semantics: C# uses `Word.ValueEquals`, which omits syntactic features; Rust explicitly generalizes syntactic +features on collisions [C5, C8]. A new global semantic-key merge is not justified by the trie proof. The +first eligible fragment should exclude differing duplicate-result representatives unless their equivalence +is certified. Second, keep template-specific effects at terminals; if an effect changes a subsequent slot +operation, it belongs in the prefix identity or prevents sharing. Tracing follows the ordinary battery, +because template-specific trace events observe separate executions. + +The new trie has at most one node per original slot-prefix position: `O(B)` grammar storage for `B` total +slots, plus rule lists and terminal membership. Evaluate it depth-first with only the current internal +`Word`, branch iterator, and admitted-template information on the stack. No completed `(trie node, Word)` +results are retained for later arrivals. Live traversal storage is bounded by grammar depth and the current +rule-result enumeration, not total states visited. Existing result accumulation is still required. If +implementation instead introduces a growing active-state chart, it has left this proposal. + +This can reduce the factor due to duplicated template control. It does **not** turn the 43,641 battery calls +into 4,988 calls; that would need different search structure or the ruled-out memo. No supplied report +measures common slot prefixes, so a compile-only prefix census is the first gate. This specialization +combines R2's structural sharing and the reports' finite-state compilation ideas at a grammar-sized +boundary. It is a derived proposal, not an experiment any report already ran. + +## 3. Partial-order reduction: preserve the trace class, then recover its orders + +R1 supplies the essential correction to canonicalization. A class contains paths related by swapping +adjacent independent **occurrences**, not by sorting a multiset and deleting its other orders. With repeated +applications, occurrence identity includes its count/position; nondeterministic allomorph outcomes must +also have a proved correspondence. Templates, compounding, and unsupported operations initially act as +dependent barriers. + +For every certified pair and reachable eligible state, require (1) either transition preserves the other's +enabledness and limits; (2) both two-step result sets have a branch-by-branch bijection; (3) matched endpoints +agree on everything subsequent unapplication, templates and lookup can read; and (4) occurrence effects can +be replayed in each recovered order with all trail, non-head and feature history preserved. Disjoint-looking +string edits, critical-pair joinability, or local confluence alone prove none of these for the complete +internal `Word`. C2 and C7 show why feature effects and captured reconstruction are part of the obligation. +Mark uncertain pairs dependent; a conservative restricted certificate is preferable to runtime speculation. + +Under these conditions, adjacent swaps neither lose a path nor invent an impossible one. Their transitive +closure partitions executions into trace classes. A sleep-set traversal can explore one representative +of each class, **including classes of every stopping prefix**; the dependency ordering of occurrences +retains its other linearizations. Template opportunities at intermediate prefixes must remain represented, +not just the terminal class. Lazy enumeration of each class's linear extensions, with exact occurrence +effects restored, reconstructs the original ordered synthesis inputs. Running ordinary synthesis on each +restores precisely the original analysis set. This is the completeness argument missing from merely +keeping the representative. + +Store the grammar's conservative pair matrix (`O(k^2)`) and current path/sleep/linearization stacks. The +occurrence bound `D` must come from actual grammar limits; a current class can use `O(D^2)` ordering data. +Replace that class's explicit histories or stream them to the existing consumer; do not add a search-wide +class registry or DPOR backtracking graph. If the certificate or stack budget cannot be satisfied, use +the ordinary traversal. A streaming consumer is an architectural change to C4, not a drop-in set key. + +For fully independent, single-use rules with one outcome per occurrence there is one class per subset, +hence `2^k` classes rather than factorially many ordered prefixes. Allomorph branching can add further +classes. **That bound is on analysis traversal, not end-to-end parsing.** If +forward synthesis or output identity distinguishes class members, all relevant orders still need exact +validation/readout. Skipping their synthesis needs a separate proof of identical complete acceptance and +reported analysis, for every root and allomorph. C5 and ledger rows 5/6 rule out assuming it globally. +R6's stronger output-invariance condition is right for **discarding** orders; R1's recovery construction +is right for avoiding repeated pre-synthesis work while preserving them. The earlier 0.00–0.05% eligibility +is a warning, not evidence that better certificates exist. + +## 4. Finite-state compilation: exact only for a certified fragment + +An exact compiler would encode a relation from surface input to complete accepted analysis labels. It +must represent the inverse/forward rule behavior, lexical alternatives, template paths, stratum control, +features, annotations and non-heads that affect validity, with every observable ordered identity retained. +Compile bounded rule-use counts as a control vector: there are `prod(i)(m_i+1)` vectors and at most `k` +choices out of one vector. The vector alone describes permission to try rules, not their semantic outputs. + +If each primitive's encoding is exact, relational union preserves all branches and composition preserves +all legal sequences. Intersecting with the input and lexical/validity constraints then preserves exactly +the accepting paths. This gives a structural induction proof for the compiled fragment. Output labels +must carry sufficient allomorph/morpheme order, root position, non-head provenance and feature information +to reconstruct the same reported analyses. If compilation is only an over-approximation, it may reject +proved-impossible branches as in candidate 1, but its surviving paths still require ordinary confirmation. + +The compiler must refuse constructs with no established regular encoding. In particular arbitrary copying +of an unbounded captured string is not an ordinary finite-state relation; finite alphabets alone do not +fix that. Bounded copying, finite feature domains and bounded compound/rule counts need explicit encodings, +not assumptions. Unknown annotation or constraint behavior also rejects exact eligibility. Compile-time +state/arc/byte limits bound grammar storage, with fallback if composition expands too far. + +For an accepted finite relation with `Q` states and `E` arcs and an input of length `n`, recognition can +use a grammar/input-bounded product, rather than materializing each complete trail. Epsilon behavior must +be handled exactly, and productive zero-input cycles must be excluded or proved bounded. An `O(nQ)` +reachability/coaccessibility structure is acceptable only with its byte bound checked before allocation; +otherwise use ordinary parsing. Recognition sharing does not bound distinct labelled outputs or the +cost of eliminating duplicate output identities. Both the compiled relation and its readout can still +be exponential; `2^k` is the **control-state** bound, not the number of analyses. + +R3 is right that a compact proposer plus exact reparse does not solve the confirmation cascade. Its +blanket `O(k^2)` description of the present proposer is too broad: the actual emitter has a subset-mask +path for eligible unordered underlying-token zones, a dedicated-level fallback, and a legacy every-rule +per-level path [C9]. The comment in `pg-foma/src/unordered.rs:44–61` describes the latter, not all current +execution. No claim about the proposer's universal completeness or exact parity is established here. + +## Where the six reports disagreed + +| Issue | Resolution after reading code and the ledger | +| --- | --- | +| **Is the cascade already subset DP?** R2/R4/R5 emphasize a new inner chart; R6 says that would repeat memoization | R2 is right about **Rust**: C1 uses recursive enumeration and C3 merges later. R6 is right about **C#**: the sequential inner cascade and battery already replay state-keyed results. A count vector is not a sufficient semantic key. A capped chart still grows with search and is outside the hard constraint | +| **What is in `AnalysisStateKey`?** The brief/R2 describe more than the older probe-design key | R2 matches Rust's morphology-history/finality fields. Applying that description to C# is wrong; C3 lists the actual different keys. Neither warrants synthesis merging | +| **Every rule is bounded by its application count**, assumed in the brief and count-vector proposals | Correct for Rust's shared gate and C# ordinary affix/compound rules; C# realizational unapplication is an exception and does not extend the pending trail [C2, C5]. The general bound is conditional, not established for every rule kind | +| **Can commuting orders be reduced?** R1 permits recoverable classes; R6 requires invariant output | Both conditions have a different scope. Recovery can preserve every path; discarding an order requires R6's stronger condition. This resolves the ledger's parse-signature objection, but does not establish a synthesis saving | +| **Packed forest/ZDD/product graph is the best lead** in R2/R4/R5, versus R6's memory objection | R5 is right that an exact product needs full synthesis state and ordered continuation/provenance; one final analysis supplies no small-state bound. R6 is right under this job's constraint. Generic product charts are not recommended. A ZDD encodes sets; ordered trails need position/occurrence encoding, whose size none of the reports bounds | +| **Optional deletion is already compact**, emphasized by R3/R6 | Correct for shape construction and Rust trie active states [C6]. It does not establish cheap general morphological matching: C7 needs captures/bindings, and ledger row 23 closes position-only deduplication | +| **The proposer is `O(k^2)`**, R3 | Verified only for the legacy branch. C9's executed mode dispatch and mask loop take precedence over the older descriptive comment | +| **Simple static precedence could remove impossible orders**, R4 | Conditional only: adjacent-pair failure does not rule out an intervening enabler, and an optional predecessor cannot become mandatory. Candidate 1 states the stronger complete-path certificate | +| **A 64-key root cache bounds walks by `min(U,64)`**, R5; R4 also proposes bounded reuse | The raw search boundary is reusable under fixed grammar/overlay [C4], but the bound is wrong when `U > 64`: uncached keys are searched on every revisit. A byte cap must also cover shapes and result lists. Under the no-memo instruction this is not recommended | + +## Considered and not recommended + +- **Count-vector charts, generic packed analysis forests, ZDD trail tables, analysis-DAG × synthesis-state + products:** search-growing tables; rows 1/4/5/6/16 already test the necessary convergence, and a fixed cap + with restart does not make these a different allowed technique. +- **Packed synthesis readout keyed only by trail position or current shape:** omits ordered continuation + and root-dependent state; rows 5/6/16 close it. `O(Q+T+A)` only helps after proving `Q` and provenance + readout small, which no report does. +- **State/position-only matcher deduplication or two-pass capture reconstruction:** row 23 and the + [two-pass record](rejected-optimizations/two-pass-nondeterministic-traversal.md) already settle these; + the safe group-free fragment is existing prior work, not a new research result. +- **Fixed-cap exact root-result cache:** useful only for raw lookup, still memoization, and neither + 56 shapes nor 64 entries bounds all `(stratum, shape)` result bytes or repeated misses. +- **Root-only lexical gates, edge prefilters, current-length bounds, feature-only abstract replay, and + nogood subsumption:** rows 8–11/14/19/22 already cover them; candidate 1 survives only with its stronger + residual relation and a new measured dead-subtree ceiling. +- **Commuting-order canonicalization without recovery, symmetry/orbit counts:** distinct rule/morpheme + identities and ordered trails remain observable; no output-fixing grammar automorphism was established. +- **Meet-in-the-middle:** large midpoint frontiers violate the memory constraint; unordered rule splits + do not automatically turn factorial work into two square-root searches. +- **A*, k-best/k-shortest paths, reverse-search enumeration alone:** all valid analyses are required; + rearranging or ranking trail enumeration does not eliminate failed trails without a sound extension oracle. +- **Generic Earley/CKY, scalar semirings, equality saturation/e-graphs:** no proved semantics-preserving + span/equivalence boundary for these transforms; adding full transformation state/provenance restores + the unresolved search size. +- **Bimachines, two-level morphology, compile-replace, or parser names alone:** constrained morphology + formalisms supply candidate 4's tools, not a correctness proof for arbitrary HermitCrab behavior. + +## Implications for the two work items + +**Lexicon-driven pruning during unapplication.** Start with candidate 1 in shadow mode at C1/C2. The +deliverable is a per-input census of proved-dead subtrees, saved descendant work, unknown cases and +abstraction cost. Include remaining morphology, template transitions, intermediate-stratum lexical goals, +optional sites and compound non-head roots. Disable pruning where guessing/overlays are uncovered. Close +it if the result merely reproduces row 10's no-op or row 11's weak root-only ceiling. Exact product +coaccessibility is a possible stronger oracle only after a grammar-sized representation exists; building +a whole search graph to ask whether that graph has a successful leaf is not an allowed shortcut. + +**Making the affix-template battery cheaper without memoization.** Start with candidate 2's static +prefix census and admission/slot work counts. Its target is repeated control **within one invocation**. +Candidate 1 can also avoid an invocation whose entire continuation is proved dead; candidate 3 might +avoid repeated prefix work if independence coverage is real. Neither the 43,641/4,988 ratio nor the +historical battery memo authorizes reinstating a cache. Keep deferred cloning, safe edge dispatch and +the final-template policy distinct from this research: their evidence and limits already have records. +In particular, the [final-template record](rejected-optimizations/final-template-prune.md) has an addendum +reopening a narrower guard; its old zero-prune result is not a new lead to rediscover here. + +For either item, compare complete analysis sets with ordered morpheme/allomorph identities, root position, +features and non-head provenance. Do not use the legacy count-degenerate signature called out in the +September disposition. Tests support a proof; a finite corpus cannot prove parity for every export. +Tracing and stateful selectors use the existing path unless their invocation/event semantics are preserved. +Configured alternative/work limits also use the existing path unless the same limit behavior is reproduced +[C1, C4]; a resource-limited optimized query returns unknown and falls back, never a partial exact result. + +## Evidence limits and references + +**Not verified:** the brief's timings/counters, the earlier POR eligibility percentage, exact applicability +of any proposed certificate/compiler to the motivating grammar, common template-prefix coverage, a small +packed product/ZDD bound, and any end-to-end speedup. No real-grammar benchmark or new optimization was +run. Historical measurements are attributed to the supplied brief or the existing records, not fresh +results. Code inspection verifies the listed control/data facts; it does not establish universal parity +of the current Rust port or the current FST proposer. All future experiments need the stronger output +signature and ordinary-parser fallback. + +All entries below are **[memory; bibliographic details and applicability not independently verified]**. +No network or publication lookup was available. They identify home fields, not proofs of these proposals. + +- **L1.** P. Cousot and R. Cousot (1977), “Abstract Interpretation: A Unified Lattice Model for Static + Analysis of Programs by Construction or Approximation of Fixpoints.” +- **L2.** C. Beeri and R. Ramakrishnan (1991), “On the Power of Magic.” +- **L3.** B. Billot and B. Lang (1989), “The Structure of Shared Forests in Ambiguous Parsing.” Structural + sharing background; the grammar-prefix specialization here does not adopt a runtime parse chart. +- **L4.** A. Mazurkiewicz (1987), “Trace Theory”; P. Godefroid (1996), *Partial-Order Methods for the + Verification of Concurrent Systems*; F. Baader and T. Nipkow (1998), *Term Rewriting and All That*. +- **L5.** R. Kaplan and M. Kay (1994), “Regular Models of Phonological Rule Systems”; M. Mohri (1997), + “Finite-State Transducers in Language and Speech Processing”; K. Beesley and L. Karttunen (2003), + *Finite State Morphology*. + +The reports also invoke Held–Karp sequencing DP, Tomita-style shared forests, Minato's ZDDs, graph +coaccessibility, and PC-KIMMO/AMPLE/XAmple. These are useful vocabulary for the considered designs; +none supplies the missing state-equivalence, memory, or arbitrary-HermitCrab encoding proof. diff --git a/docs/optimizations/deferred-template-clone.md b/docs/optimizations/deferred-template-clone.md new file mode 100644 index 000000000..59155adf1 --- /dev/null +++ b/docs/optimizations/deferred-template-clone.md @@ -0,0 +1,35 @@ +# Deferred per-template clone in `AnalysisAffixTemplateRule` + +**Status:** retained on `perf/hc-optimization-archive`, not merged. **Portable:** the observation is; the +saving is allocation, so the Rust port gains little unless it also clones eagerly. + +## The observation + +`AnalysisAffixTemplateRule.Apply` made an internal engine clone and froze it once per template before trying any +slot, then ran the slot DFS on the clone. The ledger measured 66% of those clones discarded unused +(2,505 created, 1,659 discarded, Amharic) and closed the "defer" option because it missed a 70% gate by +four points, choosing "make a clone cheaper" instead. + +Reading the code shows the clone had exactly one load-bearing job. The input is already frozen at every +call site; slot rules never mutate their input (`ApplyRhs` clones `match.Input`); intermediate `Source` +pointers are overwritten by the stratum. The only path that emits the input object itself is the +all-slots-skipped terminal, and `Apply` then mutates every emitted word's `SyntacticFeatureStruct`, so +that one path needs a private copy. + +## The change + +Run the DFS on the frozen input directly (untraced path). At the two emission points that can carry the +original input through unmodified (`ApplySlots` terminal, `ParallelApplySlots` push), emit a frozen +clone when the word is reference-equal to the input. The traced path keeps the eager clone because the +trace manager stamps `CurrentTrace` onto the words it is handed. A naive "no clone anywhere" version was +tried first and failed the new tests immediately (`FeatureStruct.Add` threw on a frozen structure). + +## Measured and verified + +Alternating wall-time A/B against the pinned baseline, 45 Sena words, min of two interleaved rounds: +1.054x overall, heavy words 1.02x to 1.08x; parity clean on all words and both rounds. Smaller than the +ledger's framing suggested: the eager clone was not where the heavy words' time went. + +Focused template tests cover required/optional slots, traced execution, and both sequential and parallel +paths. The complete conformance-inclusive HermitCrab run at the final code tip passed 603 tests with one +expected host-privilege skip. diff --git a/docs/optimizations/edge-segment-prefilter.md b/docs/optimizations/edge-segment-prefilter.md new file mode 100644 index 000000000..e6215d81b --- /dev/null +++ b/docs/optimizations/edge-segment-prefilter.md @@ -0,0 +1,64 @@ +# Edge-segment prefilter for analysis affix rules + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`AnalysisAffixProcessRule`; internal +same-binary toggle `Morpher.EdgePrefilterEnabled`). **Portable:** yes; it is a property of the pattern +semantics, not of the C# engine. + +## The observation + +Every analysis affix pattern is compiled with `AnchoredToStart = true`, `AnchoredToEnd = true`, +`MatchingMethod = Unification`, and a filter that shows the matcher only Segment-type annotations. The +pattern is built from the allomorph's Lhs/Rhs: copied stem parts become groups with quantifiers, and each +inserted segment becomes a single `Constraint` node carrying that segment's feature structure +(`InsertSegments.GenerateAnalysisLhs`). A `Constraint` compiles to one mandatory arc with no epsilon +alternative (`Constraint.GenerateNfa`). + +So when the pattern's first top-level node is a `Constraint`, the traversal's first step must unify it +with the shape's first segment node, or the traversal fails; likewise the last `Constraint` and the last +segment node. The only way an annotation can be skipped is when it is marked `Optional`. + +On the heaviest Sena word, 3,564,736 pattern-match attempts produced 170,863 matches (95% failed), and +each failed attempt still ran a full nondeterministic traversal: 4,017,574 traversal instances, +9,111,459 arc checks. + +## The change + +At compile time, `AnalysisAffixProcessAllomorphRuleSpec` records `LeftEdgeConstraint` / +`RightEdgeConstraint`: the feature structure of the pattern's first / last child when that child is a +bare `Constraint` (null when it is a group, quantifier or alternation). At run time, before each +allomorph's `MultiplePatternRule.Apply`, `AnalysisAffixProcessRule.Apply` finds the shape's first and +last Segment-type nodes (skipping boundaries exactly as the matcher's filter does) and rejects the +allomorph if `edgeNode.FeatureStruct.IsUnifiable(edgeConstraint)` fails, using the same unification call +and `UseDefaults` setting the matcher itself uses. An `Optional` edge node disables the check on that +side. When tracing, the same `MorphologicalRuleNotUnapplied` event is emitted that the matcher's failure +would have produced, so traces are unchanged. + +## Measured (deterministic counts, sequential path, one parse per word, same binary) + +| 45 Sena words | prefilter off | prefilter on | +| --- | --- | --- | +| FST traversals | 12,376,416 | 1,617,662 | +| FST arc checks | 34,170,777 | 15,583,242 | +| traversal instances used | 15,642,175 | 2,429,643 | +| pattern-match attempts (counted before the check) | 11,979,916 | 11,979,916 | + +Parity: identical analysis signature sets on all 45 Sena words and on all 464 conformance-fixture words +(33 grammars). Fixture traversals fell 21% (31,278 to 24,741); their patterns fail less often. + +`AnalysisEdgePrefilterTests` covers left/right constraints, boundary skipping, optional nodes, and the +disabled path. The broader HermitCrab and conformance suites cover integration behavior; the final +conformance-inclusive run passed 603 tests with one expected host-privilege skip. + +## Why the ledger's estimate was wrong + +The ledger (row 22) capped a prefilter at ~10% using the matcher's *self-time* share. The traversal's cost +is mostly booked elsewhere (instance and register allocation, feature unification, GC), so self-time +understated it by a large factor. By counts, the prefilter removes 87% of traversals outright. + +## Limits + +After the change, ~44% of the remaining traversals succeed, so a second-segment check has little left to +reject. The rule loop still runs ~110 attempts per cascade state; depending on which safe edge constraints +exist and whether the first rejects, an attempt performs zero, one, or two prefilter unifications before any +remaining match. Indexing allomorphs by edge segment could avoid both checks and matching work, but its +headroom was not established and it was not attempted. diff --git a/docs/optimizations/feature-struct-frozen-tree-clone.md b/docs/optimizations/feature-struct-frozen-tree-clone.md new file mode 100644 index 000000000..3677c7750 --- /dev/null +++ b/docs/optimizations/feature-struct-frozen-tree-clone.md @@ -0,0 +1,28 @@ +# Reuse or skip the feature-structure clone map + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`FeatureStruct.Clone`). +**Portable:** yes; the specific pool is .NET-oriented, while the tree proof is general. + +## Invariant + +A frozen structure marked as a flat tree has no aliases or cycles to preserve below its root. Its top-level +clone therefore cannot encounter the same value twice, so the old-to-new copies map has no semantic work to +do. Recursive cloning inside another graph must still use the caller's map even when the child itself looks +tree-shaped, because the parent graph may refer to that child elsewhere. + +## Implementation + +A top-level clone skips the map for a proven frozen flat tree. General clones borrow a per-thread map, clear +it in `finally`, and fall back to an independent map for reentrant calls. The internal recursive constructor +always honors the caller-provided map. + +## Evidence + +This removes dictionary creation and lookup from the common clone case. It travelled with the broader clone +allocation work and was not timed independently, so the record deliberately makes no isolated speed claim. + +## Correctness coverage and limits + +`FeatureStructCloneTests` covers alias preservation, cycles, reentrant cloning, and the distinction between a +top-level tree and a tree embedded in a shared graph. The complete conformance-inclusive run passed 603 tests +with one expected host-privilege skip. diff --git a/docs/optimizations/feature-struct-frozen-tree-equality.md b/docs/optimizations/feature-struct-frozen-tree-equality.md new file mode 100644 index 000000000..6f34aa04d --- /dev/null +++ b/docs/optimizations/feature-struct-frozen-tree-equality.md @@ -0,0 +1,29 @@ +# Allocation-free equality for frozen flat feature structures + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`FeatureStruct.ValueEquals`). +**Portable:** yes; the proof depends on immutability and tree shape, not the CLR. + +## Invariant + +During `Freeze`, a feature structure is marked as a flat tree only when every reachable value is a +`SimpleFeatureValue`, no value instance repeats, and no nested `FeatureStruct` is present. Once frozen, that +fact cannot change. Comparing two such structures can never trigger the general algorithm's reentrancy or +cycle guard, so a feature-count/key/value comparison is equivalent to the general visited-pair walk. + +## Implementation + +`ValueEquals` uses the direct leaf comparison only when both operands are frozen and carry the tree proof. +Every other shape—including shared leaves, nested structures, and cycles—uses the general algorithm. A +test-only slow-path entry point lets randomized tests compare both implementations on the same inputs. + +## Evidence + +The optimization removes all visited-set work from the dominant frozen, flat case. It was observed in the +same profile that motivated traversal-state reuse, but was not isolated from the other feature-structure +changes; no standalone wall-time ratio is claimed. + +## Correctness coverage and limits + +`FeatureStructTreeFastPathTests` cross-checks randomized flat structures against the slow path and exercises +false comparisons and structures that must not qualify. `FeatureStructValueEqualsFreezeTests` covers shared +and cyclic graphs. The full conformance-inclusive run passed 603 tests with one expected host skip. diff --git a/docs/optimizations/feature-struct-visited-state-reuse.md b/docs/optimizations/feature-struct-visited-state-reuse.md new file mode 100644 index 000000000..f7c2b63ee --- /dev/null +++ b/docs/optimizations/feature-struct-visited-state-reuse.md @@ -0,0 +1,29 @@ +# Reuse feature-structure traversal state + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`FeatureStruct.ValueEquals` and +`Freeze`). **Portable:** the allocation-avoidance principle is portable; the `[ThreadStatic]` pool is +.NET-specific. + +## Invariant + +Equality and freezing need visited sets only after a recursive value is reached. A top-level +operation owns its traversal state until it returns. A nested top-level call cannot borrow that same state, +so it falls back to fresh collections. + +## Implementation + +The recursive APIs accept nullable state and allocate it only when traversal requires it. Ordinary top-level +calls borrow pre-sized sets and dictionaries from a per-thread pool, clear them in `finally`, and release the +pool. The pool has an `InUse` guard so reentrant calls retain the old independent-state semantics. + +## Evidence + +The heaviest Sena profile put feature-structure equality/freezing at 13.7% of wall time, with `HashSet` +resizing a major allocation source. After this change, `ValueEqualsImpl` and those resizes disappeared from +the top of the profile. This mechanism was not timed in isolation, so no standalone speedup is claimed. + +## Correctness coverage and limits + +`FeatureStructPooledVisitedSetsTests` covers reuse, clearing after success and failure, and reentrancy. Cyclic +and shared graphs continue down the general path. The complete conformance-inclusive HermitCrab run at the +final code tip passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/flat-register-arrays.md b/docs/optimizations/flat-register-arrays.md new file mode 100644 index 000000000..553e7eb99 --- /dev/null +++ b/docs/optimizations/flat-register-arrays.md @@ -0,0 +1,26 @@ +# Flat register arrays in the FST traversal + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`SIL.Machine.FiniteState`: +`Register[]` indexed by `RegisterArray.Idx(r, k)` replaces `Register[,]` internally; the +public `FstResult.Registers` stays two-dimensional and is materialized once per accepting state). +**Portable:** no; a .NET-specific allocation path. + +## The observation + +11.5% of the heaviest Sena word's CPU profile was inside `System.Array.CreateInstanceMDArray`, the slow +generic allocator for multi-dimensional arrays, called from `Matcher.AllMatches` (8.6%) and traversal +setup (2.3%): one `new Register[n, 2]` per match call and per traversal instance. + +## The change + +Mechanical: flat arrays of length `registerCount * 2`, a two-constant index helper, `Array.Copy` on the +flat struct array, and `RegistersEqualityComparer` comparing the flat layout. No behaviour change. + +## Measured + +`CreateInstanceMDArray` 11.5% → 0.2% of the profile. Counts unchanged by construction. Together with the +other allocation changes it contributed to the heaviest word's wall time; not measured in isolation. + +`RegistersEqualityComparerTests` covers equality and hashing of the flattened layout, while the existing FSA +and FST suites cover accepting results and register materialization. The complete conformance-inclusive run +passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/shape-copy-single-pass-map.md b/docs/optimizations/shape-copy-single-pass-map.md new file mode 100644 index 000000000..900515da2 --- /dev/null +++ b/docs/optimizations/shape-copy-single-pass-map.md @@ -0,0 +1,28 @@ +# Build `Shape.CopyTo`'s node map in the copy pass + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`Shape.CopyTo`). +**Portable:** yes, when a port has the same node-remapping requirement. + +## Invariant + +Annotations refer to nodes in the source shape and must be remapped to the corresponding nodes in the copy. +The source and destination nodes are already visited in matching order while cloning, so that loop can record +the mapping directly. No second `Zip`/`ToDictionary` traversal is required. An empty source range needs no +map; every nonempty range builds one because annotations are queried only after nodes have been copied. + +## Implementation + +`Shape.CopyTo` allocates the map on the first node of a nonempty range and fills it as each node is cloned. +Annotation ranges are then copied through that map exactly as before. + +## Evidence + +The change removes one enumeration and one LINQ-built dictionary from each annotated shape copy. It was not +timed independently from shape copy-on-write and feature-structure clone work, so no standalone ratio is +claimed. + +## Correctness coverage and limits + +`ShapeCopyToTests` covers empty ranges, leaf and spanning annotations, node/range remapping, and independence +of the copy. +The complete conformance-inclusive run passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/shape-sharing-on-clone.md b/docs/optimizations/shape-sharing-on-clone.md new file mode 100644 index 000000000..a5ffd563f --- /dev/null +++ b/docs/optimizations/shape-sharing-on-clone.md @@ -0,0 +1,44 @@ +# Copy-on-write Shape sharing in `Word.Clone` + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`Word.CloneForEngine`, `EnsureOwnShape`, +`ResetShape`, `Shape.CreateEmptyLike`). **Portable:** partly. The *observation* (three of four clone sites +never use the copied shape) is engine-independent; the mechanism (shared reference to an immutable +structure) is how a Rust port would naturally write it already. + +## The observation + +A CPU profile of the heaviest Sena word (60 s sampled, memoized path) put `Word` cloning at 36% of wall, +of which `Shape.CopyTo` was 28%: every clone deep-copied every `ShapeNode`, every node's feature +structure (`SimpleFeatureValue.CloneImpl` 22%), every annotation, and built a node-mapping dictionary. +The callers: `Word.ReplayOnto` 15% (memo replay; never mutates the shape), `Word.ExpandAlternatives` 9% +(immediately *replaces* `_shape`), `AnalysisAffixProcessAllomorphRuleSpec.ApplyRhs` 5% (`GenerateShape` +immediately `Clear()`s the copy), non-head clones 2.5%, `LexicalLookup` (the `RootAllomorph` setter +replaces `_shape`). + +## The change + +- Internal engine clone: if the source shape is frozen, share the reference and mark `_shapeShared`; otherwise + deep-copy it. Public `Word.Clone()` always returns a deep, independently mutable shape. +- `EnsureOwnShape()`: clone once, on demand, before in-place mutation. Called in `AnalysisStratumRule` and + `SynthesisStratumRule` before phonological rules are applied to a fresh clone (rewrite specs insert + nodes and hold node references obtained by matching, so the un-share must precede the match). +- `ResetShape()`: replace with a new empty shape of the same configuration. Used by the four + `GenerateShape`/rebuild sites (analysis affix, analysis compounding subrule, synthesis affix, synthesis + compounding) instead of clone-then-clear. +- `ExpandAlternatives` marks the adopted shape as shared; the `RootAllomorph` setter and constructors mark + their fresh clones as owned. + +The safety net is that a shared shape is frozen: any mutation through a missed site throws +(`Shape.CheckFrozen`, frozen node feature structures; a missing guard on `Annotation.Range` was added). +During development the first full test run threw at exactly the two stratum sites, which is how they +were found; no other site surfaced across 122 HermitCrab tests, 464 fixture words and 45 Sena words. + +## Measured + +Profile after the change: `Word` clone 36% → 9% of wall, `Shape.CopyTo` 28.6% → 13% (the remainder is +the stratum's `EnsureOwnShape` and part copying inside `GenerateShape`), `ReplayOnto` 21.7% → 6.5%. Peak +working set on the heaviest word 11 GB → 6 GB. Wall-time attribution is unreliable on this machine (see +the round document); counts are unchanged by construction. + +`WordShapeSharingTests` covers public-clone independence, copy-on-write, reset paths, and missed-mutation +guards. The final conformance-inclusive run passed 603 tests with one expected host-privilege skip. diff --git a/docs/optimizations/syntactic-feature-structure-sharing.md b/docs/optimizations/syntactic-feature-structure-sharing.md new file mode 100644 index 000000000..9f99d4570 --- /dev/null +++ b/docs/optimizations/syntactic-feature-structure-sharing.md @@ -0,0 +1,33 @@ +# Copy-on-write syntactic feature structures in engine clones + +**Status:** retained on `perf/hc-optimization-archive`, not merged (`Word.CloneForEngine`, +`EnsureOwnSyntacticFeatureStruct`). **Portable:** partly; immutable sharing is general, while the ownership +plumbing is specific to this engine. + +## Invariant + +HermitCrab freezes a `Word` before it becomes an input to the analysis or synthesis graph. Internal clones may +therefore share that frozen word's syntactic feature structure until a rule needs to mutate it. The public +`Word.Clone()` contract remains a deep, mutable clone and never exposes copy-on-write sharing to callers. + +## Implementation + +Internal engine sites use `CloneForEngine()`. A per-`Morpher` setting is copied onto new words and controls +whether those clones share a frozen syntactic feature structure. The three in-place analysis mutation sites +call `EnsureOwnSyntacticFeatureStruct()` before modification; assignments of a fresh structure need no guard. +The opt-out is internal and captured per `Morpher`, preventing one parser or test from changing another +existing parser's behavior. + +## Evidence + +The heaviest Sena profile attributed roughly 15% of CPU to freezing and hashing a newly cloned syntactic +feature structure while constructing analysis keys. Sharing removes that repeated deep clone/freeze on the +engine path. The final five-language measurement is branch-wide rather than an isolated toggle result, so no +standalone speedup is assigned here. + +## Correctness coverage and limits + +`WordSyntacticFsSharingTests` covers copy-on-write, nested/shared structures, public-clone mutability, and +isolation between morphers. The broader HermitCrab suite exercises integration behavior and passed both with +sharing enabled and with `HC_SHARE_SYNTACTIC_FS=false`; the full conformance-inclusive run passed 603 tests +with one expected host skip. Sharing is intentionally unavailable for mutable source words. diff --git a/docs/rejected-optimizations/alternatives-memo-and-hoist.md b/docs/rejected-optimizations/alternatives-memo-and-hoist.md new file mode 100644 index 000000000..419cba7ef --- /dev/null +++ b/docs/rejected-optimizations/alternatives-memo-and-hoist.md @@ -0,0 +1,15 @@ +# `ExpandAlternatives` memoization and hoisting + +**Status:** rejected and removed from production. + +The experiment memoized `Source` recursion during `Word.ExpandAlternatives` and computed an analysis word's +expansion once outside the per-root-candidate loop. It was exact, but on the 45-word Sena battery it left word +clones unchanged at 4,747,013: deeper strata did not branch, so it saved only small temporary list work. + +A later change moved work back behind successful lexical lookup, but after the experimental memo/hoist was +removed the final product path is the same ordering as the pinned production baseline: each lexical candidate +expands its synthesis word. There is therefore no retained alternatives optimization in this branch, and the +earlier 75% “alternatives expanded” reduction must not be attributed to the final diff. + +The broader deduplication avenue remains closed by the complete-key census in the main ledger: once pending +trail content is included, only 11% of alternatives duplicate, for a 2.28% wall-time ceiling on Sena. diff --git a/docs/rejected-optimizations/compounding-split-prefilter.md b/docs/rejected-optimizations/compounding-split-prefilter.md new file mode 100644 index 000000000..067fbd07d --- /dev/null +++ b/docs/rejected-optimizations/compounding-split-prefilter.md @@ -0,0 +1,11 @@ +# Compounding split prefilter + +**Status:** rejected before implementation; temporary counters removed. + +The candidate was to query lexical reachability before materializing compounding splits. A census on the +heaviest Sena probe counted 35,176 splits, 5,524 root candidates, and 1,121 outputs. That work represented +about 2% of word clones and 9% of FST traversals—too small a ceiling for an additional indexing and correctness +surface, especially because real compounding prevents the simpler grammar-wide lexical gate. + +No production change was built. Reopen only if a different grammar shows compounding split construction as a +dominant measured bucket and supplies a sound, stratum-aware reachability predicate. diff --git a/docs/rejected-optimizations/final-template-prune.md b/docs/rejected-optimizations/final-template-prune.md new file mode 100644 index 000000000..a6f0002b5 --- /dev/null +++ b/docs/rejected-optimizations/final-template-prune.md @@ -0,0 +1,25 @@ +# Final-template interleaving prune + +**Status:** rejected and removed from production. **Portable:** the proof is portable where its guard holds. + +The experiment mirrored synthesis's rule that a non-template affix process normally cannot follow a final +template. Analysis could skip the inverse interleaving when every relevant template was final and the grammar +contained no partial morphemes. + +The available real grammars defeated the useful guard: Sena has 25 partial lexical entries and six partial +rules; Amharic and Mbugwe each have a partial rule. The prune fired zero times on all three. Synthetic tests +proved that the rule could fire safely, but that establishes correctness, not value. Carrying production +branches, state, toggles, and counters for a zero-effect optimization violated the final minimum-surface goal, +so all of them were removed. + +Reopen only for a corpus with the reported pathological final-template interleavings and no partial +morphemes, or with a proved narrower per-state partiality condition. Measure real prunes before shipping. + +## Addendum 2026-09-03 +Superseded in part. The same prune, measured with the partiality guard lifted, is 8-33x on Mbugwe and Sena with +identical analysis sets (298x on the single heaviest Sena word). The guard, not the mechanism, is what made it +inert: partial *lexical entries* can never rescue a pruned branch, because synthesis applies no template to a +partial root and then fails the candidate with `PartialParse`, so a rules-only, per-stratum guard is strictly +stronger and still sound. Reopen on that basis rather than on the corpus counts above. The full measurement +record is `docs/pr491-final-template-prune-upper-bound.md` on the `perf/hc-optimization-archive` branch, which +is outside this docs set. diff --git a/docs/rejected-optimizations/state-position-traversal-dedup.md b/docs/rejected-optimizations/state-position-traversal-dedup.md new file mode 100644 index 000000000..40a9fb66a --- /dev/null +++ b/docs/rejected-optimizations/state-position-traversal-dedup.md @@ -0,0 +1,95 @@ +# `(State, AnnotationIndex)` traversal deduplication + +**Status:** rejected in its general form; a narrowed form is sound and was measured effective. + +The idea, proposed independently as PR #511: inside `Traverse`, skip any newly produced traversal +instance whose `(State, AnnotationIndex)` pair was already pushed, on the grounds that the surviving +instance differs only in its registers and "registers only record the match, they do not filter the +traversal." + +## The motivating pathology is real + +The cost driver is not the state count. `TraversalMethodBase.Advance` forks a copy of the instance for +every `Optional` annotation at the next offset, so instance count grows exponentially in the number of +optional annotations along the input and is independent of `|States|`. A synthetic microbenchmark on a +**2-state** FSA measured 2,097,150 instances popped at 20 optional annotations (~2^(k+1)); at k>=24 it +exceeds a 5,000,000 budget. The originally reported "200,000 traversals on a 4-state fsa" is reproduced +and exceeded on fewer states. With the skip in place, the same cells are bounded at `N * |States|` +(measured ratio 0.5-1.0x), turning non-terminating cells into instant ones. + +## Why the general form is unsound + +A differential fuzz — 20,000 random pattern/input pairs, fixed seed, identical generator compiled against +both arms, comparing success, range, every group range and every variable binding — found **112 cases +where `Match()` differs from production**, while `AllMatches().First()` was identical in all 20,000. That +control pins the divergence to exactly the path the skip changes. + +- **108 cases: variable bindings.** Instances carry live `VariableBindings`, which filter later arcs + through `CheckInputMatch`; the key omits them. `Matcher.Compile` determinizes only variable-free + patterns, so every alpha-variable rule runs the nondeterministic method where the key is blindest. + Minimal case: pattern `high=$v0+` anchored both ends over 5 annotations, production matches `[0,5)`, + the skip returns no match at all. +- **4 cases: capture registers, on the deterministic path.** All are alternation plus capture groups and + all shorten the match *range*, e.g. `(g0(back=back+)|g1(high=high+back=back+))` going from `[0,4)` to + `[0,1)`. So restricting to the deterministic method is not by itself a sufficient guard. + +Both committed suites stay green under the unnarrowed change (826/0/3 and 98/0/0), so the existing tests +cannot see any of this. + +## The narrowed form that is sound + +Gate the skip on the deterministic method **and** the absence of capture groups, and leave the +nondeterministic method untouched: + +```csharp +bool dedupeByState = !allMatches && Fst.GroupNames.Count() <= 1; +``` + +Measured: fuzz divergences **112 -> 0**; the group-free microbenchmark sweep cell-for-cell identical to +the unnarrowed change across all 18 cells, so the full bound is preserved; grouped patterns fall back to +production behaviour exactly; both suites unchanged. `GroupNames.Count()` should be hoisted out of +`Traverse` — it is O(1) via the `ICollection` fast path but is evaluated once per call. + +## Where it cannot be extended, and why + +Two censuses attributed every FSA traversal call to its constructing matcher and counted, per category, +how many pushes a `(State, AnnotationIndex)` key would have collapsed — and critically, whether the +collapsed instance's registers were identical to the surviving one's. + +*Environment matchers* (the subject of issue #515) read only `.Success` and `.VariableBindings` from their +matches, never a range or group, so they looked like the ideal target. They are 0.20% of traversal +instances on Amharic and 1.18% on Mbugwe. Removing every redundant environment push would be worth 0.12% +and 0.004% of total pushes respectively. Not built. + +*Analysis-side morphological rules* hold nearly all the volume and all the apparent opportunity: +2,801,478 of Amharic's 2,808,088 pushes, 93.95% collapsible; Mbugwe 14.44%. Labelling the remaining +construction sites showed the entire figure is exactly `AnalysisAffixProcessRule` and +`AnalysisCompoundingRule` — both built with `AllSubmatches = true` and calling `AllMatches` precisely to +enumerate every distinct morph-boundary placement. Register comparison at each would-be collapse: + +| matcher | identical registers | differing registers | +| --- | --- | --- | +| Amharic `analysis-affix` | 7,446 (0.29%) | 2,573,312 (99.71%) | +| Amharic `analysis-compound` | 16 (0.03%) | 50,920 (99.97%) | +| Mbugwe `analysis-affix` | 3,022 (5.49%) | 52,056 (94.51%) | +| Mbugwe `analysis-compound` | 1,886 (2.93%) | 62,396 (97.07%) | + +The genuinely free slice is 0.28% of collapses on Amharic and 4.11% on Mbugwe — at most 0.6% of total +traversal instances, reachable only by adding a per-instance register-equality guard. `RootAllomorphTrie`'s +raw FST collapses 0% despite ~62k pushes, which is correct rather than a missed opportunity: a prefix trie +is already maximally state-shared at construction. + +Adding `VariableBindings` to the key costs nothing on this evidence: `collapsedWithBindings == collapsed` +on every row of both grammars, with a distinct-signature counter confirming the signature function +discriminates. The information that differs lives in the registers, which is exactly what `AllSubmatches` +consumers read. + +## Relationship to `two-pass-nondeterministic-traversal.md` + +That experiment merged on state, position **and bindings** and then rebuilt captures, and was measured +negative largely on the rebuild cost. This one merges on state and position only and keeps the first +instance's captures, which is cheaper and unsound rather than expensive and sound. The two are not +alternatives to the same design; they are the two ways of getting the key wrong in opposite directions. + +**Reopen only** with a grammar where the identical-registers split looks materially different from the +four rows above, or for the narrowed group-free form, which needs no further evidence to be safe. diff --git a/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md b/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md new file mode 100644 index 000000000..344a6bd0e --- /dev/null +++ b/docs/rejected-optimizations/two-pass-nondeterministic-traversal.md @@ -0,0 +1,18 @@ +# Two-pass nondeterministic traversal + +**Status:** measured negative and removed from production. + +This port of the `investigate_linear_traverse` approach first merged traversal instances by state, position, +and variable bindings, then rebuilt captures. HermitCrab's existing traversal already pools instances, and +its patterns do not have the repeated-group shape needed to repay the extra bookkeeping. + +On the 45-word Sena battery the alternative used 2.3 times as many traversal instances, reduced arc checks by +only 1.6%, produced more duplicate matches, and increased word clones by 27%. Conformance fixtures pointed in +the same direction. Toggle hooks, alternate algorithms, and counters were removed; only this result remains. + +Reopen only with a corpus whose automata have substantial equivalent active-state convergence and a census +showing that capture reconstruction is cheaper than the current pooled traversal. + +**See also** `state-position-traversal-dedup.md`, which drops bindings from the key instead of keeping them +and skips capture reconstruction entirely. It is cheaper than this experiment and unsound for the same +reason this one was expensive: the captures are load-bearing for `AllSubmatches` callers.