Repository navigation
WorldAlphabets alphabets: tree barely expands despite full training corpus consumption (flat/unweighted canvas) #69
Description
What
All 342 autoconverted alphabet.wa.*.xml alphabets render a nearly flat tree: at rest only 2–3 top-level boxes show, and steering into the tree yields ~2 letter boxes with a 1.1× size spread — letters look unweighted/uniform. Maintained alphabets on the same build show 25+ boxes with 2.4×+ spread.
What it is NOT
- Not the training corpus: instrumenting
CTrainer::Trainshows the full corpus is consumed —English (WorldAlphabets)trains on 1,490,517 symbols (72,326 unmapped) fromtraining_wa_en_Latn.txt(1.5 MB). The LM is learning. - Not the lazy-alphabet work (feat: lazy alphabet loading + generated alphabet metadata index #68): reproduced identically on pre-feat: lazy alphabet loading + generated alphabet metadata index #68
main(eager full parse). - Not the space symbol: WA files do define space (
<node label="␣" text=" ">); the node-leveltextattribute is honored by the loader.
Reproducer (host)
// adaptive probe: steer into the largest box for 5 rounds, dump opcode-4 box heights
dasher_set_alphabet_id(ctx, "English (WorldAlphabets)");
// → letter-boxes=2 h_min=10 h_med=11 h_max=11 spread=1.1x
dasher_set_alphabet_id(ctx, "English with limited punctuation");
// → letter-boxes=25 h_min=10 h_med=16 h_max=24 spread=2.4xProbe source available (adaptive steering into the largest frame box — rules out the pointer just missing the tree).
Suspects
- Group/child accounting in the WA file structure (
paragraphSpace/Editinggroups,iNumChildNodes) producing malformed expansion - Node-probability query path returning uniform values for these alphabets despite the trained LM (symbol numbering mismatch between the alphabet map and tree children?)
- The WA palette (
colorsName="Default") making what does render look uniformly coloured — cosmetic, but compounds the 'no weights' impression
Why it surfaced now
Dasher-Android's locale-follow (PR #31, first version) briefly defaulted en-locale devices onto these alphabets (scoring bug, since fixed) — the first code path to auto-select a WA alphabet. Before that, nothing selected them unless a user did, by hand, from a 474-entry menu.
Impact
Every language that ONLY has a WA alphabet (a large share of the 456 lang codes in the catalogue) effectively gets an unweighted Dasher. That's the next-biggest usability item after the startup work.
Activity
willwade commentedon Aug 28, 2026
Diagnosis narrowed (Aug 28, follow-up session)
New evidence from instrumented probes (all reproducible via small CAPI programs — probes attached to this comment's thread on request):
The LM learns correctly. Dumping the language model through the trainer's own empty context right after
CTrainer::Trainshows a real learned distribution — forEnglish (WorldAlphabets)the maximum sits at symbol 64 (the space character), value 2605/65536, exactly where ~17% of a 1.49M-symbol English corpus should put it.The rendered tree never sees it. The root node's probability vector (
CAlphNode::GetProbInfo→GetProbs) shows symbol 64 at the uniform floor (50) and every symbol in a near-flat band (min 50, max 1775, no space spike anywhere) — i.e. the tree queries symbol indices whose training was recorded under a different index mapping.So this is a symbol-numbering / context mismatch between the training map and the node tree for WA alphabets, not a training or corpus problem. Candidate wiring to inspect:
CAlphabetManager::InitMapvs node-child iteration for the WA group layout (which includes action-only nodes — the Editing group's ⌫/⌦deleteTextActionnodes contribute characters whoseTextis empty, and are counted iniEnd), and root-context creation.Maintained alphabets don't show the flat symptom, so whatever diverges is triggered by the WA file structure (action nodes in groups, group ordering, or the paragraphSpace/Editing layout).
Related but separate: relative data dirs never loaded any alphabet (fixed)
While probing this we found GTK's relative
"Data"data dir broke alphabet loading entirely (index paths were relative;LoadAlphabetFiletreated them as regex patterns; every selection silently fell back to the builtin 26-letter Default → the same "flat unweighted letters" symptom, different cause). Fixed in #68 (0365124): the indexer now stores absolute paths, verified with relative and absolute data dirs. If you saw flat letters on GTK dev builds, that was this; the WA tree bug is what remains for WA alphabets.Impact restated
321 of 323 languages in the catalogue are WA-only — until this is fixed, effectively every non-English language renders unweighted. This is the highest-priority engine bug in the v6 data stack.
fix(data): normalize colorInfoName to palette tokens — WA alphabets r…
willwade commentedon Aug 28, 2026
Root cause found — and fixed data-side (no engine changes). da2e558
My earlier symbol-numbering hypothesis was wrong. Bisecting with identical nodes/corpus/palettes and mutating one attribute at a time pinned it exactly:
The autoconverter emitted free-form
colorInfoNamevalues (lower case,korean trailing consonants, hundreds of pinyin/syllabary group names…), but the shipped palettes define exactly eleven tokens:lowercase, uppercase, numbers, punctuation, accents, space, paragraph, paragraphSpace, limitedPunctuation, punctuationLong, lowercaseBackground. A group whose token no palette resolves gets no colour info, and the renderer draws nothing for its nodes — the probabilities exist but are invisible. (The training, LM and symbol numbering were all fine; that's why the trainer-side dumps looked healthy while the canvas looked dead.)Fix shipped (converter-side, as preferred)
Scripts/fix-wa-colorinfo.pyrewrote all 474 autoConverted files: recognised tokens preserved, everything else keyword-mapped (upper→uppercase, digits/numerals→numbers, punctuation→punctuation, accents/diacritics/combining→accents, editing→punctuation, unknown letter groups→lowercase, the engine's own default).Verified (tree-spread probe, before → after):
Rule for the WorldAlphabets Dasher exporter
Emit only the eleven palette tokens above, or omit
colorInfoNameentirely (the engine then defaults the group tolowercase). Never free-form group names.Remaining (separate)
willwade commentedon Aug 28, 2026
Primary cause fixed data-side in #68 (da2e558): colour-token normalization across all 474 autoConverted alphabets, verified across English/Arabic/Korean. Follow-ups tracked separately: conversion-mode rendering (Mandarin pinyin), and adopting the token rule in the upstream WorldAlphabets exporter.
feat: lazy alphabet loading + generated alphabet metadata index (#68)