Repository navigation
DictionaryPredictor: score a word that starts with the prefix as a completion - #2
Merged
Merged
Conversation
…mpletion Before, the distance was measured from the prefix to the whole word. A long word got a very low score until the user typed almost all of it. Now a word that starts with the prefix gets distance 0. The number of characters still to type only breaks ties. Other words keep the whole-word distance, as typo corrections. The prefix is lowercased before the compare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 tasks done
bartekplus
added a commit
to bartekplus/FluentTyper
that referenced
this pull request
Oct 12, 2026
Rebuild libpresage with two more changes from bartekplus/presage#2: - A user word that does not start with the typed prefix is offered only when it is a typo of the word's start. Before, unrelated user words filled empty slots ("fluent" gave "kubernetes"). - The n-gram model also scores the speller corrections, times the new CORRECTION_WEIGHT (0.000001). On 800 synthetic typos per language, top-1 goes up by 2.1-2.6 points for en, pl and de. KSR does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bartekplus
changed the title
bartekplus
changed the title
Owner
Author
|
Correction: after this PR was merged, I pushed two more commits to its branch and changed this description to include them by mistake. Those commits are not in |
bartekplus
added a commit
to bartekplus/FluentTyper
that referenced
this pull request
Oct 13, 2026
…atabases (#501) * Tune Presage n-gram deltas and show user words from their first letters Set the n-gram DELTAS to 0.01 0.08 0.3 0.6. A sweep with eval_ngram.py on held-out HPLT text gives +0.23 (en), +0.13 (pl) and +0.22 (de) percentage points of KSR. Top-1 does not drop. The deltas now sum to less than 1, so an n-gram score stays below the 1.0 cap of the combiner and below an exact text expansion. Scale the Hunspell, Aspell, user dictionary and recency scores by the same factor as the unigram delta (x0.08). They keep their place among the rarest n-gram words. Rebuild libpresage with the DictionaryPredictor fix from bartekplus/presage#2: a user word that starts with the typed prefix is a completion, so it shows from its first letters. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Rank speller corrections by frequency and drop unrelated user words Rebuild libpresage with two more changes from bartekplus/presage#2: - A user word that does not start with the typed prefix is offered only when it is a typo of the word's start. Before, unrelated user words filled empty slots ("fluent" gave "kubernetes"). - The n-gram model also scores the speller corrections, times the new CORRECTION_WEIGHT (0.000001). On 800 synthetic typos per language, top-1 goes up by 2.1-2.6 points for en, pl and de. KSR does not change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Build n-gram databases with a per-word budget for longer n-grams The old pruning capped each n-gram order at 75% of the order below. For en_US it dropped about 95% of the n-grams seen 10 times or more. Now all words are kept, and the most frequent longer n-grams (orders 2 to 4 together) are kept, up to 4.5 for each word. The databases are 2.00 times the old size (29.8 MB -> 60.0 MB). KSR goes up in all 10 languages, by 0.88 (fr) to 2.99 (hr) points, and top-1 by 0.27 to 1.24 points. Regenerate the Review lexicons from the new databases and fix what the bigger databases exposed: - German gender: contrary cues under 1/20 of a word's cues are noise ("jede gerät", "was das Kunden"). The weak-noun -n filter also covers m/M ("einen Nachbarn"), and nForm no longer cuts two letters from a word that does not end in -en. - German runtime: a feminine noun in -e keeps its plural -n ("anhand ihrer Unterlagen"). - German supplement: author "altgold", which the new counts do not hold. - Spanish: author "física" as a noun. The generator also probes "c" + every verb for Bloom false positives; "cterminar" was one. - e2e: "Where wa it?" now ranks "water" second after "was". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
DictionaryPredictormeasured the Levenshtein distance from the prefix to the whole word. A long user word got a very low score until it was almost fully typed. For example, "flu" → "fluenttyper" has distance 8, so the score isPROBABILITY / e^8. In FluentTyper, the word showed only after the user typed "fluentty".Change
PROBABILITY. The number of characters still to type only breaks ties (e^(0.01 · remaining)).Result (FluentTyper live tests, en_US)
Follow-up changes (unrelated dictionary words, n-gram ranking of corrections) are in #3.
🤖 Generated with Claude Code