Skip to content

Route undecided Latin-script text to the multilingual checkpoint (fixes #6) - #7

Open
kuishou68 wants to merge 1 commit into
deepopen-com:mainfrom
kuishou68:cocoon-fix-latin-routing
Open

kuishou68 wants to merge 1 commit into
deepopen-com:mainfrom
kuishou68:cocoon-fix-latin-routing

Conversation

@kuishou68

Copy link
Copy Markdown

Fixes the third item of #6: Latin-script text with no identified language was routed to the English checkpoint, which cannot read it.

deepopen/lang.py still decided "is_english": lang in (None, "en"), so every Latin-script language with no stopword match landed on English. This copies the fix that laya 0.3.5 shipped for the same regression: undecided is not English — non-English letters alone are enough to prefer the multilingual checkpoint, and text without them still goes to English. Concretely, lang.py is synced to the 0.3.5 version (this file is a copy of Laya's, Apache-2.0): Romanian stopwords, the expanded non-English diacritic set, latin_profile() and its looks_non_english evidence, and the undecided-aware analyse().

router.route() gained the matching reason branch, so an unidentified Latin-script language reports "non-English letters" instead of quoting None.

Before / after, same sentences as the issue (verified locally, python tests/test_router.py):

language before after
Romanian english (None) multilingual (ro)
Polish english (en!) multilingual (None)
Latvian english (None) multilingual (None)
German / English unchanged unchanged

Tests: the two from laya... imports in tests/test_router.py now point at deepopen (they raised ModuleNotFoundError, so the file could not run at all), plus regression checks for Romanian/Polish/Latvian routing, the ro stopword match, and the undecided reason. 115/115 pass with this change; the pre-existing 106 passed before it.

The rest of item 2 (test_criteria/test_local_e2e imports, the CI workflow's laya/ paths) is left for the rename suggested in the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant