Repository navigation
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the third item of #6: Latin-script text with no identified language was routed to the English checkpoint, which cannot read it.
deepopen/lang.pystill decided"is_english": lang in (None, "en"), so every Latin-script language with no stopword match landed on English. This copies the fix thatlaya0.3.5 shipped for the same regression: undecided is not English — non-English letters alone are enough to prefer the multilingual checkpoint, and text without them still goes to English. Concretely,lang.pyis synced to the 0.3.5 version (this file is a copy of Laya's, Apache-2.0): Romanian stopwords, the expanded non-English diacritic set,latin_profile()and itslooks_non_englishevidence, and the undecided-awareanalyse().router.route()gained the matching reason branch, so an unidentified Latin-script language reports "non-English letters" instead of quotingNone.Before / after, same sentences as the issue (verified locally,
python tests/test_router.py):None)ro)en!)None)None)None)Tests: the two
from laya...imports intests/test_router.pynow point atdeepopen(they raisedModuleNotFoundError, so the file could not run at all), plus regression checks for Romanian/Polish/Latvian routing, therostopword match, and the undecided reason. 115/115 pass with this change; the pre-existing 106 passed before it.The rest of item 2 (test_criteria/test_local_e2e imports, the CI workflow's
laya/paths) is left for the rename suggested in the issue.