Let the user cap the wait a server asks for with Retry-After - #236
Merged
Merged
Conversation
Engine 3.50.4 obeys Retry-After on a 429 or a 503 and caps the hold at HTS_DEFAULT_MAX_RETRY_AFTER, 60 seconds. That cap already reaches this front end, because hts_create_opt() sets it and htslibjni.c calls it. What was missing is a way to change it, which the Flow control tab now offers beside Timeout and Retries: --max-retry-after caps a wait the server imposes, the same shape as -T. The value needs bounding here rather than in the layout. Outside 0..3600 htscoremain.c calls HTS_PANIC_PRINTF and returns -1, so a value the engine refuses loses the whole crawl and not just the option. A saved profile can hold any value and android:digits never sees it, so BoundedOption drops what the engine would reject. 0 stays a real setting meaning retry with no wait, so only a negative or a non-digit is absent. MaxRetryAfter joins NO_SHARED_DEFAULT because the engine's shared key table still states no agreed default for it, and lists its owners as win,web. Both want a follow-up in the engine repo now that a third front end writes the key. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
The lower bound could never fire, because isDigits already rejects a sign, so a guard that enforced nothing is gone. isInRange becomes isAtMost and BoundedOption becomes CappedOption, which is what the one caller wanted. Four mutants got past the first suite. Dropping the isDigits guard let "+60" through, which the engine panics on at the stray sign. Casting the parse to int let 4294967296 wrap into range. Deleting android:digits from the field went unasserted, and it is what keeps a non-Latin keyboard off the commandline. Hardcoding the ceiling inside emit() survived, because every case ran through the one wiring, so a case now builds its own CappedOption and checks the ceiling it was given. All four are killed. Also: the mapper lookup reuses fieldsNameToMapper rather than repeating it, the literal 3601 goes because it would red for the wrong reason if the engine raised its limit, and the variant count stops pinning a population of one that OptionsTabFieldsTest already covers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Xavier Roche <xroche@gmail.com>
xroche
enabled auto-merge (squash)
September 25, 2026 02:18
xroche
added a commit
that referenced
this pull request
Sep 25, 2026
`versionCode` goes to 106 because 105 already shipped under the tag `v3.50.2.105`, and Play refuses a second upload of the same code. The release notes still opened "HTTrack 3.50.2" in all 29 locales, so they are rewritten for 3.50.4: - #234 dropped the 25 KB/s seed, so a new copy now runs at the engine's own 100 KB/s ceiling - `Retry-After` on a 429 or a 503, with the ceiling settable since #236 - the JavaScript module imports that the engine now follows - the robots.txt group fix Two claims came out. "Fewer crashes when memory runs low" was close to inverted. Engine `d4fc328b` swapped a graceful NULL return for `assertf()`, on a path `htsname.c` reaches during any crawl. "Runs at full speed" was wrong too, because the ceiling moved to 100 KB/s rather than away. --------- Signed-off-by: Xavier Roche <xroche@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Engine 3.50.4 obeys
Retry-Afteron a 429 or a 503 and holds new launches for up to 60 seconds. That already reaches our users, becausehts_create_opt()sets the default andhtslibjni.ccalls it. This adds the field that changes the ceiling, on Flow control beside Timeout and Retries.CappedOptiondrops a value over 3600 instead of passing it on, becausehtscoremain.cpanics there and returns -1, which costs the whole crawl. It lives in the emitter becauseandroid:digitsnever sees a value a saved profile sets directly.That bound belongs in the engine, which already clamps an over-cap
Retry-Afterfrom a server. WebHTTrack emits the same option unbounded today, so it panics where we would not. Treat this as a mitigation until the engine clamps.MaxRetryAfterjoinsNO_SHARED_DEFAULTbecause the engine's key table states no agreed default and lists its owners aswin,web. The engine repo should fix both.