Repository navigation
[CELEBORN-2484][BUILD] Make sbt downloads resilient to transient Maven Central failures - #3877
Closed
afterincomparableyum wants to merge 2 commits into
Closed
afterincomparableyum wants to merge 2 commits into
afterincomparableyum wants to merge 2 commits into
Conversation
…n Central failures build/sbt fetched sbt-launch.jar from repo1.maven.org with a single `curl --silent` call and no retry. Any brief failure from Maven Central (DNS, connection reset, 403/429 throttling) failed the whole CI job before a single test ran, and the log never said why. The repo's Actions cache sits at its 10GB limit, so the cached launcher is usually evicted and every SBT matrix job downloads it at the same time. Dependency resolution had the same weak point. build/sbt-config/repositories lists Google's Maven Central mirror ahead of repo1, but at the old /repos/central/data/ path, which now returns 404. Every artifact was really fetched from repo1 alone, so a repo1 connection reset failed the build (e.g. "could not retrieve sbt 1.9.4" when the launcher couldn't download scala-reflect). - Retry the launcher download with curl/wget, and add --retry-all-errors when curl supports it (7.71+), because plain --retry skips DNS, connection and 403 errors. - Print the download error instead of hiding it. - If repo1 still fails, fetch the launcher from Google's Maven Central mirror. Skip that fallback when DEFAULT_ARTIFACT_REPOSITORY is set, so builds limited to an internal repo stay that way. - Point gcs-maven-central-mirror at the live /maven2/ path, as repositories-asia.template already does, so dependencies have a working mirror ahead of repo1.
zaynt4606
requested changes
Oct 19, 2026
zaynt4606
left a comment
zaynt4606
left a comment
Contributor
There was a problem hiding this comment.
Thanks for working on this. I found one issue in the new retry path that needs a fix before approval.
zaynt4606
approved these changes
Oct 19, 2026
zaynt4606
left a comment
zaynt4606
left a comment
Contributor
There was a problem hiding this comment.
Thanks for addressing the partial-download issue. Using curl --output resolves my concern. LGTM from the code review; CI is still running at the time of this approval.
Contributor
|
Merge Summary:
Posted by |
Contributor
|
Thanks @afterincomparableyum for the contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
build/sbt-launch-lib.bash: the sbt launcher download now retries. It adds--retry-all-errorswhen curl is 7.71 or newer, because plain--retryskips DNS, connection and 403 errors. It prints the download error, and if repo1 still fails it falls back to Google's Maven Central mirror. That fallback is skipped whenDEFAULT_ARTIFACT_REPOSITORYis set.build/sbt-config/repositories: pointgcs-maven-central-mirrorat the live/maven2/path. The old/repos/central/data/path returns 404.repositories-asia.templatealready uses/maven2/.Why are the changes needed?
SBT CI jobs fail before any test runs whenever Maven Central has a brief error:
curland no retry. The job died in under a second with onlyOur attempt to download sbt locally ... failedin the log (job 1, job 2). The Actions cache is at its 10GB limit, so the cached launcher is usually evicted and every matrix job downloads it.could not retrieve sbt 1.9.4).Does this PR resolve a correctness bug?
Does this PR introduce any user-facing change?
How was this patch tested?
curlagainst the GCS mirror:/maven2/returns 200 and/repos/central/data/returns 404.