Mirror the MinIO CI image to GHCR - #2
Merged
Merged
Conversation
MinIO has withdrawn anonymous public access to its server image: first from Docker Hub (~13 Sept), then from quay.io (~24 Sept). Both registries now answer 401 to anonymous manifest requests for every tag, and dl.min.io answers 410. Any Dockerfile doing `FROM <registry>/minio/minio` fails before tests run, which is every tox-docker job in circulation, library-registry and virtual-library-card. Pinning an older tag does not help (the whole repository is gated, not just :latest), and neither does a local cache (tox_docker passes pull=True on every build, forcing a fresh pull). This adds a mirror published as ghcr.io/thepalaceproject/palace-ci-minio, built from MinIO's GitHub release assets, the one channel still served anonymously. Every artifact is pinned by sha256, and the binaries are byte-for-byte identical to the ones inside the last upstream image CI used, verified by comparing their sha256 against /usr/bin/minio and /usr/bin/mc extracted from that image. It lives here rather than in circulation because the image is shared by three repos and belongs to none of them in particular. The image is a bare passthrough with no Palace configuration, so the consuming repos keep their own differing MinIO setups and change only a FROM line. It is pinned to an immutable release tag and publishes no `latest`, and only pushes to main publish, so an unmerged branch cannot overwrite that tag. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 24, 2026
Merged
Merged
tdilauro
approved these changes
Sep 24, 2026
dbernstein
added a commit
to ThePalaceProject/circulation
that referenced
this pull request
Sep 25, 2026
## Description Switches `docker/Dockerfile.minio.ci` from `quay.io/minio/minio:latest`, which no longer permits anonymous pulls, to the Palace mirror: ```diff -FROM quay.io/minio/minio:latest AS minio +# MinIO withdrew anonymous public access to its own image from both Docker Hub and quay.io, so +# this pulls from the Palace mirror of the upstream release instead. The mirror is an unmodified +# MinIO build, published from the ci-scripts repo: +# https://github.com/ThePalaceProject/ci-scripts/blob/main/images/minio/Dockerfile +# The tag is pinned deliberately; the mirror publishes no `latest`. +FROM ghcr.io/thepalaceproject/palace-ci-minio:RELEASE.2025-09-07T16-13-09Z AS minio ``` That is the whole change: one `FROM` line, plus a comment recording why it points somewhere unusual. The mirror is an unmodified MinIO build, so the credentials, ports and command below the `FROM` are untouched. The mirror is published and public: `ghcr.io/thepalaceproject/palace-ci-minio` is live at digest `sha256:0e8998d7ba6f742bac3c6ad22f354979b987d5d31c0a95b9b0dcad9d73e32751` with `linux/amd64` and `linux/arm64`, and an anonymous (unauthenticated) manifest request returns 200. It is published from [ci-scripts](https://github.com/ThePalaceProject/ci-scripts) rather than from here, because the image is shared by three repos and belongs to none of them in particular. Companion one-line changes: ThePalaceProject/library-registry#1066 and ThePalaceProject/virtual-library-card#1016. The three are independent and can merge in any order. ## Motivation and Context [JIRA](https://ebce-lyrasis.atlassian.net/browse/PP-5244) All tox-docker CI jobs currently fail on `main` while building the MinIO test container: ``` docker.errors.BuildError: unauthorized: access to the requested resource is not authorized ``` MinIO has withdrawn anonymous public access to its server image from Docker Hub (~13 Sept, worked around in #3728 by moving to quay.io) and now from quay.io as well. Ten checks fail on `main`: Tests (Py 3.12 / 3.13 / 3.14), OpenSearch 2.19, OpenSearch 3.5, Unit tests (amd64 / arm64), Integration test (amd64 / arm64) and the Backwards compatibility test. Lint, mypy, CodeQL, the Migration test and the Docker build are unaffected because they do not use tox-docker. Local `tox` fails identically, so developers cannot run the suites either. Pinning an older tag does not help — the whole upstream repository is gated, not just `:latest` — and neither does a local Docker cache, because `tox_docker` passes `pull=True` on every build. Full background, registry probes and the retirement plan are in ThePalaceProject/ci-scripts#2. ## How Has This Been Tested? **This PR's own CI is the acceptance test, and it passes.** All ten checks listed above are green against the published mirror, including the Backwards compatibility test and both Integration and Unit test jobs on amd64 and arm64. Locally, `tox -e py312-docker -- -m minio` was run against the real `ghcr.io/thepalaceproject/palace-ci-minio` reference with the image first cleared from the local Docker cache, so the pull genuinely came from GHCR: ``` py312-docker: docker> build .../docker/Dockerfile.minio.ci target 'minio' py312-docker: docker> run '2827512c875a' (from 'minio-circ') py312-docker: commands[0]> pytest -m minio --no-cov -q 14 passed in 16.48s ``` Before the mirror was published, the same chain was exercised by pushing the image to a throwaway `registry:2` on localhost and temporarily pointing this `FROM` line at it — `tox_docker` passes `pull=True` on every build, so a local-only tag is rejected and a real registry is required either way. That run also gave 14 passed. ## Checklist - [x] I have updated the documentation accordingly. - [x] All new and existing tests passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
dbernstein
added a commit
to ThePalaceProject/library-registry
that referenced
this pull request
Sep 25, 2026
## Description Switches the CI MinIO container from `quay.io/minio/minio:latest`, which no longer permits anonymous pulls, to the Palace mirror: ```diff -FROM quay.io/minio/minio:latest AS minio +# MinIO withdrew anonymous public access to its own image from both Docker Hub and quay.io, so +# this pulls from the Palace mirror of the upstream release instead. The mirror is an unmodified +# MinIO build, published from the ci-scripts repo: +# https://github.com/ThePalaceProject/ci-scripts/blob/main/images/minio/Dockerfile +# The tag is pinned deliberately; the mirror publishes no `latest`. +FROM ghcr.io/thepalaceproject/palace-ci-minio:RELEASE.2025-09-07T16-13-09Z AS minio ``` That is the whole change to `docker/Dockerfile.minio.ci`: one `FROM` line, plus a comment recording why it points somewhere unusual. The mirror is an unmodified MinIO build with no Palace configuration baked in, so this repo keeps its own credentials, buckets and entrypoint exactly as they are. The mirror is published and public: `ghcr.io/thepalaceproject/palace-ci-minio` is live at digest `sha256:0e8998d7ba6f742bac3c6ad22f354979b987d5d31c0a95b9b0dcad9d73e32751` with `linux/amd64` and `linux/arm64`, and an anonymous (unauthenticated) manifest request returns 200. ## Motivation and Context MinIO has progressively withdrawn public distribution of its server image — from Docker Hub around 13 Sept, and from quay.io around 24 Sept. Both registries now answer 401 to anonymous manifest requests for **every** tag, and `dl.min.io` answers 410. Any Dockerfile doing `FROM <registry>/minio/minio` therefore fails before the test suite runs: ``` docker.errors.BuildError: unauthorized: access to the requested resource is not authorized ``` This repo moved to quay.io in PP-5132 when Docker Hub closed; quay.io has now closed too, so that workaround no longer helps. Two things that look like fixes are not: pinning a digest or an older tag does not help, because the whole upstream repository is gated rather than a single tag; and a local Docker cache does not help, because `tox_docker` passes `pull=True` on every build, forcing a fresh pull. Because MinIO appears to be withdrawing public distribution progressively, any third-party MinIO source is likely to fail again, so the image is mirrored into the org's own registry as `ghcr.io/thepalaceproject/palace-ci-minio`, published from the [ci-scripts](https://github.com/ThePalaceProject/ci-scripts) repo. It is assembled from MinIO's official GitHub release binaries — the one channel still served anonymously — each pinned by sha256, and those binaries are byte-for-byte identical to the ones inside the last upstream image CI used. It is public (no `docker login` for developers, no package-access grants for CI) and pinned to an immutable tag with no `latest`. The mirror is a bridge, not a destination; [PP-5245](https://ebce-lyrasis.atlassian.net/browse/PP-5245) tracks replacing MinIO outright. Full background, registry probes and the retirement plan are in ThePalaceProject/ci-scripts#2. ## How Has This Been Tested? **This PR's own CI is the acceptance test, and it passes** against the published mirror. Locally, the container built from this repo's `docker/Dockerfile.minio.ci` came up healthy (`/minio/health/live` returned 200) and `mc` inside the container created a bucket successfully — confirming the mirror carries the `mc` client and the `curl` that this repo's setup and its `docker-compose.yml` healthcheck rely on. This file is byte-for-byte identical to circulation's `docker/Dockerfile.minio.ci` (verified with `diff`), and the same mirror was exercised end-to-end through `tox` there: all 14 `minio`-marked tests passed against a container built from it, pulling from GHCR with the image cleared from the local Docker cache. ## Checklist - [x] I have updated the documentation accordingly. - [x] All new and existing tests passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) [PP-5245]: https://ebce-lyrasis.atlassian.net/browse/PP-5245?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
dbernstein
added a commit
to ThePalaceProject/virtual-library-card
that referenced
this pull request
Sep 25, 2026
## Description Switches the CI MinIO container from `quay.io/minio/minio:latest`, which no longer permits anonymous pulls, to the Palace mirror: ```diff -FROM quay.io/minio/minio:latest +# MinIO withdrew anonymous public access to its own image from both Docker Hub and quay.io, so +# this pulls from the Palace mirror of the upstream release instead. The mirror is an unmodified +# MinIO build, published from the ci-scripts repo: +# https://github.com/ThePalaceProject/ci-scripts/blob/main/images/minio/Dockerfile +# The tag is pinned deliberately; the mirror publishes no `latest`. +FROM ghcr.io/thepalaceproject/palace-ci-minio:RELEASE.2025-09-07T16-13-09Z ``` That is the whole change to `ci/minio/Dockerfile`: one `FROM` line, plus a comment recording why it points somewhere unusual. The mirror is an unmodified MinIO build with no Palace configuration baked in, so this repo keeps its own credentials, its `vlc-test` bucket and its `entrypoint.sh` exactly as they are. **The README needed fixing too.** Step 3 of the local setup ("Set up public MinIO bucket") told developers to `docker run quay.io/minio/minio`, which now 401s just as CI did — so a new developer could not get VLC running locally. That command needed more than a registry swap: the upstream image wrapped the binary in an entrypoint script, so `... quay.io/minio/minio server /data` worked, whereas the mirror ships no entrypoint and the same arguments fail with `exec: "server": executable file not found in $PATH`, leaving the container in `Created` and never running. The documented command now names the `minio` binary explicitly and says why. The mirror is published and public: `ghcr.io/thepalaceproject/palace-ci-minio` is live at digest `sha256:0e8998d7ba6f742bac3c6ad22f354979b987d5d31c0a95b9b0dcad9d73e32751` with `linux/amd64` and `linux/arm64`, and an anonymous (unauthenticated) manifest request returns 200. ## Motivation and Context MinIO has progressively withdrawn public distribution of its server image — from Docker Hub around 13 Sept, and from quay.io around 24 Sept. Both registries now answer 401 to anonymous manifest requests for **every** tag, and `dl.min.io` answers 410. Any Dockerfile doing `FROM <registry>/minio/minio` therefore fails before the test suite runs: ``` docker.errors.BuildError: unauthorized: access to the requested resource is not authorized ``` This repo moved to quay.io in PP-5132 when Docker Hub closed; quay.io has now closed too, so that workaround no longer helps. Two things that look like fixes are not: pinning a digest or an older tag does not help, because the whole upstream repository is gated rather than a single tag; and a local Docker cache does not help, because `tox_docker` passes `pull=True` on every build, forcing a fresh pull. Because MinIO appears to be withdrawing public distribution progressively, any third-party MinIO source is likely to fail again, so the image is mirrored into the org's own registry as `ghcr.io/thepalaceproject/palace-ci-minio`, published from the [ci-scripts](https://github.com/ThePalaceProject/ci-scripts) repo. It is assembled from MinIO's official GitHub release binaries — the one channel still served anonymously — each pinned by sha256, and those binaries are byte-for-byte identical to the ones inside the last upstream image CI used. It is public (no `docker login` for developers, no package-access grants for CI) and pinned to an immutable tag with no `latest`. The mirror is a bridge, not a destination; [PP-5245](https://ebce-lyrasis.atlassian.net/browse/PP-5245) tracks replacing MinIO outright. Full background, registry probes and the retirement plan are in ThePalaceProject/ci-scripts#2. ## How Has This Been Tested? **This PR's own CI is the acceptance test, and it passes** against the published mirror. This repo asks more of the image than the other two do — its `entrypoint.sh` needs a shell, `curl` for the readiness loop, and the `mc` client — so the image was also built from `ci/minio/Dockerfile` and run locally against the mirror. The entrypoint ran to completion: ``` Waiting for MinIO to start... MinIO is ready Creating bucket: vlc-test Bucket created successfully `vlcminio/vlc-test`. Setting bucket policy to download (public read) Access permission for `vlcminio/vlc-test` is set to `download` MinIO setup complete ``` `/minio/health/live` returned 200, and an anonymous read of `vlc-test` returned 200, confirming the download policy took effect. The README's instructions were then run verbatim against the published mirror — the corrected `docker run`, `docker exec -it minio bash`, `mc alias set`, `mc mb` and `mc anonymous set public`. All succeeded; the console on :9001 returned 200 and an anonymous read of the bucket returned 200. The pre-correction command was also run, to confirm it genuinely fails rather than merely looking wrong: ``` exec: "server": executable file not found in $PATH ``` ## Checklist - [x] I have updated the documentation accordingly. - [x] All new and existing tests passed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) [PP-5245]: https://ebce-lyrasis.atlassian.net/browse/PP-5245?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
dbernstein
added a commit
that referenced
this pull request
Sep 25, 2026
## Description Names PP-5245 as the retirement ticket for the MinIO mirror, in the three places that previously said it should be retired without saying where to find the ticket: - `images/minio/Dockerfile` — the "bridge, not a destination" note - `.github/workflows/build-minio-mirror.yml` — the header - `README.md` — a new `### Retirement` section Two smaller cleanups while in there: - The README's retirement section now carries the 5,000-download deletion threshold **next to** the `gh api -X DELETE` command, so the cost of waiting is visible right where someone would go to act. Previously the threshold was a trailing sentence and the command lived only in a PR description. - The workflow's "one-time manual step: flip the package to public" note read as a pending TODO. It has been done, so it now records what was done and notes it would need doing again if the package were ever recreated — GHCR creates packages private and the workflow cannot change that. ## Motivation and Context The mirror exists because MinIO withdrew anonymous public access to its server image from both Docker Hub and quay.io (#2). It is deliberately a stopgap, and `images/minio/Dockerfile` tells readers "do not add features to it" — but it then pointed at a "retirement ticket" that had no identifier, so there was no way to follow the instruction to its conclusion. That gap matters more than a missing cross-reference normally would. Retiring this gets strictly harder with time: GitHub does not allow self-service deletion of a public package once any version passes 5,000 downloads, after which it takes a Support request. With `pull=True` on every tox-docker build, ephemeral CI runners and three repos pulling, an untracked mirror is one that quietly becomes permanent. ## Also: the mirror's entrypoint difference from upstream A second commit records a behavioural difference that had already caused a real breakage. The upstream image set `ENTRYPOINT` to a `docker-entrypoint.sh` that supplied the `minio` binary, so `docker run <image> server /data` worked. This mirror ships no entrypoint, so that same invocation fails with `exec: "server": executable file not found in $PATH` and the container never leaves `Created`. The three CI Dockerfiles all set their own command, so none of them hit this — which is why it went unnoticed. But virtual-library-card's README documented exactly that invocation for local development, and it broke silently (fixed in ThePalaceProject/virtual-library-card#1016). The Dockerfile now says so next to its `CMD`, so the next person invoking the image directly finds it before debugging a container that refuses to start. ## How Has This Been Tested? The Dockerfile change is comment-only — no instruction changes — so the image contents are unchanged: the same release binaries pinned by sha256 and the same UBI base pinned by digest. Note that merging this **does** re-trigger the publish workflow, because both changed files are in its `paths:` filter. That is expected and harmless; the workflow's PR run on this branch builds both architectures and verifies every pinned checksum without pushing, which is the check that the Dockerfile is still valid. The published manifest digest may change even though the contents do not, since buildx attaches fresh provenance to each build — consumers pin the tag, not the digest, so nothing downstream is affected. The entrypoint behaviour documented in the second commit was verified against the published image: `docker run <mirror> server /data --console-address ":9001"` fails as described and leaves the container in `Created`, while `docker run <mirror> minio server /data --console-address ":9001"` comes up healthy in ~2s, with `mc` and the console both working inside it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a mirror of the upstream MinIO server image, published as
ghcr.io/thepalaceproject/palace-ci-minio:images/minio/Dockerfile— assembles the image from MinIO's official GitHub release binaries(
minioRELEASE.2025-09-07T16-13-09Z,mcRELEASE.2025-08-13T08-35-41Z) plus the AGPLlicense files, each pinned by sha256 via BuildKit's
ADD --checksum..github/workflows/build-minio-mirror.yml— buildslinux/amd64+linux/arm64and pushes toGHCR. Runs only when the Dockerfile or the workflow changes, or on demand.
This lives in ci-scripts rather than circulation because the image is shared by three repos and
belongs to none of them in particular. GHCR packages are namespaced by org, not repo, so one mirror
serves all three. Supersedes ThePalaceProject/circulation#3766, which is now closed.
Design notes
closed to anonymous pulls, so there is no image left to
docker pull, anddl.min.iois gonetoo (410). GitHub release assets are the one channel MinIO still serves anonymously. The binaries
are byte-for-byte identical to the ones inside the last upstream image CI used — verified below.
immediately, so each repo keeps its own Dockerfile and changes only its
FROMline.latest. Following a moving upstream tag is part of how wegot here.
RELEASE.2025-09-07T16-13-09Zis also the last MinIO release to publish binaries atall — later tags ship no assets — so there is no newer version to move to.
mainpublishes. Pull requests andworkflow_dispatchruns from a branch build botharchitectures and verify the checksums, then discard the result. An unmerged branch must not be
able to overwrite a tag three other repos pin their CI to.
circ-webapp,circ-scripts,circ-execandcirc-baseimage, so nodocker loginfor developers and no package-access grants for CI.palace-rather thancirc-prefix, because the image is shared across three repos.GHCR packages are created private. Once the workflow has run, set
palace-ci-minioto public(Org → Packages →
palace-ci-minio→ Package settings). Until that is done, the three consumer PRswill still fail.
Consumer PRs, all draft, each a single
FROMline:ThePalaceProject/circulation#3767, ThePalaceProject/library-registry#1066,
ThePalaceProject/virtual-library-card#1016.
Motivation and Context
All tox-docker CI jobs in circulation fail while building the MinIO test container, and the other
two repos will hit the same wall on their next run:
MinIO has progressively withdrawn public distribution — Docker Hub around 13 Sept (worked around in
ThePalaceProject/circulation#3728 by moving to quay.io), and quay.io around 24 Sept. Anonymous
probes on 24 Sept:
quay.io/minio/minio:latestmanifestquay.io/minio/minio:RELEASE.2024-01-16T16-07-38Z(pinned old tag)registry-1.docker.io/minio/minio:latestmanifestminio/minioobject not founddl.min.ioserver/client binariesquay.io/prometheus/busybox:latest(control)Quay issues an anonymous pull token and then refuses the manifest, so the repository is gated rather
than the network; the control confirms anonymous pulls work from the same machine. Two consequences:
pinning a digest or an older tag will not help, because the whole repository is closed; and a local
cache will not help, because
tox_dockerpassespull=Trueon every build(
tox_docker/plugin.py), forcing a fresh pull. Localtoxfails identically, so developers cannotrun the suites either.
Retirement
This mirror is a bridge, not a destination — the intent is to drop MinIO for a maintained
S3-compatible image. It should be short-lived for two reasons: we do not want to become a de-facto
public distributor of a frozen MinIO build; and GitHub does not allow self-service deletion of a
public package once any version exceeds 5,000 downloads, above which it becomes a Support request.
With
pull=Trueon every build, ephemeral runners and three repos pulling, that threshold arrivesquickly.
When the time comes:
How Has This Been Tested?
Verified locally on macOS / Docker 29.6.1 (arm64 host).
The mirrored artifacts are identical to the withdrawn upstream image. A cached copy of
quay.io/minio/minio:latest(labelledRELEASE.2025-09-07T16-13-09Z) was still present locally;its binaries and license files hash the same as what the mirror downloads:
Build. Builds for
linux/amd64andlinux/arm64; both architectures' checksums verify (amismatch fails the build by design). The final stage has no
RUNinstructions, so it cross-buildswithout QEMU — confirmed on a CI runner that advertised only
linux/amd64…/386as supported.Runtime.
minio --versionandmc --versionreport the expected releases; the server comes upand
/minio/health/livereturns 200 in ~3s; the console on :9001 returns 200;curlinside thecontainer returns 200 against the health endpoint (the check
docker-compose.ymluses); andmc alias set/mc mb/mc anonymous set downloadall succeed (the operationsvirtual-library-card's entrypoint needs).
/licenses/LICENSEand/licenses/CREDITSare presentand hash-identical to upstream's.
End-to-end through tox. Because
pull=Truerejects a local-only tag, the image was pushed to athrowaway
registry:2on localhost and circulation'sdocker/Dockerfile.minio.citemporarilypointed at it, so the whole chain was exercised — tox building the consumer Dockerfile, pulling the
mirror from a real registry, starting the container and running the tests:
The same local-registry substitution was used to build and run library-registry's
docker/Dockerfile.minio.ci(healthy,mcworks) and virtual-library-card'sci/minio/Dockerfile(entrypoint completes — bucket created, anonymous download policy set, anonymous bucket read returns
200).
🤖 Generated with Claude Code