Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 127 additions & 0 deletions .github/workflows/build-minio-mirror.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
name: Build MinIO Mirror

# Publishes ghcr.io/thepalaceproject/palace-ci-minio, the MinIO image used by the test suites of
# circulation, library-registry and virtual-library-card. MinIO withdrew anonymous public access
# to its own image (Docker Hub, then quay.io), so we host a copy built from its official GitHub
# release binaries. See images/minio/Dockerfile for the full rationale.
#
# The image is pinned to a single upstream release and its content is fully determined by the
# checksums in that Dockerfile, so there is no scheduled rebuild: running this again produces the
# same image. It runs only when the Dockerfile or this workflow changes, or on demand.
#
# Only main publishes. Pull requests and manual runs from a branch build both architectures and
# verify the pinned checksums, then throw the result away — an unmerged branch must never be able
# to overwrite the release tag that three other repos pin their CI to.
#
# One-time manual step: GHCR packages are created private. After the first successful run, set the
# package visibility to public (Org -> Packages -> palace-ci-minio -> Package settings), matching
# the other Palace images. Without that, every developer and CI job would need a docker login.

on:
push:
branches:
- main
paths:
- .github/workflows/build-minio-mirror.yml
- images/minio/Dockerfile
# Build (but do not push) on PRs that touch the mirror, so a broken Dockerfile or workflow is
# caught in review rather than on main, where the failure would leave the image unpublished.
pull_request:
paths:
- .github/workflows/build-minio-mirror.yml
- images/minio/Dockerfile
workflow_dispatch:

concurrency:
group: build-minio-mirror-${{ github.ref_name }}-${{ github.event_name }}
cancel-in-progress: true

jobs:
build:
name: Build MinIO Mirror
runs-on: ubuntu-24.04
timeout-minutes: 30
permissions:
contents: read
packages: write

steps:
- uses: actions/checkout@v7
with:
persist-credentials: false

# See comment here: https://github.com/actions/runner-images/issues/1187#issuecomment-686735760
- name: Disable network offload
run: sudo ethtool -K eth0 tx off rx off

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v4

# Publishing is gated on the ref rather than the event, so a workflow_dispatch from an
# unmerged branch builds and validates but cannot push over the pinned tag.
- name: Check whether this run publishes
id: gate
run: |
set -euo pipefail
if [[ "${GITHUB_REF}" == "refs/heads/main" ]]; then
echo "This run publishes to GHCR."
echo "publish=true" >> "$GITHUB_OUTPUT"
else
echo "Ref is ${GITHUB_REF}, not refs/heads/main: building for validation only."
echo "publish=false" >> "$GITHUB_OUTPUT"
fi

- name: Login to GitHub Container Registry
if: steps.gate.outputs.publish == 'true'
uses: docker/login-action@v4.6.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

# The Dockerfile is the single source of truth for which upstream release we mirror, so the
# tag is read back out of it rather than duplicated here where the two could drift apart.
- name: Determine image and tag
id: image
run: |
set -euo pipefail
image="ghcr.io/${GITHUB_REPOSITORY_OWNER,,}/palace-ci-minio"
tag=$(sed -n 's/^ARG MINIO_RELEASE=\(.*\)$/\1/p' images/minio/Dockerfile | head -1)
if [[ -z "$tag" ]]; then
echo "::error::Could not read MINIO_RELEASE from images/minio/Dockerfile"
exit 1
fi
echo "Image: $image:$tag"
echo "image=$image" >> "$GITHUB_OUTPUT"
echo "tag=$tag" >> "$GITHUB_OUTPUT"

# The mirror contains no RUN instructions, so both architectures cross-build on this single
# amd64 runner with no QEMU emulation and no per-arch runner matrix.
- name: Build mirror image
uses: docker/build-push-action@v7
with:
context: images/minio
file: ./images/minio/Dockerfile
target: minio
platforms: linux/amd64,linux/arm64
push: ${{ steps.gate.outputs.publish == 'true' }}
# Deliberately no `latest` tag: consumers pin this exact release. Following a moving
# upstream tag is part of how we ended up needing this mirror.
tags: ${{ steps.image.outputs.image }}:${{ steps.image.outputs.tag }}

- name: Verify published image
if: steps.gate.outputs.publish == 'true'
run: |
set -euo pipefail
ref="${{ steps.image.outputs.image }}:${{ steps.image.outputs.tag }}"
docker buildx imagetools inspect "$ref"
# Confirm both architectures actually made it into the published manifest list.
platforms=$(docker buildx imagetools inspect "$ref" --raw \
| jq -r '.manifests[] | select(.platform.os != "unknown") | "\(.platform.os)/\(.platform.architecture)"')
echo "Published platforms: $platforms"
for platform in linux/amd64 linux/arm64; do
grep -qx "$platform" <<< "$platforms" \
|| { echo "::error::$platform missing from $ref"; exit 1; }
done
# Smoke-test the runner's native architecture.
docker run --rm --entrypoint /usr/bin/minio "$ref" --version
28 changes: 28 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,33 @@
# CI-Scripts

Shared CI helpers for the Palace Project repositories.

## images/minio

A mirror of the upstream MinIO server image, published to
`ghcr.io/thepalaceproject/palace-ci-minio` by
[`.github/workflows/build-minio-mirror.yml`](.github/workflows/build-minio-mirror.yml).

MinIO withdrew anonymous public access to its server image from both Docker Hub and quay.io, so
`FROM minio/minio` now fails with a 401 before any test suite starts. The mirror is assembled from
MinIO's official GitHub release binaries — the one channel still served anonymously — each pinned
by sha256. It is a bare passthrough: no credentials, no buckets, no entrypoint script, because the
repos that consume it each configure MinIO differently.

Consumers pin the exact release tag; the mirror deliberately publishes no `latest`:

```dockerfile
FROM ghcr.io/thepalaceproject/palace-ci-minio:RELEASE.2025-09-07T16-13-09Z
```

Used by `circulation`, `library-registry` and `virtual-library-card`. Only pushes to `main`
publish — pull requests and manual runs from a branch build and validate, then discard, so an
unmerged branch cannot overwrite the tag those repos depend on.

This mirror is a bridge, not a destination: the intent is to drop MinIO for a maintained
S3-compatible image. Note that GitHub does not allow self-service deletion of a public package
once any version passes 5,000 downloads.

## sync.py

`sync.py` is a helper script used in our CI process to keep a branch on our repositories in sync with upstream.
Expand Down
118 changes: 118 additions & 0 deletions images/minio/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
# syntax=docker/dockerfile:1.7
#
# Mirror of the upstream MinIO server image, published as
# ghcr.io/thepalaceproject/palace-ci-minio.
#
# Why this exists
# ---------------
# MinIO withdrew anonymous public access to its server image: first from Docker Hub
# (~13 Sept 2026), then from quay.io (~24 Sept 2026). Both registries now answer 401 to
# anonymous manifest requests for every tag, and the binary download host (dl.min.io) answers
# 410. Every Palace repo that ran `FROM <registry>/minio/minio` in its test setup therefore
# fails before its test suite starts. Pinning an older tag or relying on a local Docker cache
# does not help: the whole repository is gated, and tox-docker passes `pull=True` on every
# build, forcing a fresh pull.
#
# The one channel MinIO still serves anonymously is GitHub release assets, so this image is
# assembled from the official release binaries rather than pulled and re-tagged. The binaries
# below are byte-for-byte identical to the ones inside the last upstream image CI used
# (quay.io/minio/minio:latest, labelled RELEASE.2025-09-07T16-13-09Z) — verified by comparing
# their sha256 against /usr/bin/minio and /usr/bin/mc extracted from that image. The same holds
# for the license files in /licenses.
#
# Note that RELEASE.2025-09-07T16-13-09Z is the last MinIO release to publish binaries at all;
# later tags ship no assets. There is no newer version to move to.
#
# Scope
# -----
# This is a bare passthrough. It contains no Palace configuration: no credentials, no buckets,
# no entrypoint script. The repos that consume it (circulation, library-registry,
# virtual-library-card) each configure MinIO differently and keep doing so in their own
# Dockerfiles — they change only their `FROM` line. Baking configuration in here would fork
# the image between repos immediately.
#
# This mirror is a bridge, not a destination; see the retirement ticket for replacing MinIO
# outright. Do not add features to it.
#
# Updating
# --------
# Every downloaded artifact is pinned by sha256. The release strings and their checksums are a
# matched set: if you bump a version you MUST also replace the matching `--checksum=` values, or
# the build will fail (by design). Checksums come from the `.sha256sum` asset published
# alongside each binary. The base image is pinned by digest so that a rebuild reproduces the
# same image rather than picking up whatever `9.6` points at that day.

# Declared once here and inherited by every stage below with a bare `ARG`. Keeping a single
# definition means the tag the workflow publishes, the binaries the checksums cover and the
# labels on the image cannot drift apart in a version bump.
ARG BASE_IMAGE=registry.access.redhat.com/ubi9/ubi-minimal:9.6@sha256:34880b64c07f28f64d95737f82f891516de9a3b43583f39970f7bf8e4cfa48b7
ARG MINIO_RELEASE=RELEASE.2025-09-07T16-13-09Z
ARG MC_RELEASE=RELEASE.2025-08-13T08-35-41Z

# MinIO publishes one binary per platform rather than a multi-arch artifact, and each has its
# own checksum, so the download is split into a per-architecture stage that `fetch` selects
# from using TARGETARCH. This keeps checksum verification native to BuildKit's ADD.
FROM ${BASE_IMAGE} AS fetch-amd64
ARG MINIO_RELEASE
ARG MC_RELEASE
ADD --chmod=755 --checksum=sha256:7c5bd8512c6e966455b1d198209358b2d191c77a83ab377c4073281065fb855f \
https://github.com/minio/minio/releases/download/${MINIO_RELEASE}/minio.linux-amd64.${MINIO_RELEASE} \
/staging/minio
ADD --chmod=755 --checksum=sha256:01f866e9c5f9b87c2b09116fa5d7c06695b106242d829a8bb32990c00312e891 \
https://github.com/minio/mc/releases/download/${MC_RELEASE}/mc.linux-amd64.${MC_RELEASE} \
/staging/mc

FROM ${BASE_IMAGE} AS fetch-arm64
ARG MINIO_RELEASE
ARG MC_RELEASE
ADD --chmod=755 --checksum=sha256:5c83cd2cf151717ba0243f73e1c7802ff36e272b67144bdd7f1f7d684fd6f03d \
https://github.com/minio/minio/releases/download/${MINIO_RELEASE}/minio.linux-arm64.${MINIO_RELEASE} \
/staging/minio
ADD --chmod=755 --checksum=sha256:14c8c9616cfce4636add161304353244e8de383b2e2752c0e9dad01d4c27c12c \
https://github.com/minio/mc/releases/download/${MC_RELEASE}/mc.linux-arm64.${MC_RELEASE} \
/staging/mc

FROM fetch-${TARGETARCH} AS fetch

FROM ${BASE_IMAGE} AS minio
ARG MINIO_RELEASE
ARG MC_RELEASE

LABEL org.opencontainers.image.title="palace-ci-minio" \
org.opencontainers.image.description="Unmodified MinIO server build, mirrored for Palace CI because upstream withdrew anonymous registry access." \
org.opencontainers.image.version="${MINIO_RELEASE}" \
org.opencontainers.image.source="https://github.com/ThePalaceProject/ci-scripts" \
org.opencontainers.image.vendor="The Palace Project" \
org.opencontainers.image.licenses="AGPL-3.0-or-later" \
io.palace.minio.release="${MINIO_RELEASE}" \
io.palace.mc.release="${MC_RELEASE}"

COPY --from=fetch /staging/minio /usr/bin/minio
COPY --from=fetch /staging/mc /usr/bin/mc

# MinIO and mc are AGPL-3.0, and this image redistributes them publicly, so it carries the same
# license text and dependency attribution the upstream image shipped in /licenses. Both files
# are byte-identical to the ones in quay.io/minio/minio:latest. `mc` is under the same license,
# and its LICENSE file has the same contents, so one copy covers both.
ADD --chmod=644 --checksum=sha256:0d96a4ff68ad6d4b6f1f30f713b18d5184912ba8dd389f86aa7710db079abcb0 \
https://raw.githubusercontent.com/minio/minio/${MINIO_RELEASE}/LICENSE \
/licenses/LICENSE
ADD --chmod=644 --checksum=sha256:113b8c63dfd987d3652950d7c2aecb1c0952b41781e79bcdac6a316418d48542 \
https://raw.githubusercontent.com/minio/minio/${MINIO_RELEASE}/CREDITS \
/licenses/CREDITS

# Matches the upstream image: MinIO reads credentials from these files when the corresponding
# environment variables are not set, and `mc` needs a writable config dir.
ENV MINIO_ROOT_USER_FILE=access_key \
MINIO_ROOT_PASSWORD_FILE=secret_key \
MINIO_KMS_SECRET_KEY_FILE=kms_master_key \
MINIO_CONFIG_ENV_FILE=config.env \
MC_CONFIG_DIR=/tmp/.mc

# Declaring the volume creates the mount point, so no RUN is needed here. Keeping this stage
# free of RUN instructions means the image cross-builds for every architecture without QEMU.
VOLUME ["/data"]
EXPOSE 9000 9001

# A plain, unconfigured server. Consumers override this with their own CMD/ENTRYPOINT.
CMD ["minio", "server", "/data", "--address", ":9000", "--console-address", ":9001"]