Skip to content

ci(WIP): build and publish arm64 container images natively - #139

Draft
zarcen wants to merge 2 commits into
bootc-dev:mainfrom
zarcen:bink-arm64-ci-images
Draft

zarcen wants to merge 2 commits into
bootc-dev:mainfrom
zarcen:bink-arm64-ci-images

Conversation

@zarcen

@zarcen zarcen commented Sep 15, 2026 •

Copy link
Copy Markdown

Summary

bink CLI, cluster, and DNS images are amd64-only, blocking arm64 support for bootc-operator (bootc-dev/bootc-operator#145). None of the three cross-compile: the CLI links CGO, cluster/DNS are pure RPM installs. This builds each arch natively (ubuntu-24.04, ubuntu-24.04-arm), then assembles a v2s2 manifest list in a separate push job, mirroring bootc-operator's existing pattern (bootc-dev/bootc-operator@7b3b091).

Testing

Dispatched all three workflows with push: false on this branch from my fork. Both arches built successfully for all three images, including the CGO build. The push/manifest job itself needs GHCR write access and wasn't exercised here; its logic mirrors bootc-operator's already-merged manifest job.

How to validate

gh workflow run build-bink-image.yaml --ref bink-arm64-ci-images -f push=true
gh workflow run build-cluster-image.yaml --ref bink-arm64-ci-images -f push=true
gh workflow run build-dns-image.yaml --ref bink-arm64-ci-images -f push=true

on a fork, or merge and watch the real push run on main.

Validation result

Also tested to really push to my ghcr so I can sanity check the image manifest:

docker pull ghcr.io/zarcen/bink/dns:latest
docker pull ghcr.io/zarcen/bink/cluster:latest
docker pull ghcr.io/zarcen/bink/bink:latest

Started a local bink cluster locally on Apple Silicon (arm64) :

# loading arm64 images to podman
podman machine ssh -- podman pull ghcr.io/zarcen/bink/bink:latest
podman machine ssh -- podman pull ghcr.io/zarcen/bink/cluster:latest
podman machine ssh -- podman pull ghcr.io/zarcen/bink/dns:latest
podman machine ssh -- podman tag ghcr.io/zarcen/bink/bink:latest    ghcr.io/bootc-dev/bink/bink:latest
podman machine ssh -- podman tag ghcr.io/zarcen/bink/cluster:latest ghcr.io/bootc-dev/bink/cluster:latest
podman machine ssh -- podman tag ghcr.io/zarcen/bink/dns:latest    ghcr.io/bootc-dev/bink/dns:latest

# start cluster locally
bink cluster start --cluster-name test --api-port 0
bink cluster list
Found 1 cluster(s):

  ✓ test (1 node(s), running)
bink node list --cluster-name test
Found 1 cluster node(s):

  ✓ node1 (role: control-plane, status: running, created: 2026-09-15 16:33:43)

Out of scope

ghcr.io/bootc-dev/bink/node (the bootc VM disk image, built in integration-tests.yml's build-node-images job) is not covered here. Turns out getting it to arm64 is bigger than expected: the current bcvk to-disk step hard-requires /dev/kvm, and GitHub-hosted ubuntu-24.04-arm runners don't expose one at all. There's a promising KVM-free path via bootc-image-builder, but that means swapping the actual disk-build tool, a bigger architecture change than fits here. I will track it in a follow-up PR.

Partially addresses #140: covers the bink CLI, cluster, and DNS images


Assisted-by: AI
I'm knowledgeable in this problem domain and reviewed the diff and CI run logs carefully. Besides, to ensure the PR quality, several rounds of manually validation and test are conducted

@zarcen
zarcen force-pushed the bink-arm64-ci-images branch from 6b58b85 to e32ee21 Compare September 15, 2026 23:50
Cross-compiling doesn't work here (CGO for bink, dnf-installed RPMs
for cluster/dns), so build each arch natively and assemble a manifest
list, mirroring bootc-operator's own multi-arch pattern.

Assisted-by: AI
Signed-off-by: Wei-Chen Chen <zarcen@gmail.com>
@zarcen
zarcen force-pushed the bink-arm64-ci-images branch from e32ee21 to 6498296 Compare September 16, 2026 00:26
@zarcen
zarcen marked this pull request as ready for review September 16, 2026 00:31
@alicefr

alicefr commented Sep 17, 2026

Copy link
Copy Markdown
Collaborator

@zarcen many thanks for the work, the problem is that without the node image we cannot run the tests, and it becomes hard to validate the entire setup.

As far as it regards, the access to kvm. How does it work with emulation? Is it too slow or is this somehow acceptable?

@zarcen

zarcen commented Sep 18, 2026

Copy link
Copy Markdown
Author

@alicefr I think this PR is better to pair with the node image PR I am working on later. The issue is github's arm builder doesn't come with kvm. It is quite slow and I'm thinking if we could use BIB (the bootc-image-builder). I gave it a try and result is good. However, it means to move away from bcvk. I don't know what is the historical reason to pick and use bcvk. I'm still trying if there is a minimal change requirement for it

@alicefr

alicefr commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

@alicefr I think this PR is better to pair with the node image PR I am working on later. The issue is github's arm builder doesn't come with kvm. It is quite slow and I'm thinking if we could use BIB (the bootc-image-builder). I gave it a try and result is good. However, it means to move away from bcvk. I don't know what is the historical reason to pick and use bcvk. I'm still trying if there is a minimal change requirement for it

The main reason for using bcvk was that we can do rootless builds, does BIB run only privileged?

@zarcen

zarcen commented Sep 30, 2026

Copy link
Copy Markdown
Author

The main reason for using bcvk was that we can do rootless builds, does BIB run only privileged?

BIB does require full --privileged. Still thinking what is the most non-disruptive way to support the node image in arm

@alicefr

alicefr commented Oct 1, 2026

Copy link
Copy Markdown
Collaborator

@zarcen I would open an issue in bootc and tag @cgwalters and @jmarrero they might be able to guide you and for sure we are all interested in supporting arm image builds with rootless. If there is no other way at the moment you could use BIB for ARM


- name: Assemble manifest list
run: |
podman load -i cluster-image-amd64.tar

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we do have https://github.com/bootc-dev/actions/blob/main/build-push-image/action.yml which would make sense to use...

Hmm we need an agentic review checklist item here for "check to see if the org's actions have this"

@cgwalters

Copy link
Copy Markdown

See bootc-dev/bcvk#73 (and all the other PRs/issues it links to) but I think wrapping it in bcvk would streamline things the most.

@zarcen zarcen changed the title ci: build and publish arm64 container images natively ci(WIP): build and publish arm64 container images natively Oct 1, 2026
@zarcen
zarcen marked this pull request as draft October 1, 2026 17:20
… bootc-dev/actions

Node image disk build moves from bcvk to-disk to bootc-image-builder
for the default (non-composefs) path: bcvk boots a real VM to install
and hard-requires /dev/kvm, which GitHub-hosted ubuntu-24.04-arm
doesn't have. bootc-image-builder builds via osbuild's loop-device
pipeline instead, no VM boot needed. composefs stays on bcvk
(amd64-only): bootc-image-builder has no override to force that
backend. The copr repo enable is also fixed to translate TARGETARCH
rather than assume x86_64.

Also replaces the hand-rolled per-arch build+tar-artifact+manifest
pattern in build-bink-image.yaml, build-cluster-image.yaml, and
build-dns-image.yaml with bootc-dev/actions/{bootc-host-setup,
build-push-image}, per cgwalters' review on PR bootc-dev#139, plus a small
manifest-assembly step (kept local rather than using create-manifest,
to preserve the existing --format v2s2 requirement, which the action's
own manifest push does not support overriding).

Extends node image publishing to real multi-arch: the bootc image and
default disk image are now published as proper manifest lists
(previously the bootc image was a plain single-arch amd64 image, and
arm64 disk images were built but never published at all). composefs
disk images stay amd64-only (bcvk-only), unchanged. The per-kube-minor
node images aren't routed through build-push-image/create-manifest:
those actions' digest-artifact naming only disambiguates by arch
(container-digests-${arch}), not by kube-minor, so two kube-minor
legs on the same arch would collide on the same artifact name.

Tested: all three CLI-image workflows and all 6 build-node-images
(kube-minor x arch) legs pass end to end, including live manifest
pushes verified against a real registry. The new manifest-assembly
shell logic (both the digest-based and local-storage-based variants)
was additionally dry-run end-to-end locally against a throwaway
registry. integration-tests-k8s-versions (1.36) failed consistently
across three CI attempts with a cri-o-socket-not-ready error during
kubeadm init -- unrelated to this change (untouched job/files,
unrelated to arch or manifest logic) and not chased further here.

Assisted-by: AI
Signed-off-by: Wei-Chen Chen <zarcen@gmail.com>
@zarcen
zarcen force-pushed the bink-arm64-ci-images branch from b50550a to f565400 Compare October 1, 2026 20:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants