deploy: Pull bound images before staging via composefs mount

[cgwalters]

We were still staging a new bootloader entry even when we failed to pull a LBI. Switch to only doing an image pull and not a deployment, then pulling referenced LBIs from that.

Closes: #2013

[gursewak1997]

Needs cargo fmt

[jmarrero]

lgtm

[cgwalters]

: error: Switching: Switching (ostree): Staging: Mounting ostree commit b579414cd3d4d034ac8c9377579de4033025dfefc637bfdab95aa5ec8b9ddb5b: composefs_fsmount: Block device required (os error 15)

Looks legit

[cgwalters]

OK, this one should be good to go now

[cgwalters]

Hmm, hitting #2340 on this on centos-9.

[cgwalters]

OK this rolls in the fix for that

[cgwalters]

Looks like there are some potential latent issues with general tests on c9s

[cgwalters]

I don't yet have an idea of what's going on with the shadow generator here, looking

[cgwalters]

This took SO MUCH time to figure out

[jeckersb]

Oof that's a rough one!

[cgwalters-bot]

CentOS Stream 9 composefs/virtiofs investigation

Summary

The affected CI configuration is:

centos-9, composefs, ext4, systemd, UKI, sealed

The visible last line in failed jobs was often:

> mkfs.fat /dev/vdb2 -n EFI-SYSTEM

Trace logging from a preserved local hang proved that both mkfs.ext4 and
mkfs.fat exited successfully. The actual stall began immediately afterward,
during the direct composefs pull from bcvk's containers-storage additional
image store (AIS).

Neither serializing composefs layer work nor locking bootc's current mappings
eliminated the stall. The only reliable workaround tested was exporting the
source to an OCI layout and importing it into guest-local containers storage
before running bootc.

Full investigation details

Unless noted otherwise, evidence paths are relative to the
pr-2290-repro worktree root, not to this document's target/ directory.

CI observations

• PR: deploy: Pull bound images before staging via composefs mount #2290
• Commit: a8df21a8ad3e82902ae60b2ea98e328c3f4fb6dc
• Run: https://github.com/bootc-dev/bootc/actions/runs/33818853327
• Exact tuple had historically been excluded from CI.
• No full successful run of this exact tuple was found.
• PR Add support for multiple composefs= digests #2248 also exercised this newly enabled tuple and encountered a similar
  base-disk installation failure.

Representative PR #2290 attempts:

Attempt	Job	Installation	Later result
1	100864735896	Completed	Upgrade reboot failed
2	101015633516	Stalled during hidden post-mkfs work	Six-hour cancellation
3	101141123861	Completed	Upgrade reboot failed
4	101951476627	Stalled during hidden post-mkfs work	Six-hour cancellation

The install-stage failure is intermittent. The complete job has not been shown
to be flaky-successful: attempts that completed installation encountered a
separate sealed-upgrade failure described below.

The key run used bcvk 0.19.0. Updating from local bcvk 0.18.0 to 0.19.0 changed
local incidence but did not fix the issue; stock 0.19.0 subsequently reproduced
the stall.

bcvk topology

host rootless containers/storage
  └─ bcvk outer Podman container
       ├─ virtiofsd: source image as guest rootfs
       ├─ virtiofsd: host containers/storage as read-only AIS
       └─ QEMU
            └─ CentOS Stream 9 installer guest
                 └─ privileged inner Podman container
                      └─ bootc install to-disk
                           └─ composefs direct pull via AIS

The installer sets:

STORAGE_OPTS=additionalimagestore=/run/virtiofs-mnt-hoststorage/

Both virtiofsd exports normally use --cache=never --allow-mmap. Composefs
uses containers-image-proxy/skopeo to read blobs and overlay reconstruction
metadata from the host store while writing its repository to the target disk.

Preserved hang evidence

The strongest capture is under:

target/pr2290-debug/hang-control-004/

It shows:

• mkfs.fat returned ExitStatus(0).

• /dev/vdb2 contained a valid VFAT filesystem labeled EFI-SYSTEM.

• Composefs repository initialization and direct pull started afterward.

• Two bootc Tokio workers were in uninterruptible D state:

  folio_wait_bit_common
  filemap_fault
  __handle_mm_fault
  asm_exc_page_fault
  

• Skopeo had an AIS-backed overlay-layers/...tar-split.gz open.

• udevadm settle --timeout=5 returned success.

• /dev/vdb2 had no holders and no in-flight I/O.

• No composefs mount or loop device existed yet.

• The guest still had about 2.2 GiB available memory and no observed OOM.

This refutes ESP formatting, partition scanning, udev settlement, and RHEL 9
composefs loopification as the immediate blocking operation.

The stack proves a file-backed page fault, but not which mapping faulted. The
blocked threads belonged to bootc, not skopeo. The mapped object could be
bootc's ELF/shared libraries backed through the image store; the capture lacked
the fault address and /proc/<pid>/maps needed to prove that.

No useful kernel error was captured. Preserved host dmesg and guest journal
files were empty; there was no recorded OOM, FUSE error, I/O timeout, soft
lockup, or filesystem error. This is absence of diagnostics, not proof that the
kernel or virtiofs is correct.

In a later live stall, QEMU remained runnable and consumed CPU while both
virtiofsd processes slept. The guest stopped accepting SSH, and even an
in-guest RuntimeMaxSec failed to recover the run. Host cgroups were not frozen
or OOM-throttled.

Experiments

True composefs concurrency

Composefs-rs v0.9 has separate semaphores for repository object writes and
layer fetches. An experimental patch controlled both with
BOOTC_COMPOSEFS_PULL_CONCURRENCY.

Mode	Passed	Direct-pull stalls
Default	8	2
Fetch/write concurrency 1	9	1

The serialized stall had confirmed activation. Fisher's exact test was
p = 1.0; serialization does not eliminate the deadlock and the observed rate
difference is not meaningful.

Evidence is under the sibling worktree:

../pr-2290-composefs-concurrency/target/pr2290-composefs-concurrency/

Locking bootc mappings

An experimental branch calls mlockall(MCL_CURRENT) after detecting direct
virtiofs execution or when forced. Valid treatment runs used an unlimited
memlock limit and confirmed approximately 49.5 MiB in VmLck before direct
pull.

Mode	Passed	Direct-pull stalls
Mlock treatment	7	3

Mlock itself completed in 6–9 ms in focused tests. Since three fully activated
treatments still stalled, locking bootc's current mappings is not a fix.

• Branch: https://github.com/cgwalters-bot/bootc/tree/experiment/pr-2290-virtiofs-mlock-20260908
• Commit: f86b31adc850a6a9c8c092e43d33ee356298886b
• Logs: target/pr2290-debug/mlock-experiment/

Plain ephemeral/manual path

A plain bcvk ephemeral run -K, followed by bcvk ephemeral ssh, mounted the
same host-storage export and exercised:

• bounded raw reads and gzip -t of the implicated tar-split metadata;
• AIS skopeo inspect;
• AIS-to-guest-local image copy;
• the privileged inner-container bootc install to-disk command.

That run completed. The tar-split file read in milliseconds, the image copy
took about 14 seconds, and the manual install completed in about 14 seconds.
This proves basic AIS access is not deterministically broken, but the preceding
reads likely warmed caches and one pass does not exclude an intermittent race.

Summary: target/pr2290-debug/ephemeral-004/result-summary.txt.

Virtiofs cache modes

Temporary bcvk instrumentation varied rootfs and AIS cache modes independently.
No mode was a reliable fix. Repeated configurations alternated among success,
direct-pull stalls, and explicit composefs finalization errors such as:

Enabling verity on tmpfile: Input/output error
Opening temp file in objects directory: Read-only file system

The sample was not a controlled statistical matrix, so cache mode remains a
possible trigger rather than an established cause.

Guest-local OCI staging

The only reliable workaround was an opt-in bcvk prototype that:

1. exports the host source image to an OCI layout with preserved digests;
2. shares only that layout with the installer VM;
3. imports it into guest-local containers storage;
4. invokes bootc without the host overlay AIS.

At least ten controlled staged installs, plus two independent review runs,
completed successfully. Export and import preserved the manifest digest, and
the installed disk contained a UKI signed by the expected test key.

Prototype: /tmp/opencode/bcvk-019-plain/.

This is a workaround, not proof of the lower-level cause. It adds full-image
I/O and temporary space requirements and does not promise preservation of
external signature metadata or multi-platform manifest lists.

Separate sealed-upgrade failure

When installation completes, CentOS 9 cannot use bcvk's normal
--bind-storage-ro path. The in-guest upgrade fallback in tap.nu hardcodes:

seal_state=unsealed

It generates an unsigned replacement UKI. Secure Boot then reports:

Error loading ...efi: Access denied
BdsDxe: No bootable option or device was found.

This deterministic reboot failure is separate from the intermittent direct-pull
stall.

Timeout and cleanup lessons

• timeout --signal=KILL against the bcvk client is insufficient: detached
  Podman/conmon/QEMU/virtiofsd descendants can survive.
• In-guest systemd deadlines are not reliable once the guest becomes broadly
  unresponsive.
• Run bcvk under a recorded process group with a unique container name/label.
• Monitor from the host, preserve evidence before intervention, remove only the
  exact container, then kill and reap the exact launcher process group.
• Never use bcvk ephemeral rm-all -f during a measured run.
• Live stderr tee instrumentation is needed because bcvk normally captures
  bootc stderr; otherwise mkfs.fat remains the misleading last visible line.

Current conclusion

The failure is intermittent and was observed during bcvk to-disk while a
CentOS Stream 9 guest performed a composefs direct pull through a
virtiofs-backed containers-storage AIS. One successful manual ephemeral path
suggests that to-disk orchestration or timing contributes, but does not prove
that it is the cause. True layer serialization, locking bootc's current
mappings, bcvk 0.19, and simple virtiofs cache-mode changes did not fix it.

The evidence is consistent with a guest virtiofs/page-cache failure under this
workload, but lacks the guest fault address, complete kernel logs, serial
console, and FUSE request state needed to identify the exact kernel or server
defect.

Recommended next steps:

1. Keep guest-local OCI staging as the practical CI workaround.
2. Reproduce with serial console and guest kernel logging enabled from boot.
3. Capture fault address, /proc/<pid>/maps, FUSE queue state, and blocked-task
   output before SSH becomes unavailable.
4. Preserve and inspect QEMU and both virtiofsd process/thread states.
5. Track the separate unsigned-UKI upgrade issue independently.