Copy the datadog-agent checkout with cp instead of installing rsync

[ai-collaboration-app]

Every job in this repository has failed since 2026-09-07 in the first CI step, on both architectures. Not a CVE fix, but it blocks every candidate here, including the CVE-2026-84445 grpc bump in #278.

What is broken

E: Release file for http://deb.debian.org/debian-security/dists/bullseye-security/InRelease is expired
Process completed with exit code 100

The prebuild images datadog_build_system-probe_x64 / _arm64 at 61b4ad67 are Debian 11 bullseye, which is EOL. Its bullseye-security Release file expired on 2026-09-07, so apt-get update exits 100 and nothing can be installed from those images.

That step only ran apt to get rsync, which the image lacks, purely to copy the checkout into the work directory.

Reproduced inside quay.io/stackstate/datadog_build_system-probe_x64:61b4ad67 rather than read off the log: apt-get update exits 100 on that single expired Release file, and /etc/apt/sources.list points at deb.debian.org, which no longer carries bullseye.

Why cp -au

Equivalent for this use and needs no network. Both are non-deleting copies of the directory contents that skip files no newer than the destination, so re-running stays cheap; cp -a additionally preserves hard links, which rsync -a does not.

Verified in the same image (GNU coreutils 8.32): identical file count and modes, second run copies nothing.

Scope

Last green CI was 2026-09-07T09:35Z. #277 failed the same way on 2026-09-08, a day before #278 existed, so this is pre-existing and unrelated to either.

Pinning the prebuild images to a supported Debian release is the durable fix and belongs with whoever owns those images. This only removes the dependency on installing packages inside them, which is what unblocks the queue today.

[ai-collaboration-app]

Independent CVE review of 711b2ac55d1f9fc458d28570021ee9f205244908 (supervised remediation run, dev scan 34330594840). Verdict: needs correction — one blocking finding.

cp -au is not equivalent to the rsync it replaces on a reused workdir. On a fresh destination it is faithful for permissions, symlinks, timestamps, hidden files and non-delete behaviour, and it additionally preserves hard links. But when a source path changes from a file or symlink into a directory, rsync replaces it while GNU cp 8.32 exits 1. Dependency-update runs reuse this workdir; hosted CI starts fresh — so the green run cannot surface this failure. Please correct the reused-workdir semantics and rerun CI.

Confirmed good, and worth keeping: the CI diagnosis is sound, and this PR is on the critical path since the repo has been red on every branch since 2026-09-07. The published amd64 and arm64 RPM databases contain the exact fixed versions of all nine packages. No suppressions were added; commits and scope comply; no YAML or workflow changes.

Non-blocking: the PR description is longer than house style; the EOL Debian bullseye prebuild images remain a separate BCI finding worth its own ticket.

Human review and merge remain mandatory.

[ai-collaboration-app]

Corrected in 901935ce. The finding reproduces, it is wider than reported, and the fix is now measured against real rsync rather than assumed equivalent. rsync was not reintroduced.

The finding reproduces, and it wedges the workdir permanently

WORKDIR is datadog-agent-workdir, constant; SOURCEDIR is prebuild_artifacts/checkout/$DEPENDENCY_VERSION, version-scoped. So a dependency bump copies a different tree into the same workdir — exactly when a path changes type. --clean doesn't remove the workdir either, so a developer keeps it across versions indefinitely.

Replayed two consecutive runs over one workdir in datadog_build_system-probe_x64:61b4ad67 (coreutils 8.32), upstream turning a file into a directory and a symlink into a directory:

cp: cannot overwrite non-directory '/tmp/workdir/./pkg/ebpf/legacy_helper' with directory
cp: cannot overwrite non-directory '/tmp/workdir/./pkg/tools_shim' with directory

Worth adding to the finding: every subsequent run then fails the same way in 173ms. The workdir stays wedged until someone deletes it by hand. Hosted CI starts from an empty workspace, which is why 34335257004 was green.

Wider than the two hard errors

Four more transitions were wrong, silently. With the destination no older, cp -u left a stale file where the source had become a symlink, and a stale symlink where it had become a file — exit 0 both times. That is a build against a tree that does not match the source, which is worse than a crash.

rsync's actual contract, measured

rsync is absent from both prebuild images, so I installed rsync 3.2.7 from a working channel and ran the same matrix rather than reasoning about it:

Transition	rsync 3.2.7	cp -au (711b2ac)	now
file → directory	replaces	exit 1	replaces
symlink → directory	replaces	exit 1	replaces
dangling symlink → directory	replaces	exit 1	replaces
symlink-to-dir → real directory	replaces, no write-through	exit 1	replaces, no write-through
file → symlink	replaces	stale file, exit 0	replaces
symlink → file	replaces	stale symlink, exit 0	replaces
directory → file	exit 23 "could not make way for"	stale dir, exit 0	refuses, names the path
directory → symlink	exit 23 "could not make way for"	stale dir, exit 0	refuses, names the path

The last two rows are why this is not "replace anything that differs". rsync deliberately refuses to replace a directory with a non-directory, and that refusal is protective here: rm -rfing such a destination would delete generated eBPF output living under it. So the fix reproduces the refusal with a diagnostic naming the path, instead of either silently keeping a stale tree or destroying build output.

Consequently no directory is ever deleted. rm -f rather than rm -rf enforces that in the code: a directory would fail rather than be removed. Paths absent from SOURCEDIR are untouched, so the non-delete behaviour git ls-files --others --ignored depends on is preserved.

Evidence

The harness extracts the copy block from the committed script, so the shipped code is what runs — not a transcription of it.

Type-change matrix — 15 pass on 901935ce, and 10 of those fail on 711b2ac5:

  PASS  file    -> directory                  PASS  directory -> file    refused
  PASS  symlink -> directory                  PASS  directory -> symlink refused
  PASS  dangling symlink -> directory         PASS  no write-through into an outside directory
  PASS  symlink-to-dir -> real directory      PASS  non-delete: generated artifact preserved
  PASS  file    -> symlink                    PASS  -u: newer destination not overwritten
  PASS  symlink -> file                       PASS  -u: older destination refreshed
  PASS  nested: a/b file -> a/b/c dir          PASS  fresh-dest fidelity (perms, symlink, hidden, hard links)
                                              PASS  re-run on populated workdir exits 0
  === 15 passed, 0 failed ===        (on 711b2ac5: 5 passed, 10 failed)

Reused-workdir reproduction at scale — 21,646 files, run 1 on v1, generated usm.o/tracer.o seeded into the workdir, run 2 on v2 where legacy_helper becomes a directory and tools_shim goes symlink → directory:

	901935ce	711b2ac5
run 2 exit	0	1
run 3 (re-run) exit	0	1
assertions	10/10 pass	5/10 fail

Both generated artifacts survive, source content refreshes to v2, symlinks stay symlinks, and mode 755 is preserved.

Two things the reviewer could not have seen

• Log volume. Under set -ex the loop emitted ~14 trace lines per path — 28,021 lines for 2,000 files, so ~300k on the real tree, burying the build log. xtrace is now off for the loop only: 5 lines. The cp itself is still traced, and the refusal diagnostic still goes to stderr.
• Cost. The walk adds roughly 1.2–1.7s per run at 21,646 files, against a multi-minute eBPF build.

Unchanged from the reviewed head

apt-get update still exits 100 in these images on the expired bullseye-security Release file, so installing rsync remains impossible. A BCI-based prebuild image is the deeper fix and belongs in its own ticket, as noted. The CI diagnosis, the RPM evidence and the scope of this PR are otherwise untouched — this commit only changes the copy step.

CI is running on 901935ce; I will post the dual-architecture result. Merge remains a human decision.

[ai-collaboration-app]

Fresh CI on the corrected head 901935ce: run 34345282150 — all 8 jobs green on amd64 and arm64, through signed multi-architecture publication of :901935ce.

The step this PR changes (Prebuild, generated-code check, build, and unit tests) passes on both architectures, so the copy replacement drives a real eBPF prebuild end to end.

Scope note on what CI does and does not prove. Hosted CI starts from an empty workspace, so this run exercises the fresh-destination path — it confirms no regression, but by construction it cannot reach the type-change path. That path is covered by the in-image reproduction in the previous comment, which fails on 711b2ac5 and passes on 901935ce.

Log volume held. The whole run is 17,017 log lines against 16,993 for 711b2ac5 — +24, confirming the xtrace suppression works in CI rather than only locally.

Evidence from 711b2ac5 is unchanged. The nine base-image packages in the newly built image, both architectures:

curl           8.14.1-150700.7.26.1      libopenssl3    3.5.0-150700.5.50.1
libbz2-1       1.0.8-150400.3.4.1        libusb-1_0-0   1.0.24-150400.3.6.1
libcares2      1.34.8-150000.3.29.1      openssl-3      3.5.0-150700.5.50.1
libcurl4       8.14.1-150700.7.26.1      wget           1.24.5-150700.3.12.1
libnghttp2-14  1.64.0-150700.3.6.1

Evaluator, identical on both architectures:

image-pipeline evaluator (mode: inform)
  total in-scope findings:  2
  suppressed by exception:  1
  expired:                  0
  unmanaged:                1
  unused exceptions:        0
Unmanaged findings:
  - CVE-2026-84445 [HIGH] pkg:golang/google.golang.org/grpc@v1.83.1 [grype,trivy]

Still 37 of 38 base-image rows cleared with the one remaining unmanaged finding being exactly what #278 fixes — unchanged, as expected, since this commit only touches the copy step.