Skip to content

CI: cache UT C++ objects and scope the axis check to one platform - #2616

Merged
ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:ci/ut-job-budget
Oct 18, 2026
Merged

ChaoWao merged 1 commit into
hw-native-sys:mainfrom
ChaoWao:ci/ut-job-budget

Conversation

@ChaoWao

@ChaoWao ChaoWao commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Problem and behavior

The no-hardware ut job carries a 20-minute budget. Over the last 40 PR runs its macOS leg has a median of 991 s and a p90 of 1187 s — 13 seconds of headroom — and four runs were stopped at the limit (1202/1217/1223/1223 s). GitHub records a timeout-minutes expiry as conclusion: cancelled, so gh pr checks renders these as fail and they read as flaky infrastructure rather than as a job that ran out of time.

Phase breakdown of the 1202 s run on #2614's head (bc46b441), from its own log timestamps:

phase cost
checkout, Python, brew, pip install, Python UT 458 s
cmake configure 4 s
compile 1290 translation units into 326 targets 434 s
ctest 47 s
check_ut_cpp_axis.py 252 s
check_ut_cpp_stub_linkage.py + swimlane ~6 s

Two of those are avoidable: the compilation is cold on every run, and the axis check is run once per platform to establish an answer that does not vary by platform.

Implementation

Configure the C++ unit-test build with ccache as its compiler launcher, restoring and saving the directory through a new cache-ccache composite action modelled on the existing cache-pip. The launcher is passed only when ccache is on PATH, so a runner whose package manager could not serve it configures and builds exactly as before; ccache itself is an optional package on Linux and a non-fatal brew install on macOS. The step prints ccache --show-stats after the build, because a restored-but-unusable cache and a cold one otherwise produce an identical full-length build.

Run check_ut_cpp_axis.py on the Linux leg only. Which configurations a case is built for follows from the include paths this repository's CMakeLists compose, so the answer is the same on every host — and Linux is the leg that can establish it, because its build graph is the superset: a non-Linux configure drops test_thread_scheduling.cpp, whose wrapped sched_* symbols are a GNU ld feature (326 targets on macOS against 327 on Linux). The non-Linux branch prints why it skipped rather than passing silently.

check_ut_cpp_stub_linkage.py keeps running on every leg. Which archive member resolves a symbol, and whether a weak definition wins, is decided by the platform's linker rather than by this repository, so it is genuinely per-platform — and at ~14 s it is not a cost.

No timeout is raised and no step is split; #2588 already carries both.

Validation

Measured on an aarch64 Linux host with --parallel 4:

  • Full build, empty cache: 381 s, 1290 misses, leaving 567 MB on disk (84 MB compressed, which is what the cache upload moves).
  • Fresh build directory against that warm cache: 16 s at a 100% hit rate.
  • A semantic edit to src/common/worker/chip_worker.h recompiles 8 of 1290 units in 9 s.
  • ctest -LE requires_hardware on the cached objects: 327/327 passed.

The compiler-launcher/axis-check interaction was the risk worth checking, since the axis check re-executes each compile_commands.json line with -E -H: both the ccache-wrapper form and the ccache /usr/bin/c++ launcher form return rc=0 with identical 37-line -H output, so the recorded command stays usable as a preprocess command.

The edited step body was executed verbatim: configure with the launcher succeeds, RUNNER_OS=Linux runs the axis check (38.9 s), RUNNER_OS=macOS takes the skip branch, stub linkage runs in 6.8 s, swimlane reports 12 passed. pre-commit run --all-files exits 0.

Real per-step costs for the two checks come from #2588's six split-step runs, which already report them separately: axis averages 175 s on macOS against 44 s on Linux (max 229 s), while stub linkage is ~14 s on both.

What to expect in CI

The first run on this branch writes a cache entry and will not be faster — the gain appears on the second run. ccache --show-stats in the build step is the acceptance signal: a hit rate near zero on a re-run means the key stopped matching.

🤖 Generated with Claude Code

The no-hardware UT job compiles 1290 translation units from scratch on
every run and then reads every one of their include trees twice, once per
platform. On macOS that is 434 s of compilation and 175 s of axis check
inside a 20-minute budget, and four runs have been stopped at the limit.

Configure the C++ unit-test build with ccache as its compiler launcher,
restoring and saving the directory through a new cache-ccache composite
action. The launcher is passed only when ccache is on PATH, so a runner
whose package manager cannot serve it configures and builds as before,
and the step prints the hit rate because a restored-but-unusable cache
and a cold one otherwise produce the same full-length build.

Run check_ut_cpp_axis.py on the Linux leg alone. Which configurations a
case is built for follows from the include paths the CMakeLists compose,
so it is the same answer on every host, and the Linux build graph is the
superset: a non-Linux configure drops test_thread_scheduling.cpp, whose
wrapped sched_* symbols are a GNU ld feature. check_ut_cpp_stub_linkage.py
keeps running everywhere, because which archive member resolves a symbol,
and whether a weak definition wins, is the platform linker's answer.

Measured on this aarch64 host: a full build with an empty cache takes
381 s and leaves 567 MB (84 MB compressed); a fresh build directory
against that warm cache takes 16 s at a 100% hit rate, and an edit to
chip_worker.h recompiles 8 of the 1290 units. ctest reports 327/327 on
the cached objects.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 0fca36c3-17c9-4d67-8812-32f5df769999
📥 Commits

Reviewing files that changed from the base of the PR and between 8b46679 and 7914e08.

📒 Files selected for processing: 4
  • .github/actions/cache-ccache/action.yml
  • .github/workflows/_ut-no-hardware.yml
  • docs/ci.md
  • docs/testing.md

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

The no-hardware C++ test workflow adds optional ccache support and uses it for builds when available. It runs the configuration-axis check on Linux only and keeps the stub-linkage check on every platform. CI and testing documentation describe these settings.

Changes: C++ build caching

Layer / File(s) Summary
Define ccache setup and cache
.github/actions/cache-ccache/action.yml
The composite action sets the ccache directory, compression, and size limit. It caches ~/.ccache using OS- and architecture-based keys.
Use ccache in the test workflow
.github/workflows/_ut-no-hardware.yml, docs/ci.md
The workflow installs ccache when available, adds caching for the GitHub setup variant, and uses it as the CMake compiler launcher when present. The CI documentation describes cache use, statistics, and self-hosted runner behavior.

Platform-specific C++ checks

Layer / File(s) Summary
Set platform scope for C++ checks
.github/workflows/_ut-no-hardware.yml, docs/ci.md, docs/testing.md
The configuration-axis check runs on Linux only. The stub-linkage check remains enabled on every platform. The documentation describes the platform-specific build graph and check scope.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 7914e

The C++ test job can build without ccache if installation fails. No change is needed before merging beyond normal checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Title check Passed The title clearly summarizes the two main changes: caching C++ unit-test objects and limiting the axis check to one platform.
Description check Passed The description directly explains the timeout problem, ccache implementation, platform-specific axis-check scope, validation results, and expected CI behavior.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit checks the cache at dawn
Compiler crumbs are stored and drawn
On Linux, axes take their turn
All platforms check links in turn
The workflow hops along its way
With cached builds to greet the day

Comment @coderabbitai help to get the list of available commands.

@ChaoWao
ChaoWao merged commit 937c57e into hw-native-sys:main Oct 18, 2026
37 checks passed
@ChaoWao
ChaoWao deleted the ci/ut-job-budget branch October 18, 2026 07:25
@ChaoWao
ChaoWao restored the ci/ut-job-budget branch October 18, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant