From 19e46329262cf9324325a76577c3c7aeb693eb83 Mon Sep 17 00:00:00 2001 From: Derek Date: Thu, 3 Sep 2026 17:49:09 +1000 Subject: [PATCH 1/4] fix(developer-rust): derive CARGO_HOME from the host The role hard-coded CARGO_HOME to ~/.cargo in cargo_env. On a host that relocates it, the role wrote a correct config.toml to a directory cargo never reads, and the stale file that stayed in effect named a rustc-wrapper which no longer existed. Every cargo build failed with "No such file or directory" while the converge reported success. Four occurrences were confirmed on the affected workstation, the most recent after a hand-patch had supposedly fixed it. CARGO_HOME and RUSTUP_HOME now come from the target user's own login shell, which is the environment their cargo runs in. ansible_facts.env cannot answer this - it is the connection user's environment, root under become. Output is sentinel-delimited because a login shell runs /etc/profile.d and ~/.profile first, and anything they print would otherwise land ahead of the answer. A post-condition at the end of the run reads BOTH candidate configs and fails the converge when either names a wrapper that does not resolve. Reading only the resolved one would let a wrong resolution certify itself. A config superseded by a relocated CARGO_HOME is renamed to config.toml.superseded rather than left as a trap. Also replace the global build mutex with a memory semaphore. One build at a time was correct on a 30 GB box and is the concurrency ceiling on a 246 GB one: twelve sessions queue while the machine idles, and a corpus gate parallelised from 102s to 18s spends the gain again the moment work is farmed out to agents. N = min(MemoryHigh / per-build allowance, pool/2), jobs = pool/N, both resolved at run time from the live host so a resized VM needs no re-converge. N=1 reproduces the old behaviour and is what a 32 GB host computes. Slots are one fd reopened across N lock files, so slot choice is which file is opened - no eval and no bash 4 descriptors. The build is run and waited on rather than exec'd, with fd 9 closed for the child: exec handed the lock to any daemon the build started, which then held it for its own lifetime. At saturation a build degrades to the floor job count under its own memory cap instead of proceeding unbounded, which is what the old timeout did at the exact moment contention was highest. CARGO_INCREMENTAL is left alone by default. Forcing it off makes workspace crates cacheable, but cargo never builds dependencies incrementally, so it buys less than a full-rebuild measurement suggests and costs an edit-rebuild loop the incremental it depends on. Opt in per host for build boxes and CI runners. Add zram_swap. MemoryHigh throttles by reclaim, so on a swapless host a build past its budget stalls rather than slows, and the governor now warns at converge time when it lands on one. Load local-config/vars.yml when present, tagged always. A host setting with nowhere to live is lost at the next converge - a box with a dedicated cache volume had its build pool silently relocated to the root filesystem because its cache root had only ever been passed as -e. The sccache unit now carries its own cache size and base directories. The server reads them once at startup, and a unit inherits pam_env but not /etc/profile.d, so a hand-set value in /etc/environment silently beat the role's and the running server had a ceiling nobody configured. The role restarts the server when the unit changes, only where the server is the unit's own. The prune timers carry CARGO_HOME on both systemd and launchd, since neither inherits a shell profile and the pruner reads it to find the config that names the pool. shellcheck in CI now covers the governor. --- README.md | 10 + ansible/playbooks/main.yml | 25 ++ ansible/roles/developer-rust/README.md | 11 +- .../roles/developer-rust/defaults/main.yml | 88 ++++- .../developer-rust/files/hyperi-rust-govern | 372 ++++++++++++++---- .../roles/developer-rust/tasks/governor.yml | 55 ++- ansible/roles/developer-rust/tasks/rust.yml | 312 +++++++++++++-- .../developer-rust/tasks/sccache_service.yml | 38 ++ .../hyperi-rust-cache-prune-guard.service.j2 | 3 + .../hyperi-rust-cache-prune.service.j2 | 5 + .../templates/hyperi-sccache.service.j2 | 12 + .../io.hyperi.rust-cache-prune-guard.plist.j2 | 8 + .../io.hyperi.rust-cache-prune.plist.j2 | 8 + .../templates/rust-build.slice.j2 | 17 +- .../templates/rust-governor.conf.j2 | 25 +- ansible/roles/zram_swap/README.md | 64 +++ ansible/roles/zram_swap/defaults/main.yml | 37 ++ ansible/roles/zram_swap/tasks/main.yml | 178 +++++++++ .../templates/zram-generator.conf.j2 | 10 + docs/rust-build-governor.md | 84 ++++ tools/ci/run-tests.sh | 3 +- 21 files changed, 1219 insertions(+), 146 deletions(-) create mode 100644 ansible/roles/zram_swap/README.md create mode 100644 ansible/roles/zram_swap/defaults/main.yml create mode 100644 ansible/roles/zram_swap/tasks/main.yml create mode 100644 ansible/roles/zram_swap/templates/zram-generator.conf.j2 create mode 100644 docs/rust-build-governor.md diff --git a/README.md b/README.md index a233273..87811c7 100644 --- a/README.md +++ b/README.md @@ -87,6 +87,7 @@ flowchart TD | `vpn-clients` | OpenVPN 3, WireGuard, Tunnelblick (macOS) | | `vm` | VM guest optimisations (QEMU/SPICE agents) | | `power-profile` | Sleep/idle/lid policy. `always-on` (default) or `vm`, via `-e power_profile=` | +| `zram` | Compressed-RAM swap so cgroup memory limits throttle instead of stalling. Pairs with the Rust build governor | | `arcane` | Arcane container UI, localhost-only. Off unless `-e soe_arcane_enabled=true`. Add `-e soe_arcane_long_session=true` for a year-long login | | `local-services` | Persistent local ClickHouse + Redpanda for spikes. Off unless `-e soe_local_services_enabled=true` | @@ -158,6 +159,14 @@ That matters most on a fleet machine reached over SSH as a service account, because that account's home is not the desktop. Get it wrong and the run still reports success -- the settings simply land where nobody sees them. +Settings that are true of one machine rather than of the repo -- a dedicated +cache volume, say -- go in `local-config/vars.yml` at the repo root, which is +gitignored and loaded on every run, tagged or not. A setting passed once as +`-e` and never written down is lost at the next converge: a box with a 512G +cache volume had its Rust build pool silently moved onto the root filesystem +that way. Anything in that file runs with the playbook's own authority -- root, +under `become` -- so treat it exactly as you would the playbook. + ## What Gets Installed **Default** (`./install.sh`) - a lightweight generic CLI dev base, nothing HyperI-specific: @@ -175,6 +184,7 @@ reports success -- the settings simply land where nobody sees them. - `contributor`: hyperi-ci and the tools its checks drive (semgrep, alint), gitleaks, trivy, hadolint, pip-audit, ansible-lint, pre-commit, act, and git-scrub for rewriting AI residue or a leaked secret out of git history -- gitleaks scans FULL history, so a secret removed from HEAD still fails the gate. macbash is here too: it flags the GNU-only bash constructs that break on macOS, and CONTRIBUTING.md asks for it on every shell change - `soe` / `soe-gui`: HyperI org policy: VPN clients, Claude Code, Slack, LibreOffice, RDP client, telemetry-disable, auto-updates, GNOME taskbar - `power-profile` (off by default, and deliberately not in `soe`): sleep, idle and lid policy, selected per machine. `always-on` (the default profile) never idle-suspends on mains power and does not sleep when the lid shuts -- for a repurposed laptop doing build work, or a desktop that has to answer ssh. `vm` never sleeps or suspends at all, for an unattended RDP guest that nobody can walk over and wake. Battery behaviour stays stock under `always-on`, because a machine that will not sleep in a bag cooks itself. Profiles are data files, so adding one is adding a file -- see [roles/power-profile/README.md](ansible/roles/power-profile/README.md) +- `zram` (off by default, and deliberately not in `soe`): a small compressed-RAM swap device, sized from the host's RAM and capped at 8 GiB. It exists because cgroup `MemoryHigh` throttles by reclaim, and on a swapless host the only reclaimable memory is page cache -- so a build past its budget stalls rather than slows. Pairs with the Rust build governor, which warns at converge time when it lands on a swapless host. Never restarts a running device, so a size change waits for a reboot -- see [roles/zram_swap/README.md](ansible/roles/zram_swap/README.md) - `arcane` (off by default): [Arcane](https://getarcane.app), a web UI for the containers on the box. Enable it with `-e soe_arcane_enabled=true` and you get a daemon on `http://localhost:3552` that comes back after a reboot and keeps itself updated. Works against docker-ce on Linux and colima on macOS. Bound to loopback because it holds the Docker socket, so whatever reaches that port owns the machine. Login is whatever Arcane seeds -- `arcane` / `arcane-admin` as upstream documents it. The role sets neither, and only clears the forced first-login password prompt, which it does by re-submitting that seeded password so the credentials stay unchanged. That needs the password policy relaxed to `basic` (`soe_arcane_password_policy`), because upstream's default `strong` policy rejects its own seeded password. There is still a login -- auto-login sits behind a `buildables` Go build tag that no published image is compiled with, so zero-auth is not available without building your own image. The login lasts about a day by default, which is a prompt every morning on a dev box -- `-e soe_arcane_long_session=true` stretches it to a year from each login. It is off by default because Arcane holds the Docker socket, and it takes one log-out and log-in to take effect, since the session expiry is stamped at login - `local-services` (off by default): a persistent local ClickHouse and Redpanda for ad-hoc work -- somewhere to poke at a query or hand-feed a topic without waiting for a suite to build. Enable with `-e soe_local_services_enabled=true`. Deployed **stopped**: `restart: no`, so a reboot leaves them down and they cost nothing until `local-services up`, which pulls latest and takes seconds. Both capped at 1GB and bound to loopback. They are spike instances -- integration and e2e suites create and tear down their own containers, because a shared daemon makes a suite non-hermetic and order-dependent diff --git a/ansible/playbooks/main.yml b/ansible/playbooks/main.yml index b6baac0..52684f5 100644 --- a/ansible/playbooks/main.yml +++ b/ansible/playbooks/main.yml @@ -13,6 +13,22 @@ gather_facts: true pre_tasks: + # Per-machine settings that cannot live in the repo, because the repo is + # cloned onto every machine. `local-config/` is already gitignored. + # + # This exists because a host setting with no home is silently lost: a box + # with a dedicated cache volume that passed `-e rust_cache_root=/cache` once + # gets the DEFAULT on the next converge, and its build pool moves off the + # volume it was put on without a word. + # `always`, because a tagged run needs these too -- and a tagged run is + # exactly when a missing host setting does its damage, since a targeted + # re-converge is the common case. + - name: Load per-machine overrides if this host has any + ansible.builtin.include_vars: + file: "{{ item }}" + loop: "{{ q('first_found', [playbook_dir ~ '/../../local-config/vars.yml'], errors='ignore') }}" + tags: ['always'] + - name: Display system information ansible.builtin.debug: msg: | @@ -306,6 +322,15 @@ become: false tags: ['power-profile', 'never'] + # zram swap, so cgroup memory limits throttle instead of stalling. Opt-in + # and per-machine like power-profile: it is a host-wide memory policy, and + # a box that already has disk swap wants different numbers. Pairs with + # developer-rust's build governor, whose MemoryHigh has nothing to reclaim + # on a swapless host. + - role: zram_swap + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + tags: ['zram', 'never'] + # System cleanup — always runs at the end. - role: system_cleanup become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" diff --git a/ansible/roles/developer-rust/README.md b/ansible/roles/developer-rust/README.md index 48673f7..285d876 100644 --- a/ansible/roles/developer-rust/README.md +++ b/ansible/roles/developer-rust/README.md @@ -134,12 +134,21 @@ developer installed is their call. `build.build-dir` is stable from Rust 1.91. On an older toolchain the setup tool says so and leaves the per-project layout alone, so the default stays safe. +## Bounding concurrent builds + +`hyperi-rust-govern` is installed as `~/.local/bin/cargo`, ahead of the real +cargo on PATH, so a developer or an agent who knows none of this runs +`cargo build` and is governed: the build lands in `rust-build.slice` and holds +one of N slots, where N is derived from the slice's memory budget. How N is +chosen, what happens at saturation, and why it needs the `zram_swap` role are in +[docs/rust-build-governor.md](../../../docs/rust-build-governor.md). + ## SSoT This role is the source of truth for the global Cargo config. It was `hyperi-ci`'s `scripts/setup-rust-dev.py` until 2026-07-17. -That script is not what got ported. A review found it could not be adopted: +That script is not what got ported. It could not be adopted: - It moved `target/` directories across filesystems with `os.rename`, which raises `EXDEV` and has no copy fallback. Proven on a real host where diff --git a/ansible/roles/developer-rust/defaults/main.yml b/ansible/roles/developer-rust/defaults/main.yml index 0178593..88f3c93 100644 --- a/ansible/roles/developer-rust/defaults/main.yml +++ b/ansible/roles/developer-rust/defaults/main.yml @@ -5,6 +5,35 @@ # current rustup-init together with the checksum published beside it, and # rustup then tracks stable on its own. +# --------------------------------------------------------------------------- +# Toolchain location +# --------------------------------------------------------------------------- +# Empty means "ask the host". The role probes the target user's own login +# environment for CARGO_HOME/RUSTUP_HOME and falls back to ~/.cargo and +# ~/.rustup, which is what cargo and rustup themselves do. +# +# Hard-coding ~/.cargo here is what broke this: a host that relocates +# CARGO_HOME (via /etc/environment, a profile drop-in, whatever) got a correct +# config.toml written to a directory cargo never reads, and the stale file that +# stayed in effect named a rustc-wrapper that no longer existed. Every build +# failed and the converge reported success. Set these only to override the +# host's own answer. +rust_cargo_home: "" +rust_rustup_home: "" + +# Post-condition check at the end of the toolchain run: fail the converge when +# the config cargo will actually read names a rustc-wrapper that does not +# resolve. Off only for a box where you know the wrapper is provisioned after +# this role runs. +rust_verify_wrapper: true + +# On a host that relocates CARGO_HOME, a config.toml left in ~/.cargo is inert +# while the relocation holds and becomes live the moment it does not -- which is +# how a file naming a long-gone rustc-wrapper stayed in effect here. Renamed +# rather than deleted, matching what hyperi-rust-setup does when it takes over +# an existing config, so nothing is destroyed on a host we do not control. +rust_retire_superseded_config: true + # --------------------------------------------------------------------------- # Build cache caps # --------------------------------------------------------------------------- @@ -80,14 +109,49 @@ rust_sccache_service_manage: true # shim, the slice and the config. rust_governor_enabled: true -# One rust build at a time per user; later invocations wait for the lock. -rust_governor_serialize: true - -# A daemon started inside a governed build inherits the lock and holds it for -# its own lifetime, which would otherwise hang every later cargo command. After -# this many seconds the build proceeds unserialised and says so. +# How many rust builds may be resident at once, per user. A memory semaphore, +# not a queue length: memory is spent per CRATE rather than per job, so the +# thing worth bounding is concurrent BUILDS and the bound is the slice's memory +# budget divided by what one build costs. +# +# auto derive from this host's RAM (see rust_governor_build_allowance_gb) +# pin the count; 1 is a global mutex, which is what this used to be +# 0 no semaphore at all +# +# `auto` on a 32 GB host computes 1, which is the check that the model +# reproduces the behaviour that is known to work. +rust_governor_slots: auto + +# What one build is assumed to cost, in GB: the largest single rustc plus +# headroom. Derived from an 11.6 GB peak on one workspace, so it is a starting +# point rather than a constant -- a codebase whose memory scales with the job +# count instead of with one huge crate will want a different number. +rust_governor_build_allowance_gb: 14 + +# Seconds to wait for a slot. On expiry the build proceeds at the FLOOR job +# count, still inside the slice -- it degrades, it is not released. Waiting the +# full time and then building unbounded would drop the limit at exactly the +# moment contention is highest. rust_governor_lock_wait_seconds: 1800 +# Whether governed builds run with CARGO_INCREMENTAL=0, so sccache can cache +# them. sccache refuses to cache any rustc call carrying -C incremental. +# +# OFF by default, because the win is narrower than it first looks. Cargo never +# builds DEPENDENCIES incrementally, so those were always cacheable; what this +# buys is caching of WORKSPACE crates, and it pays for it by losing incremental +# on those same crates. That is a straight regression for an edit-rebuild loop, +# worst on a large crate that compiles as a single rustc -- edit it and sccache +# misses on the changed source with no incremental left to fall back on. +# +# Turn it ON for a build box, a CI runner, or a workstation running many +# sessions against the same workspaces, where builds start from clean trees and +# there is no incremental state to lose. +# +# Note for anyone opting out by hand: setting CARGO_INCREMENTAL=1 makes sccache +# REFUSE the build outright rather than fall back to compiling. +rust_governor_no_incremental: false + # Percentages of the HOST's physical RAM (systemd resolves them all against # RAM, the swap cap included), so the same numbers ship everywhere. High # throttles and reclaims; Max kills the build -- the build dies, never the @@ -101,11 +165,19 @@ rust_governor_swap_max_pct: 25 rust_governor_cpu_weight: 30 # Cores held back from builds on any host with 4 or more; smaller hosts keep -# every core. Also caps CARGO_BUILD_JOBS, because the cgroup quota is -# invisible to rustc. +# every core. Taken off the top BEFORE the remainder is divided between +# concurrent builds, so the desktop keeps its share at every slot count. rust_governor_cpu_reserve_cores: 2 # macOS has no cgroups; builds run under this taskpolicy QoS clamp instead. # `utility` sits below the desktop without background's disk-IO throttle and # E-core pinning. rust_governor_macos_qos: utility + +# The user manager's runtime directory, for every `systemctl --user` call the +# role makes. Defined once because the fallback is wrong on a fleet box: +# actual_user_uid is set by the developer role's init, which does not run under +# a --tags run, so every call then falls back to the CONNECTION user's uid. +# When that is fixed it is fixed here, in one place. +developer_rust_user_env: + XDG_RUNTIME_DIR: "/run/user/{{ actual_user_uid | default(ansible_facts['user_uid']) }}" diff --git a/ansible/roles/developer-rust/files/hyperi-rust-govern b/ansible/roles/developer-rust/files/hyperi-rust-govern index efdc4d8..de8637b 100644 --- a/ansible/roles/developer-rust/files/hyperi-rust-govern +++ b/ansible/roles/developer-rust/files/hyperi-rust-govern @@ -1,31 +1,52 @@ #!/usr/bin/env bash -# Governs Rust builds so they cannot take the machine down: one build at a -# time per user, the whole process tree inside the rust-build.slice budget, +# Governs Rust builds so they cannot take the machine down: at most N builds at +# a time per user, the whole process tree inside the rust-build.slice budget, # and priority below the desktop. # -# Installed as ~/.local/bin/cargo, which sits ahead of ~/.cargo/bin on the -# SOE PATH, so cargo and everything it spawns is captured. sccache-hosted -# compiles are captured separately: hyperi-sccache.service carries -# Slice=rust-build.slice. +# N is a MEMORY semaphore, not a queue length. Memory is spent per CRATE, not +# per job -- one enormous crate compiles as a single rustc however high -j goes +# -- so the bound is the slice's memory budget divided by what one build costs. +# A small host computes N=1, which is a global mutex. +# +# Installed as ~/.local/bin/cargo, which sits ahead of the cargo bin directory +# on the SOE PATH, so cargo and everything it spawns is captured. +# sccache-hosted compiles are captured separately: hyperi-sccache.service +# carries Slice=rust-build.slice. # # HYPERI_RUST_GOVERNOR=off cargo build # bypass one invocation # hyperi-rust-govern [args...] # govern any other command # -# cargo run holds the build lock for the program's whole lifetime, because -# cargo stays resident wrapping the binary. Use the bypass for long-running -# `cargo run` services. +# cargo run holds its slot for the program's whole lifetime, because cargo +# stays resident wrapping the binary. Under a semaphore that costs 1/N of +# capacity rather than everything, but the bypass is still right for a +# long-running `cargo run` service. # # Written for bash 3.2: stock macOS runs this. set -u +# HOME is unset in a system unit, in a container whose uid has no passwd entry, +# and under env -i. Unset, set -u aborts at the first $HOME below before any +# tool runs; empty degrades to an unusable slot directory and an ungoverned +# build, which is the right failure. +HOME="${HOME-}" + CONF="${XDG_CONFIG_HOME:-$HOME/.config}/hyperi/rust-governor.conf" # shellcheck source=/dev/null [ -r "$CONF" ] && . "$CONF" -SERIALIZE="${HYPERI_RUST_GOVERN_SERIALIZE:-1}" +# auto = derive from this host's RAM. A positive integer pins the count. 0/off +# disables the semaphore entirely. 1 is the old global mutex. +SLOTS="${HYPERI_RUST_GOVERN_SLOTS:-auto}" RESERVE="${HYPERI_RUST_GOVERN_CPU_RESERVE:-2}" QOS="${HYPERI_RUST_GOVERN_MACOS_QOS:-utility}" LOCK_WAIT="${HYPERI_RUST_GOVERN_LOCK_WAIT:-1800}" +# Must match rust-build.slice's MemoryHigh. Both resolve a percentage of the +# same memory total -- systemd against MemTotal, this script against the cgroup +# limit where one is set and MemTotal otherwise -- so on a bare host they agree. +MEM_PCT="${HYPERI_RUST_GOVERN_MEMORY_HIGH_PCT:-50}" +# What one build costs: the largest single rustc plus headroom. Derived from a +# 11.6 GB peak on one workspace, so it is a starting point, not a constant. +ALLOWANCE_GB="${HYPERI_RUST_GOVERN_BUILD_ALLOWANCE_GB:-14}" TOOL="$(basename "$0")" if [ "$TOOL" = "hyperi-rust-govern" ]; then @@ -80,82 +101,238 @@ fi case "$cores" in ''|*[!0-9]*) cores=2 ;; esac + +# Leading zeros are stripped before every numeric guard: `08` passes a digit +# test and then aborts bash arithmetic as an invalid octal constant, which +# under set -u leaves N empty and silently removes the protection. +RESERVE="${RESERVE#"${RESERVE%%[!0]*}"}" +[ -z "$RESERVE" ] && RESERVE=0 case "$RESERVE" in - ''|*[!0-9]*) RESERVE=2 ;; + *[!0-9]*) RESERVE=2 ;; esac case "$LOCK_WAIT" in ''|*[!0-9]*) LOCK_WAIT=1800 ;; esac +MEM_PCT="${MEM_PCT#"${MEM_PCT%%[!0]*}"}" +[ -z "$MEM_PCT" ] && MEM_PCT=0 +case "$MEM_PCT" in + *[!0-9]*) MEM_PCT=50 ;; +esac +ALLOWANCE_GB="${ALLOWANCE_GB#"${ALLOWANCE_GB%%[!0]*}"}" +case "$ALLOWANCE_GB" in + ''|*[!0-9]*) ALLOWANCE_GB=14 ;; +esac +# taskpolicy's clamp names. That path has no fallback, so anything else would +# make every cargo on a Mac fail with a usage error. +case "$QOS" in + background|utility|maintenance|default) ;; + *) QOS=utility ;; +esac -# The cgroup quota is invisible to rustc, which sizes its parallelism off the -# visible core count and thrashes inside the cap -- so the job count is -# lowered to match. A box under 4 cores keeps them all. -jobs="$cores" +# Cores builds may use at all. The reserve is a property of the HOST -- keep +# the desktop responsive -- so it comes off the top, before anything is divided +# between concurrent builds. A box under 4 cores keeps them all. +pool="$cores" if [ "$cores" -ge 4 ]; then - jobs=$((cores - RESERVE)) + pool=$((cores - RESERVE)) fi -[ "$jobs" -lt 1 ] && jobs=1 -export CARGO_BUILD_JOBS="${CARGO_BUILD_JOBS:-$jobs}" +[ "$pool" -lt 1 ] && pool=1 + +# Read the memory total rather than asking systemd. `systemctl show` is a D-Bus +# round trip on every cargo invocation, reports `infinity` when the slice is not +# loaded, and does not exist on macOS or in a container -- all three of which +# this script has to keep working in. Reproducing systemd's own arithmetic +# against the same total gives the same answer with none of that. +# +# The cgroup limit is preferred where one is set, because /proc/meminfo reports +# the HOST's memory inside a container: a 4 GB container on a 256 GB box would +# otherwise admit eight builds against an allowance it does not have. +mem_total_kb() { + limit="$(cat /sys/fs/cgroup/memory.max 2>/dev/null || true)" + case "$limit" in + ''|max|*[!0-9]*) ;; + *) echo $((limit / 1024)); return 0 ;; + esac + if [ -r /proc/meminfo ]; then + awk '/^MemTotal:/ { print $2; exit }' /proc/meminfo 2>/dev/null + return 0 + fi + if command -v sysctl >/dev/null 2>&1; then + bytes="$(sysctl -n hw.memsize 2>/dev/null)" + case "$bytes" in + ''|*[!0-9]*) return 0 ;; + esac + echo $((bytes / 1024)) + fi +} + +# N = min(memory budget / per-build allowance, pool / 2). +# +# The pool/2 term is not a safety margin: admitting more concurrent builds than +# there are cores to give each one a couple of jobs makes every build slower +# for no gain, and it is what makes the jobs arithmetic below unable to reach 0. +compute_slots() { + total_kb="$(mem_total_kb)" + case "$total_kb" in + ''|*[!0-9]*) echo 1; return 0 ;; + esac + + # Divide before multiplying: the loss is under a percent and it cannot + # overflow on a 32-bit shell. + high_kb=$(( (total_kb / 100) * MEM_PCT )) + allowance_kb=$(( ALLOWANCE_GB * 1024 * 1024 )) + n=$(( high_kb / allowance_kb )) + + n_cpu=$(( pool / 2 )) + [ "$n" -gt "$n_cpu" ] && n="$n_cpu" + [ "$n" -lt 1 ] && n=1 + echo "$n" +} + +# The same strip as the guards above, on a copy: `auto` has to survive it. +SLOTS_N="${SLOTS#"${SLOTS%%[!0]*}"}" +[ -z "$SLOTS_N" ] && SLOTS_N=0 + +# Anything unrecognised falls to 1 rather than to 0: a typo in the config must +# not silently remove the protection. That also catches Jinja rendering a +# leftover boolean `rust_governor_serialize: true` as `True`, which means +# serialise, which is one slot. +case "$SLOTS" in + auto|AUTO) N="$(compute_slots)" ;; + 0|off|OFF|false|False|FALSE|no|No|NO) N=0 ;; + *[!0-9]*) N=1 ;; + *) N="$SLOTS_N" ;; +esac + +# Jobs scale with the number of neighbours the semaphore admits, rather than +# assuming sole occupancy. Floored at 2: a measured single-job build took 2.3x +# as long, which is never the right computed default. +if [ "$N" -ge 1 ]; then + jobs=$((pool / N)) +else + jobs="$pool" +fi +[ "$jobs" -lt 2 ] && jobs=2 +[ "$pool" -lt 2 ] && jobs=1 + export HYPERI_RUST_GOVERNED=1 -LOCK="${XDG_RUNTIME_DIR:-${TMPDIR:-/tmp}}/hyperi-rust-govern-$(id -u).lock" +# Never /tmp. A world-writable slot directory lets any local account take an +# exclusive flock on every slot file -- read access is enough for flock(2) -- +# and pin every build on the host at the degraded job count. It also lets one +# be pre-created unwritable, which is worse. ~/.cache is the fallback because +# XDG_RUNTIME_DIR is absent for cron, containers, CI and plain `ssh host cargo`. +# +# Tested for emptiness, not just for being unset: an XDG_RUNTIME_DIR set to "" +# passes :- and yields a path at the filesystem root. +if [ -n "${XDG_RUNTIME_DIR:-}" ]; then + SLOT_BASE="$XDG_RUNTIME_DIR" +else + SLOT_BASE="${XDG_CACHE_HOME:-$HOME/.cache}/hyperi" +fi +SLOT_DIR="$SLOT_BASE/hyperi-rust-govern-$(id -u).slots" uname_s="$(uname -s)" +slot=0 + notify_wait() { - echo "hyperi-rust-govern: another rust build holds the lock -- waiting (HYPERI_RUST_GOVERNOR=off to bypass)" >&2 + echo "hyperi-rust-govern: all $N build slots are taken -- waiting (HYPERI_RUST_GOVERNOR=off to bypass)" >&2 } -# The same fd inheritance that makes the lock outlive a crash also hands it to -# any daemon the build starts, which then holds it for its own lifetime. -notify_timeout() { - if [ -n "${1:-}" ]; then - echo "hyperi-rust-govern: lock held by PID(s) $1 for over ${LOCK_WAIT}s -- building unserialised (HYPERI_RUST_GOVERNOR=off to silence)" >&2 - else - echo "hyperi-rust-govern: lock held for over ${LOCK_WAIT}s -- building unserialised (HYPERI_RUST_GOVERNOR=off to silence)" >&2 - fi +# No holder PIDs are reported. fuser on the slot files lists every process with +# the fd open, which is waiters and the reporting process too, so the number was +# consistently larger than the number of builds and told nobody anything. +notify_degraded() { + echo "hyperi-rust-govern: no slot after ${LOCK_WAIT}s -- building at -j$jobs, capped at ${ALLOWANCE_GB}G" >&2 } -# Linux: flock on fd 9. The fd survives exec, so the lock lives exactly as -# long as the build and releases on any exit, crash included. -if [ "$SERIALIZE" = "1" ] && command -v flock >/dev/null 2>&1; then - # An XDG_RUNTIME_DIR that is set but absent leaves fd 9 unopened, and every - # flock below then fails instantly -- which is not the same as contention. - # No 2>/dev/null here: on success that redirect would persist and silence - # the build's own stderr. - if ! exec 9>"$LOCK"; then - echo "hyperi-rust-govern: cannot open $LOCK -- building unserialised" >&2 - elif ! flock -n 9; then - notify_wait - # Serialisation is an optimisation, so a stuck holder degrades the build - # rather than hanging it. - if ! flock -w "$LOCK_WAIT" 9; then - # fuser space-separates several holders, so squeeze rather than - # delete or they concatenate into one meaningless number. - lock_holder="" - command -v fuser >/dev/null 2>&1 && - lock_holder="$(fuser "$LOCK" 2>/dev/null | tr -s '[:space:]' ' ')" - lock_holder="${lock_holder# }" - notify_timeout "${lock_holder% }" +# One fd over N files: reopening fd 9 closes the previous description and +# releases any flock on it, so slot choice is which FILE you open. That keeps +# this to bash 3.2 with no eval and no {var}> named descriptors. +# +# `<>` rather than `>`: `>` truncates, so probing a busy slot would wipe the +# holder's file on the way past. +take_slot() { + i=0 + while [ "$i" -lt "$N" ]; do + n=$(( (start_at + i - 1) % N + 1 )) + i=$((i + 1)) + # No 2>/dev/null on an exec redirect: on success it would persist and + # silence the build's own stderr. + exec 9<>"$SLOT_DIR/slot.$n" || continue + if flock "$@" 9; then + slot="$n" + return 0 fi + done + return 1 +} + +# A slot directory that cannot be used is NOT contention, and must not be +# mistaken for it: every lock would fail instantly, the wait loop would spin its +# whole budget in zero seconds printing a line per attempt, and the build would +# end at the degraded job count on a completely idle host. Checked once, and +# treated as ungoverned so the computed job count survives. +# +# The file open is probed as well as the mkdir: mkdir -p succeeds on a directory +# that already exists whether or not it is writable, and it is the open inside +# take_slot that has to work. ENOSPC and a read-only remount both land here. +# +# Created 0700, because a traversable home is not a private one: read access is +# all flock(2) needs, so a 0755 $HOME with a 0775 cache directory lets any local +# account hold every slot and pin every build at the degraded job count. +if [ "$N" -ge 1 ] && { ! (umask 077; mkdir -p "$SLOT_DIR") || ! : >>"$SLOT_DIR/slot.1"; } 2>/dev/null; then + echo "hyperi-rust-govern: cannot use $SLOT_DIR -- building ungoverned at -j$jobs" >&2 + N=0 +fi + +if [ "$N" -ge 1 ] && command -v flock >/dev/null 2>&1; then + # Spread waiters across slots rather than having every arrival stampede + # slot 1 and hand it to whoever wakes first. + start_at=$(( ($$ % N) + 1 )) + + if ! take_slot -n; then + notify_wait + waited=0 + # Short blocking waits cycled across every slot, so a waiter takes + # whichever frees first. One pass costs up to 2N seconds. + while [ "$waited" -lt "$LOCK_WAIT" ]; do + take_slot -w 2 && break + waited=$(( waited + 2 * N )) + done fi -elif [ "$SERIALIZE" = "1" ]; then +elif [ "$N" -ge 1 ]; then # macOS ships no flock(1). A pid-stamped mkdir lock is atomic everywhere; - # a holder that died without cleaning up is detected and cleared. - LOCK_D="$LOCK.d" + # a holder that died without cleaning up is detected and cleared. The trap + # is not inherited by children, so this path never had the daemon problem + # the flock path needed 9>&- to fix -- but a trap does not fire on SIGKILL, + # which is why the liveness check below is still required. + start_at=$(( ($$ % N) + 1 )) notified=0 waited=0 - holder="" - locked=0 while [ "$waited" -lt "$LOCK_WAIT" ]; do - if mkdir "$LOCK_D" 2>/dev/null; then - locked=1 - break - fi - holder="$(cat "$LOCK_D/pid" 2>/dev/null || true)" - # A failed clear falls through to the wait, so the loop always advances. - if [ -n "$holder" ] && ! kill -0 "$holder" 2>/dev/null; then - rm -rf "$LOCK_D" && continue - fi + i=0 + while [ "$i" -lt "$N" ]; do + n=$(( (start_at + i - 1) % N + 1 )) + i=$((i + 1)) + if mkdir "$SLOT_DIR/slot.$n.d" 2>/dev/null; then + # Stamped BEFORE the loop exits, so a reclaimer can never see a + # live holder's directory with no pid in it and clear it. + printf '%s\n' "$$" >"$SLOT_DIR/slot.$n.d/pid" + slot="$n" + break + fi + holder="$(cat "$SLOT_DIR/slot.$n.d/pid" 2>/dev/null || true)" + [ -n "$holder" ] || continue + kill -0 "$holder" 2>/dev/null && continue + # Rename rather than remove: exactly one reclaimer can win the mv, + # so two waiters cannot both clear and hand the same slot to two + # different builds. + if mv "$SLOT_DIR/slot.$n.d" "$SLOT_DIR/slot.$n.stale.$$" 2>/dev/null; then + rm -rf "$SLOT_DIR/slot.$n.stale.$$" + fi + done + [ "$slot" -ne 0 ] && break if [ "$notified" -eq 0 ]; then notify_wait notified=1 @@ -163,35 +340,72 @@ elif [ "$SERIALIZE" = "1" ]; then sleep 2 waited=$((waited + 2)) done - if [ "$locked" -eq 1 ]; then - printf '%s\n' "$$" >"$LOCK_D/pid" + if [ "$slot" -ne 0 ]; then # Single quotes: expanded when the trap fires, not here. - trap 'rm -rf "$LOCK_D"' EXIT INT TERM - else - # No trap: the directory belongs to whoever still holds it. - notify_timeout "$holder" + trap 'rm -rf "$SLOT_DIR/slot.$slot.d"' EXIT INT TERM fi fi +# No slot after the wait: degrade, do not release. The old behaviour proceeded +# unserialised after the timeout, dropping the bound at the exact moment +# contention was highest. The floor job count alone would not bound memory -- +# it is spent per crate, and one big crate peaks the same at -j2 as at -j30 -- +# so the systemd path below also caps the build at one allowance. +SLOTLESS_CAP="" +if [ "$slot" -eq 0 ] && [ "$N" -ge 1 ]; then + jobs=2 + [ "$pool" -lt 2 ] && jobs=1 + SLOTLESS_CAP="--property=MemoryMax=${ALLOWANCE_GB}G" + notify_degraded +fi + +# After acquisition, never before: a build that had to degrade needs the lower +# count, and a caller's own CARGO_BUILD_JOBS still wins over both. +export CARGO_BUILD_JOBS="${CARGO_BUILD_JOBS:-$jobs}" + +# sccache cannot cache a rustc call carrying -C incremental. Off by default: +# cargo never builds dependencies incrementally, so this only makes WORKSPACE +# crates cacheable, and it costs incremental on those same crates. Worth it on a +# box where builds start from clean trees, not on an edit-rebuild loop. +# +# A caller's own value still wins, but setting it to 1 makes sccache refuse the +# build outright rather than fall back to compiling. +if [ "${HYPERI_RUST_GOVERN_NO_INCREMENTAL:-0}" = "1" ]; then + export CARGO_INCREMENTAL="${CARGO_INCREMENTAL:-0}" +fi + # Linux with a live user manager: the scope joins rust-build.slice, whose -# memory/CPU limits are deployed by the developer-rust role. The exec keeps -# the flock fd, the TTY, signals and the exit code. +# memory/CPU limits are deployed by the developer-rust role. +# +# Run and wait rather than exec, and close fd 9 for the child. The fd is what +# holds the slot, and exec handed it to every descendant -- so any daemon the +# build started kept the slot for its own lifetime. Waiting costs one bash +# process, outside the scope; the kernel still releases the slot on a crash, +# because flock is held against the open file description. +# +# XDG_RUNTIME_DIR must be non-empty, not merely defaulted: systemd-run --user +# cannot reach the bus without it even when the socket is found by path. if [ "$uname_s" = "Linux" ] && command -v systemd-run >/dev/null 2>&1 \ - && [ -S "${XDG_RUNTIME_DIR:-/run/user/$(id -u)}/bus" ]; then - exec systemd-run --user --scope --quiet --collect \ - --slice=rust-build.slice \ - --property=CPUQuota="$((jobs * 100))%" \ - -- "$REAL" "$@" + && [ -n "${XDG_RUNTIME_DIR:-}" ] && [ -S "$XDG_RUNTIME_DIR/bus" ]; then + systemd-run --user --scope --quiet --collect \ + --slice=rust-build.slice ${SLOTLESS_CAP:+"$SLOTLESS_CAP"} \ + -- "$REAL" "$@" 9>&- + exit "$?" fi # macOS: QoS clamp puts every thread below the desktop's priority. No memory # cap here -- taskpolicy's jetsam limit is per-process and cannot bound the -# tree -- so the job cap above is the memory lever. +# tree -- so on that platform the semaphore and the job cap ARE the memory +# model, with nothing behind them. +# +# 9>&- on these two as well: a Mac with Homebrew util-linux has flock(1), so it +# takes the flock path above and reaches here holding fd 9. Closing an fd that +# was never opened is a no-op, so this is safe on the mkdir path too. if [ "$uname_s" = "Darwin" ] && command -v taskpolicy >/dev/null 2>&1; then - taskpolicy -c "$QOS" "$REAL" "$@" + taskpolicy -c "$QOS" "$REAL" "$@" 9>&- exit "$?" fi # No systemd user manager (container, degraded session): priority only. -nice -n 19 "$REAL" "$@" +nice -n 19 "$REAL" "$@" 9>&- exit "$?" diff --git a/ansible/roles/developer-rust/tasks/governor.yml b/ansible/roles/developer-rust/tasks/governor.yml index 42e1120..792c5ac 100644 --- a/ansible/roles/developer-rust/tasks/governor.yml +++ b/ansible/roles/developer-rust/tasks/governor.yml @@ -1,13 +1,30 @@ --- -# The rust build governor: one build at a time per user, the whole tree in -# the rust-build.slice budget, priority below the desktop. Opt-OUT -- -# `rust_governor_enabled: false` removes everything it deployed. +# The rust build governor: at most N builds at a time per user, N derived from +# the slice's memory budget, the whole tree in the rust-build.slice budget, +# priority below the desktop. Opt-OUT -- `rust_governor_enabled: false` +# removes everything it deployed. # # The shim shadows ~/.cargo/bin/cargo from ~/.local/bin, which the SOE PATH # puts first. sccache is captured separately: compiles execute inside the # sccache SERVER, so hyperi-sccache.service carries Slice=rust-build.slice # (see hyperi-sccache.service.j2), or the budget would miss most of the work. +# rust_governor_serialize was a boolean when the governor was a global mutex. +# It is now a count, under a name that says so. Failing here rather than +# ignoring it is deliberate: a host that still sets `false` believes it has +# disabled serialisation, and silently applying `auto` instead would hand it +# eight concurrent builds. A setting that is quietly ignored is the same defect +# class as a config written to a directory the tool never reads. +- name: Fail on the retired rust_governor_serialize variable + ansible.builtin.fail: + msg: >- + rust_governor_serialize has been replaced by rust_governor_slots, which + takes a COUNT rather than a boolean. `true` becomes + rust_governor_slots: 1 (one build at a time, the old behaviour), + `false` becomes rust_governor_slots: 0, and rust_governor_slots: auto + derives the count from this host's RAM. + when: rust_governor_serialize is defined + - name: Deploy the rust build governor when: rust_governor_enabled | bool block: @@ -103,14 +120,39 @@ scope: user become: true become_user: "{{ actual_user }}" - environment: - XDG_RUNTIME_DIR: "/run/user/{{ actual_user_uid | default(ansible_facts['user_uid']) }}" + environment: "{{ developer_rust_user_env }}" when: - ansible_facts['system'] == 'Linux' - not ansible_check_mode - developer_rust_governor_slice is changed failed_when: false + # MemoryHigh throttles by reclaim, and with no swap the only reclaimable + # memory is page cache -- so a build past its budget stalls rather than + # slows. The governor and the zram_swap role are two halves of one thing, + # and this is what makes that visible at converge time rather than only in + # the README. + - name: Check whether this host has any swap + ansible.builtin.command: + cmd: swapon --show=NAME --noheadings + changed_when: false + failed_when: false + check_mode: false + register: developer_rust_swap + when: ansible_facts['system'] == 'Linux' + + - name: Warn that the memory budget has nothing to reclaim to + ansible.builtin.set_fact: + deploy_warnings: >- + {{ deploy_warnings | default([]) + + ['Rust governor: deployed on a host with no swap. MemoryHigh' + ~ ' throttles by reclaim, so a build over budget will STALL' + ~ ' rather than slow. Apply the zram_swap role' + ~ ' (--tags zram) to give it somewhere to reclaim to.'] }} + when: + - ansible_facts['system'] == 'Linux' + - developer_rust_swap.stdout | default('') | trim | length == 0 + rescue: - name: Record that the rust governor did not deploy ansible.builtin.set_fact: @@ -179,8 +221,7 @@ scope: user become: true become_user: "{{ actual_user }}" - environment: - XDG_RUNTIME_DIR: "/run/user/{{ actual_user_uid | default(ansible_facts['user_uid']) }}" + environment: "{{ developer_rust_user_env }}" when: - ansible_facts['system'] == 'Linux' - not ansible_check_mode diff --git a/ansible/roles/developer-rust/tasks/rust.yml b/ansible/roles/developer-rust/tasks/rust.yml index e2534aa..a890559 100644 --- a/ansible/roles/developer-rust/tasks/rust.yml +++ b/ansible/roles/developer-rust/tasks/rust.yml @@ -20,11 +20,109 @@ # (hyperi-update runs it), so what a developer compiles with is whatever stable # is on the day, edition 2024 included. +# ============================================================ +# Where the toolchain actually lives +# ============================================================ +# Everything below addresses cargo through these two facts, never through a +# hard-coded ~/.cargo. Cargo reads $CARGO_HOME/config.toml and rustup reads +# $RUSTUP_HOME, and neither is required to be under the home directory -- a +# box with a dedicated toolchain volume relocates both. Writing to ~/.cargo on +# such a host produces a file the tool silently ignores while a stale one stays +# in effect, which is a failure mode with no error message anywhere in it. +# +# Ask the target user's own login shell, because that is the environment the +# developer's cargo runs in: it sees /etc/environment (via pam_env under +# become), /etc/profile.d and ~/.profile, which is where a relocation is +# declared. Ansible's `ansible_facts['env']` is the CONNECTION user's +# environment -- root under become -- so it cannot answer this. +# +# `${CARGO_HOME:-$HOME/.cargo}` is resolved by that shell, so the fallback uses +# the user's real home and matches hyperi-rust-govern's own resolution. +# +# The sentinel is not decoration: a login shell runs /etc/profile.d and +# ~/.profile first, and anything they print lands on stdout ahead of the answer. +# Selecting on a marker rather than on line 0 is what stops a login banner +# becoming CARGO_HOME. +# One task for every platform. Linux runs it under become as the target user; +# macOS as the connecting user, who is the target there. `timeout` is Linux +# coreutils: a profile that blocks would otherwise hang the converge, and +# `failed_when: false` offers no protection against a hang. +# +# Two sentinel lines rather than one line split on whitespace, so a path with a +# space in it resolves whole instead of silently as its first word. `last`, +# not `first`: the profile runs before the printf, so ours is always last, and +# a profile that prints the sentinel itself cannot win. +- name: Probe the target user's cargo environment + ansible.builtin.command: + cmd: >- + {{ 'zsh' if ansible_facts['distribution'] == 'MacOSX' else 'timeout 30 bash' }} + -lc 'printf "HYPERI_CARGO_HOME %s\nHYPERI_RUSTUP_HOME %s\n" "${CARGO_HOME:-$HOME/.cargo}" "${RUSTUP_HOME:-$HOME/.rustup}"' + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" + register: developer_rust_env_probe + changed_when: false + failed_when: false + # Must run in check mode too: the facts below are undefined without it, and + # every later task then templates an empty path. + check_mode: false + tags: ['rust-cache', 'rust-governor'] + +# An explicit role variable, else what the host said, else the upstream default +# so nothing downstream is ever templated blank. Trailing slashes are stripped +# because the retire task below compares this against ~/.cargo as a string, and +# `~/.cargo/` would read as a relocation and rename the live config. +- name: Resolve the effective cargo and rustup homes + ansible.builtin.set_fact: + developer_rust_cargo_home: >- + {{ rust_cargo_home | default(_probed_cargo, true) + | default(user_home ~ '/.cargo', true) | regex_replace('/+$', '') }} + developer_rust_rustup_home: >- + {{ rust_rustup_home | default(_probed_rustup, true) + | default(user_home ~ '/.rustup', true) | regex_replace('/+$', '') }} + vars: + _lines: "{{ developer_rust_env_probe.stdout_lines | default([], true) }}" + _probed_cargo: >- + {{ _lines | select('match', '^HYPERI_CARGO_HOME ') | list | last + | default('') | regex_replace('^\S+ ', '') }} + _probed_rustup: >- + {{ _lines | select('match', '^HYPERI_RUSTUP_HOME ') | list | last + | default('') | regex_replace('^\S+ ', '') }} + tags: ['rust-cache', 'rust-governor'] + +# Absolute, and free of the two characters that would break the root-owned +# unit file this value is templated into: `%` is a systemd specifier and `"` +# ends the quoted assignment. +- name: Check the resolved toolchain paths are usable + ansible.builtin.assert: + that: + - developer_rust_cargo_home is match('^/') + - developer_rust_rustup_home is match('^/') + - developer_rust_cargo_home is not search('[%"]') + - developer_rust_rustup_home is not search('[%"]') + fail_msg: >- + Resolved CARGO_HOME={{ developer_rust_cargo_home }}, + RUSTUP_HOME={{ developer_rust_rustup_home }} -- not an absolute path, or + contains % or ". Set rust_cargo_home/rust_rustup_home explicitly. + quiet: true + tags: ['rust-cache', 'rust-governor'] + +- name: Report where the toolchain was resolved to + ansible.builtin.debug: + msg: >- + CARGO_HOME={{ developer_rust_cargo_home }} + RUSTUP_HOME={{ developer_rust_rustup_home }} + verbosity: 1 + tags: ['rust-cache', 'rust-governor'] + # ============================================================ # Rust Installation (Linux -- one path for both distros) # ============================================================ - name: Install Rust via rustup (Linux) + # No rescue: every `cargo install` below needs the toolchain this block + # bootstraps, so warning past it only moves the abort to a task that reports + # it as a missing command. + when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] block: - name: Remove distro Rust packages if present (Fedora) ansible.builtin.dnf: @@ -54,7 +152,7 @@ # otherwise the download-then-delete pair below reports changed every time. - name: Check whether rustup is already installed ansible.builtin.stat: - path: "{{ user_home }}/.cargo/bin/rustup" + path: "{{ developer_rust_cargo_home }}/bin/rustup" register: developer_rustup_bin - name: Bootstrap the Rust toolchain @@ -84,12 +182,12 @@ - name: Install the Rust toolchain via rustup-init ansible.builtin.command: cmd: /tmp/rustup-init -y --default-toolchain stable --no-modify-path - creates: "{{ user_home }}/.cargo/bin/rustup" + creates: "{{ developer_rust_cargo_home }}/bin/rustup" become: true become_user: "{{ actual_user }}" environment: - CARGO_HOME: "{{ user_home }}/.cargo" - RUSTUP_HOME: "{{ user_home }}/.rustup" + CARGO_HOME: "{{ developer_rust_cargo_home }}" + RUSTUP_HOME: "{{ developer_rust_rustup_home }}" - name: Remove the rustup-init download ansible.builtin.file: @@ -102,12 +200,12 @@ # non-zero merely because a rustup self-update is offered. - name: Check whether the Rust toolchain is current ansible.builtin.command: - cmd: "{{ user_home }}/.cargo/bin/rustup check" + cmd: "{{ developer_rust_cargo_home }}/bin/rustup check" become: true become_user: "{{ actual_user }}" environment: - CARGO_HOME: "{{ user_home }}/.cargo" - RUSTUP_HOME: "{{ user_home }}/.rustup" + CARGO_HOME: "{{ developer_rust_cargo_home }}" + RUSTUP_HOME: "{{ developer_rust_rustup_home }}" register: developer_rustup_check changed_when: false failed_when: false @@ -121,12 +219,12 @@ # silently deciding there was nothing to do. - name: Update the Rust toolchain to current stable ansible.builtin.command: - cmd: "{{ user_home }}/.cargo/bin/rustup update stable" + cmd: "{{ developer_rust_cargo_home }}/bin/rustup update stable" become: true become_user: "{{ actual_user }}" environment: - CARGO_HOME: "{{ user_home }}/.cargo" - RUSTUP_HOME: "{{ user_home }}/.rustup" + CARGO_HOME: "{{ developer_rust_cargo_home }}" + RUSTUP_HOME: "{{ developer_rust_rustup_home }}" changed_when: true when: >- developer_rustup_check.rc | default(1) != 0 @@ -136,12 +234,12 @@ - name: Install Rust components via rustup ansible.builtin.command: - cmd: "{{ user_home }}/.cargo/bin/rustup component add rustfmt clippy" + cmd: "{{ developer_rust_cargo_home }}/bin/rustup component add rustfmt clippy" become: true become_user: "{{ actual_user }}" environment: - CARGO_HOME: "{{ user_home }}/.cargo" - RUSTUP_HOME: "{{ user_home }}/.rustup" + CARGO_HOME: "{{ developer_rust_cargo_home }}" + RUSTUP_HOME: "{{ developer_rust_rustup_home }}" # `component add` already prints "component X is up to date" and no-ops # when present, so key changed off that rather than always reporting it. register: developer_rustup_components @@ -151,11 +249,6 @@ # "not changed", which is honest: the components are present either way. changed_when: "'is up to date' not in developer_rustup_components.stderr" - # No rescue: every `cargo install` below needs the toolchain this block - # bootstraps, so warning past it only moves the abort to a task that reports - # it as a missing command. - when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] - # brew's rust formula conflicts with rustup and brew refuses the pair, so a # Mac already on rustup keeps it -- rustup's toolchains carry cargo and # rustfmt, the same arrangement as the Linux path. @@ -183,18 +276,22 @@ # Cargo bin directory setup # ============================================================ +# As the user, not as root. The path comes from the user's own environment, and +# every other consumer of it runs as the user -- having root create an arbitrary +# user-supplied directory tree and hand the leaf to that user is a privilege +# boundary this task has no reason to cross. - name: Create cargo bin directory (Linux) ansible.builtin.file: - path: "{{ user_home }}/.cargo/bin" + path: "{{ developer_rust_cargo_home }}/bin" state: directory mode: '0755' - owner: "{{ actual_user }}" - group: "{{ actual_user }}" + become: true + become_user: "{{ actual_user }}" when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] - name: Create cargo bin directory (macOS) ansible.builtin.file: - path: "{{ user_home }}/.cargo/bin" + path: "{{ developer_rust_cargo_home }}/bin" state: directory mode: '0755' become: false @@ -207,7 +304,13 @@ # # The guard is required either way -- an unguarded prepend stacks a copy per # nested shell -- and it also makes this a no-op on a soe box, where -# /etc/profile.d/dev-path.sh already covers ~/.cargo/bin. +# /etc/profile.d/dev-path.sh already covers the cargo bin directory. +# +# `${CARGO_HOME:-$HOME/.cargo}` rather than the resolved path: pam_env has +# already applied /etc/environment by the time .profile runs, so the shell +# knows the answer itself. That keeps the entry correct if the toolchain is +# relocated later without a converge, and it is the same expression +# hyperi-rust-govern resolves the real cargo with. - name: Add cargo bin to PATH in .profile (Linux) ansible.builtin.blockinfile: path: "{{ user_home }}/.profile" @@ -218,8 +321,8 @@ mode: '0644' block: | case ":$PATH:" in - *":$HOME/.cargo/bin:"*) ;; - *) export PATH="$HOME/.cargo/bin:$PATH" ;; + *":${CARGO_HOME:-$HOME/.cargo}/bin:"*) ;; + *) export PATH="${CARGO_HOME:-$HOME/.cargo}/bin:$PATH" ;; esac when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] @@ -232,11 +335,22 @@ failed_when: false when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] +# Same shell-derived form as the Linux .profile block above, for the same +# reason. +# +# The regexp matches the two forms this role has ever written and nothing else. +# It has to match the new line as well as the retired one, or lineinfile appends +# a fresh copy every run -- but a loose `.*cargo.*/bin` also swallows an +# unrelated `export PATH=` line that merely mentions cargo somewhere, replacing +# every other entry on it. +# +# No `mode`: lineinfile applies it to a pre-existing file too, so setting 0644 +# would widen a .zshrc the user had deliberately kept at 0600. - name: Add cargo bin to PATH in .zshrc (macOS) ansible.builtin.lineinfile: path: "{{ user_home }}/.zshrc" - regexp: "^export PATH=.*\\.cargo/bin" - line: 'export PATH="{{ user_home }}/.cargo/bin:$PATH"' + regexp: '^export PATH="?(\$\{CARGO_HOME:-\$HOME/\.cargo\}|\S*/\.cargo)/bin' + line: 'export PATH="${CARGO_HOME:-$HOME/.cargo}/bin:$PATH"' state: present create: true become: false @@ -305,9 +419,9 @@ - name: Set cargo environment (Linux) ansible.builtin.set_fact: cargo_env: - PATH: "{{ user_home }}/.cargo/bin:{{ ansible_facts['env'].PATH }}" - CARGO_HOME: "{{ user_home }}/.cargo" - RUSTUP_HOME: "{{ user_home }}/.rustup" + PATH: "{{ developer_rust_cargo_home }}/bin:{{ ansible_facts['env'].PATH }}" + CARGO_HOME: "{{ developer_rust_cargo_home }}" + RUSTUP_HOME: "{{ developer_rust_rustup_home }}" when: ansible_facts['distribution'] in ['Fedora', 'Ubuntu'] tags: ['rust-cache'] @@ -316,8 +430,14 @@ - name: Set cargo environment (macOS) ansible.builtin.set_fact: cargo_env: - PATH: "/opt/homebrew/opt/rustup/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:{{ user_home }}/.cargo/bin:{{ ansible_facts['env'].PATH }}" - CARGO_HOME: "{{ user_home }}/.cargo" + PATH: >- + {{ '/opt/homebrew/opt/rustup/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:' + ~ developer_rust_cargo_home ~ '/bin:' ~ ansible_facts['env'].PATH }} + CARGO_HOME: "{{ developer_rust_cargo_home }}" + # Resolves to ~/.rustup when the host says nothing, which is what rustup + # would have defaulted to anyway. Carried explicitly so a Mac that + # relocates it is not the one platform still hard-coded. + RUSTUP_HOME: "{{ developer_rust_rustup_home }}" when: ansible_facts['distribution'] == 'MacOSX' tags: ['rust-cache'] @@ -329,7 +449,7 @@ - name: Install cargo-binstall (prebuilt cargo tool fetcher) ansible.builtin.command: cmd: cargo install cargo-binstall --locked - creates: "{{ user_home }}/.cargo/bin/cargo-binstall" + creates: "{{ developer_rust_cargo_home }}/bin/cargo-binstall" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -344,7 +464,7 @@ - name: Install the cargo tool set ansible.builtin.command: cmd: cargo install {{ item.crate }} --locked - creates: "{{ user_home }}/.cargo/bin/{{ item.bin }}" + creates: "{{ developer_rust_cargo_home }}/bin/{{ item.bin }}" loop: - {crate: bacon, bin: bacon} - {crate: cargo-nextest, bin: cargo-nextest} @@ -366,14 +486,14 @@ - name: Install cargo-llvm-cov (code coverage) ansible.builtin.command: cmd: cargo install cargo-llvm-cov --locked - creates: "{{ user_home }}/.cargo/bin/cargo-llvm-cov" + creates: "{{ developer_rust_cargo_home }}/bin/cargo-llvm-cov" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" - name: Install the llvm-tools component (cargo-llvm-cov dependency) ansible.builtin.command: - cmd: "{{ user_home }}/.cargo/bin/rustup component add llvm-tools-preview" + cmd: "{{ developer_rust_cargo_home }}/bin/rustup component add llvm-tools-preview" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" environment: "{{ cargo_env }}" @@ -386,7 +506,7 @@ - name: Remove the superseded cargo-tarpaulin ansible.builtin.command: cmd: cargo uninstall cargo-tarpaulin - removes: "{{ user_home }}/.cargo/bin/cargo-tarpaulin" + removes: "{{ developer_rust_cargo_home }}/bin/cargo-tarpaulin" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -394,7 +514,7 @@ - name: Install cargo-pgo (profile-guided optimisation) ansible.builtin.command: cmd: cargo install cargo-pgo --locked - creates: "{{ user_home }}/.cargo/bin/cargo-pgo" + creates: "{{ developer_rust_cargo_home }}/bin/cargo-pgo" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -402,7 +522,7 @@ - name: Install cargo-chef (Docker layer caching) ansible.builtin.command: cmd: cargo install cargo-chef --locked - creates: "{{ user_home }}/.cargo/bin/cargo-chef" + creates: "{{ developer_rust_cargo_home }}/bin/cargo-chef" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -414,7 +534,7 @@ - name: Install cargo-audit (RustSec advisory audit) ansible.builtin.command: cmd: cargo install cargo-audit --locked - creates: "{{ user_home }}/.cargo/bin/cargo-audit" + creates: "{{ developer_rust_cargo_home }}/bin/cargo-audit" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -422,7 +542,7 @@ - name: Install cargo-hack (feature-matrix testing) ansible.builtin.command: cmd: cargo install cargo-hack --locked - creates: "{{ user_home }}/.cargo/bin/cargo-hack" + creates: "{{ developer_rust_cargo_home }}/bin/cargo-hack" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -435,7 +555,7 @@ - name: Install cargo-update (for updating cargo tools) ansible.builtin.command: cmd: cargo install cargo-update --locked - creates: "{{ user_home }}/.cargo/bin/cargo-install-update" + creates: "{{ developer_rust_cargo_home }}/bin/cargo-install-update" environment: "{{ cargo_env }}" become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" @@ -524,7 +644,7 @@ environment: "{{ cargo_env | combine(hyperi_github_env) }}" register: developer_rust_setup changed_when: "'CHANGED' in developer_rust_setup.stdout" - # An optional build accelerator must never abort the run (see S1.T11). + # An optional build accelerator must never abort the run. failed_when: false tags: ['rust-cache'] @@ -534,6 +654,112 @@ when: developer_rust_setup.rc | default(0) != 0 tags: ['rust-cache'] +# ============================================================ +# Post-condition: the toolchain the converge just left behind works +# ============================================================ +# A rustc-wrapper that does not exist breaks EVERY cargo command on the box +# with `No such file or directory`, and nothing upstream of here notices: the +# accelerator is optional and swallows its own failures, the tools all report +# success, and the config that is actually in effect may not even be the one +# this run wrote. That combination produced a workstation where every build had +# been failing since the wrapper's binary moved, through repeated green +# converges. +# +# So this is a post-condition on the file cargo will really read, not a check on +# what we intended to write, and it is fatal. A broken wrapper is not a degraded +# accelerator, it is a toolchain that cannot compile anything. + +# BOTH candidate locations are checked, not just the resolved one. Checking +# only where the role wrote would make this self-certifying: a resolution that +# picked the wrong directory would write a good config there, read the same +# good config back, and pass, while cargo went on reading a stale file +# somewhere else. A leftover config in the location that is NOT in effect is +# exactly the trap this whole change exists to close. +- name: Verify no cargo config names an unusable rustc-wrapper + when: rust_verify_wrapper | default(true) | bool + tags: ['rust-cache'] + block: + - name: Read every candidate cargo config + ansible.builtin.slurp: + src: "{{ item }}" + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" + register: developer_rust_cargo_configs + # A config that is absent is a legitimate state -- cargo has defaults for + # everything in it. Only one that NAMES a broken wrapper is a fault. + failed_when: false + check_mode: false + loop: "{{ [developer_rust_cargo_home ~ '/config.toml', + user_home ~ '/.cargo/config.toml'] | unique }}" + + # Both TOML quote styles and the dotted-key form, because the file is not + # always ours -- the whole point is to catch a foreign or stale one. + # `[^"'\n]` and not `[^"']`: the latter spans newlines, so a config whose + # value is unterminated swallows the rest of the file and the check then + # passes on nonsense. + - name: Extract the rustc-wrapper each config names + ansible.builtin.set_fact: + developer_rust_wrappers: >- + {{ developer_rust_cargo_configs.results + | selectattr('content', 'defined') | map(attribute='content') + | map('b64decode') | join('\n') + | regex_findall('(?m)^[ \t]*(?:build\.)?rustc-wrapper[ \t]*=[ \t]*["\x27]([^"\x27\n]+)["\x27]') + | unique }} + + # argv, not cmd: the command module shlex-splits a cmd string, so a value + # like `sccache --flag` would be split and `command -v sccache` would pass + # while every build failed. Cargo treats the whole value as one program. + # `-x` on the result, because `command -v` also answers for a builtin or a + # profile-defined function, neither of which cargo can execute. + - name: Check that each configured rustc-wrapper resolves + ansible.builtin.command: + argv: [bash, -lc, '[ -x "$(command -v -- "$1")" ]', bash, "{{ item }}"] + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" + register: developer_rust_wrapper_checks + changed_when: false + failed_when: false + check_mode: false + loop: "{{ developer_rust_wrappers }}" + + # Renamed, not deleted. It is only reachable if the relocation is ever + # dropped, at which point cargo silently starts obeying whatever it says -- + # so it is worth neutralising, but not worth destroying a file on a host we + # do not own. `removes:` makes this a no-op once the file is gone. + - name: Retire a cargo config superseded by a relocated CARGO_HOME + ansible.builtin.command: + argv: + - mv + - "{{ user_home }}/.cargo/config.toml" + - "{{ user_home }}/.cargo/config.toml.superseded" + removes: "{{ user_home }}/.cargo/config.toml" + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + become_user: "{{ actual_user if ansible_facts['distribution'] != 'MacOSX' else omit }}" + register: developer_rust_retired + when: + - rust_retire_superseded_config | default(true) | bool + - developer_rust_cargo_home != user_home ~ '/.cargo' + + # Fatal even on the run that just retired the offending file. The toolchain + # WAS broken, and a converge that repairs it and then says so is the loud + # failure this check exists to produce; the next run is clean. + - name: Fail when a cargo config names a rustc-wrapper that does not exist + ansible.builtin.fail: + msg: >- + A cargo config on this host sets rustc-wrapper = + "{{ developer_rust_wrapper_checks.results + | selectattr('rc', 'defined') | selectattr('rc', 'ne', 0) + | map(attribute='item') | join('", "') }}", + which does not resolve for {{ actual_user }}. Every cargo command that + reads it fails with "No such file or directory". The effective + CARGO_HOME here is {{ developer_rust_cargo_home }}. + {{ 'The stale ' ~ user_home ~ '/.cargo/config.toml has just been retired to config.toml.superseded -- re-run the converge.' + if developer_rust_retired is changed + else 'Check that is where you expect, then re-run /usr/local/bin/hyperi-rust-setup with that CARGO_HOME set.' }} + when: developer_rust_wrapper_checks.results | default([]) + | selectattr('rc', 'defined') | selectattr('rc', 'ne', 0) + | list | length > 0 + # ============================================================ # Build cache caps # ============================================================ diff --git a/ansible/roles/developer-rust/tasks/sccache_service.yml b/ansible/roles/developer-rust/tasks/sccache_service.yml index 8138748..69794a7 100644 --- a/ansible/roles/developer-rust/tasks/sccache_service.yml +++ b/ansible/roles/developer-rust/tasks/sccache_service.yml @@ -95,6 +95,41 @@ - developer_rust_sccache_start is failed or developer_rust_sccache_start.rc | default(0) != 0 +# A server started before the unit changed is still running the old ceiling and +# base directories, because sccache reads them once at startup. Restarting is +# only safe for a server this unit owns -- an unmanaged one may have a compile +# executing inside it, which is why the start task above refuses to touch it. +- name: Check whether the running server is this unit's + ansible.builtin.systemd_service: + name: hyperi-sccache.service + scope: user + become: true + become_user: "{{ actual_user }}" + environment: + XDG_RUNTIME_DIR: "/run/user/{{ actual_user_uid | default(ansible_facts['user_uid']) }}" + register: developer_rust_sccache_active + changed_when: false + failed_when: false + check_mode: false + +- name: Restart the sccache server so the new settings take effect + ansible.builtin.systemd: + name: hyperi-sccache.service + state: restarted + scope: user + daemon_reload: true + become: true + become_user: "{{ actual_user }}" + environment: + XDG_RUNTIME_DIR: "/run/user/{{ actual_user_uid | default(ansible_facts['user_uid']) }}" + when: + - not ansible_check_mode + - developer_rust_sccache_unit is changed + - developer_rust_sccache_active.status.ActiveState | default('') == 'active' + # A server the start task just brought up already runs the new unit. + - developer_rust_sccache_start is skipped + failed_when: false + - name: Note that the sccache unit takes over at the next login ansible.builtin.set_fact: deploy_warnings: >- @@ -102,6 +137,9 @@ + ['sccache: an unmanaged server is already running, so the new unit' ~ ' was enabled but not started. It takes over at the next login,' ~ ' or run: systemctl --user start hyperi-sccache'] }} + # pgrep matches the unit's own server too, so without the ActiveState test + # this warned on every healthy converge. when: - not ansible_check_mode - developer_rust_sccache_running.rc == 0 + - developer_rust_sccache_active.status.ActiveState | default('') != 'active' diff --git a/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune-guard.service.j2 b/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune-guard.service.j2 index d179598..7babee9 100644 --- a/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune-guard.service.j2 +++ b/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune-guard.service.j2 @@ -4,6 +4,9 @@ Description=Prune the pooled Rust build cache when free space runs short [Service] Type=oneshot User={{ actual_user }} +# A systemd unit inherits none of the user's shell environment, and the pruner +# reads $CARGO_HOME to find the config that names the pool. +Environment="CARGO_HOME={{ developer_rust_cargo_home }}" # Same prune with the same ceiling, gated on free space. Above the floor it # exits after one statvfs without walking the pool. ExecStart=/usr/local/bin/hyperi-rust-cache-prune --yes --max-size {{ rust_cache_build_dir_max }} --max-age-days {{ rust_cache_max_age_days }}{{ ' --cache-root ' ~ rust_cache_root if rust_cache_root else '' }} --if-free-below {{ rust_cache_prune_free_floor }} diff --git a/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune.service.j2 b/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune.service.j2 index 60e0c4b..6b5b2c3 100644 --- a/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune.service.j2 +++ b/ansible/roles/developer-rust/templates/hyperi-rust-cache-prune.service.j2 @@ -4,6 +4,11 @@ Description=Bound the pooled Rust build cache [Service] Type=oneshot User={{ actual_user }} +# A systemd unit inherits none of the user's shell environment, and the pruner +# reads $CARGO_HOME to find the config that names the pool. Without this it +# falls back to ~/.cargo and prunes against the wrong config on exactly the +# hosts that relocate the toolchain. +Environment="CARGO_HOME={{ developer_rust_cargo_home }}" # Runs entirely inside the user's own cache directory, so it needs no sudo. ExecStart=/usr/local/bin/hyperi-rust-cache-prune --yes --max-size {{ rust_cache_build_dir_max }} --max-age-days {{ rust_cache_max_age_days }}{{ ' --cache-root ' ~ rust_cache_root if rust_cache_root else '' }} # Walking a large pool is IO-bound; it must never compete with an active build. diff --git a/ansible/roles/developer-rust/templates/hyperi-sccache.service.j2 b/ansible/roles/developer-rust/templates/hyperi-sccache.service.j2 index 1607fc8..c30c939 100644 --- a/ansible/roles/developer-rust/templates/hyperi-sccache.service.j2 +++ b/ansible/roles/developer-rust/templates/hyperi-sccache.service.j2 @@ -18,6 +18,18 @@ Environment=SCCACHE_NO_DAEMON=1 # The server exits after 600s idle by default, which systemd would read as the # service failing. Environment=SCCACHE_IDLE_TIMEOUT=0 +# The cap and the base directories are set HERE, not left to the shell profile. +# The server reads them once at startup and holds them for its whole life, and a +# unit inherits pam_env rather than /etc/profile.d -- so a hand-set value in +# /etc/environment silently beat the role's, and the running server ended up +# with a ceiling nobody configured. A unit Environment= is applied last and +# wins, and changing it here is what tells the role to restart the server. +Environment=SCCACHE_CACHE_SIZE={{ rust_cache_sccache_max }} +{% if rust_cache_sccache_basedirs %} +# sccache keys on absolute paths, so the same source built under a different +# root misses. Listed roots are stripped before hashing. +Environment=SCCACHE_BASEDIRS={{ rust_cache_sccache_basedirs }} +{% endif %} ExecStart=/usr/local/bin/sccache Restart=on-failure RestartSec=5 diff --git a/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune-guard.plist.j2 b/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune-guard.plist.j2 index a02341e..8dd9064 100644 --- a/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune-guard.plist.j2 +++ b/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune-guard.plist.j2 @@ -21,6 +21,14 @@ {{ rust_cache_prune_free_floor }} + + EnvironmentVariables + + CARGO_HOME + {{ developer_rust_cargo_home }} + + diff --git a/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune.plist.j2 b/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune.plist.j2 index 4390e0f..83ada0f 100644 --- a/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune.plist.j2 +++ b/ansible/roles/developer-rust/templates/io.hyperi.rust-cache-prune.plist.j2 @@ -20,6 +20,14 @@ {% endif %} + + EnvironmentVariables + + CARGO_HOME + {{ developer_rust_cargo_home }} + + StartCalendarInterval diff --git a/ansible/roles/developer-rust/templates/rust-build.slice.j2 b/ansible/roles/developer-rust/templates/rust-build.slice.j2 index e4ef417..347cc9b 100644 --- a/ansible/roles/developer-rust/templates/rust-build.slice.j2 +++ b/ansible/roles/developer-rust/templates/rust-build.slice.j2 @@ -2,15 +2,28 @@ [Unit] Description=Governed Rust builds (hyperi-developer) +# systemd reads `-` as a hierarchy separator, so this unit lands under an +# auto-created `rust.slice` that nothing manages and nothing limits. Harmless, +# because the limits below bind on this slice -- but limits set on "rust.slice" +# expecting them to reach builds would land on an empty wrapper. [Slice] # Percentages resolve against this host's own RAM and swap, so one unit fits # every machine. Both capture prongs land here: cargo trees via the # hyperi-rust-govern scope, sccache-hosted compiles via Slice= on # hyperi-sccache.service. +# +# MemoryHigh is also what the shim divides to size its build semaphore, so the +# two are derived from one number and cannot drift. +# +# MemorySwapMax only means anything where swap exists. On a swapless host +# MemoryHigh has nothing but page cache to reclaim, so it stalls a build +# instead of slowing it -- the zram_swap role is what makes this budget behave. MemoryHigh={{ rust_governor_memory_high_pct }}% MemoryMax={{ rust_governor_memory_max_pct }}% MemorySwapMax={{ rust_governor_swap_max_pct }}% # Yields to the desktop under contention; no effect while the machine is idle. -# The cores-minus-reserve CPU quota is not here: hyperi-rust-govern sets it on -# each scope from the live core count, so a resized VM needs no re-converge. +# No CPU quota anywhere: a quota wastes cores whenever a session sits idle, +# whereas weight gives proportional share under contention and the whole box +# when nothing is competing. Parallelism is bounded by CARGO_BUILD_JOBS, which +# the shim scales to the number of neighbours the semaphore admits. CPUWeight={{ rust_governor_cpu_weight }} diff --git a/ansible/roles/developer-rust/templates/rust-governor.conf.j2 b/ansible/roles/developer-rust/templates/rust-governor.conf.j2 index f413d1b..2f04d06 100644 --- a/ansible/roles/developer-rust/templates/rust-governor.conf.j2 +++ b/ansible/roles/developer-rust/templates/rust-governor.conf.j2 @@ -1,7 +1,22 @@ # Deployed by hyperi-developer (developer-rust role). Edits are overwritten. # Read by hyperi-rust-govern at the start of every governed invocation. -HYPERI_RUST_GOVERN_SERIALIZE={{ '1' if rust_governor_serialize else '0' }} -HYPERI_RUST_GOVERN_CPU_RESERVE={{ rust_governor_cpu_reserve_cores }} -HYPERI_RUST_GOVERN_MACOS_QOS={{ rust_governor_macos_qos }} -# Seconds to wait for the build lock before giving up and building unserialised. -HYPERI_RUST_GOVERN_LOCK_WAIT={{ rust_governor_lock_wait_seconds }} +# +# Policy only. Every size is resolved by the shim against the live host, so a +# resized VM needs no re-converge -- the same bargain the CPU reserve makes. +# +# Each value is written as a DEFAULT rather than an assignment. The shim sources +# this file before reading its variables, so a plain assignment here would +# silently beat anything the caller set, leaving no way to tune one invocation +# and no way to test the shim against a real deployed config. +HYPERI_RUST_GOVERN_SLOTS="${HYPERI_RUST_GOVERN_SLOTS:-{{ rust_governor_slots }}}" +HYPERI_RUST_GOVERN_CPU_RESERVE="${HYPERI_RUST_GOVERN_CPU_RESERVE:-{{ rust_governor_cpu_reserve_cores }}}" +HYPERI_RUST_GOVERN_MACOS_QOS="${HYPERI_RUST_GOVERN_MACOS_QOS:-{{ rust_governor_macos_qos }}}" +# Seconds to wait for a build slot before building at the floor job count. +HYPERI_RUST_GOVERN_LOCK_WAIT="${HYPERI_RUST_GOVERN_LOCK_WAIT:-{{ rust_governor_lock_wait_seconds }}}" +# Matches rust-build.slice's MemoryHigh so the shim's slot count and systemd's +# memory ceiling are derived from the same number. +HYPERI_RUST_GOVERN_MEMORY_HIGH_PCT="${HYPERI_RUST_GOVERN_MEMORY_HIGH_PCT:-{{ rust_governor_memory_high_pct }}}" +HYPERI_RUST_GOVERN_BUILD_ALLOWANCE_GB="${HYPERI_RUST_GOVERN_BUILD_ALLOWANCE_GB:-{{ rust_governor_build_allowance_gb }}}" +# sccache cannot cache an incremental rustc call, and cargo's dev profile turns +# incremental on by default. +HYPERI_RUST_GOVERN_NO_INCREMENTAL="${HYPERI_RUST_GOVERN_NO_INCREMENTAL:-{{ '1' if rust_governor_no_incremental else '0' }}}" diff --git a/ansible/roles/zram_swap/README.md b/ansible/roles/zram_swap/README.md new file mode 100644 index 0000000..f7a1156 --- /dev/null +++ b/ansible/roles/zram_swap/README.md @@ -0,0 +1,64 @@ +# zram_swap + +A small zram-backed swap device, so cgroup memory limits throttle instead of +stalling. **Opt-in** -- not in the default install, not in `contributor`, not in +`soe`. + + ./install.sh --tags zram + +## Why this is not "add some swap" + +`MemoryHigh` on a cgroup is implemented as **reclaim**, not as a hard block. It +does not refuse an allocation; it pushes the cgroup into reclaim and lets it +carry on, which is what makes it a soft limit worth setting. + +On a host with no swap the only reclaimable memory is page cache. So once a +process group's **anonymous** memory passes `MemoryHigh` there is nothing left +to reclaim, and the throttle stops behaving like a slowdown and starts behaving +like a stall. + +zram fixes that by making anonymous pages reclaimable: they are compressed in +place rather than written anywhere. It adds no capacity and is not a swap tier. +It is somewhere for the throttle to push, which is why a few GB is the right +size and a disk-backed swap file is not a substitute. + +This matters most on a host running `developer-rust`'s build governor, whose +`rust-build.slice` sets `MemoryHigh` and `MemorySwapMax` as percentages of RAM. +Without swap, those limits do not degrade a build -- they wedge it. + +## What it touches + +- `/etc/systemd/zram-generator.conf` -- the device definition. This **replaces** + the distro default rather than merging with it, deliberately: Fedora's + `zram-generator-defaults` ships a `host-memory-limit`, and a host with more + RAM than that limit silently gets no device at all. +- `/etc/sysctl.d/zzz-50-hyperi-zram.conf` -- `vm.swappiness`, defaulting to 180 + rather than the stock 60. Swappiness balances reclaiming page cache against + reclaiming anonymous memory, and 60 was tuned for swap that costs a disk seek. + zram costs a compress, so anonymous memory is the cheaper thing to reclaim. + Set `zram_swap_swappiness: ""` to leave the host alone -- which is what you + want if the box also has disk swap, since 180 would then push pages to a disk. +- `systemd-zram-setup@zram0.service` -- started. Generator-created units + cannot be enabled; the generator wires it into `swap.target` itself. + +## Sizing + +| variable | default | on 246 GB | on 32 GB | on 8 GB | +|---|---|---|---|---| +| `zram_swap_size` | `min(ram / 8, 8192)` | 8 GiB | 4 GiB | 1 GiB | + +Capped at 8 GiB because the device is a reclaim target, not capacity. + +## Re-running with a changed size + +The role never restarts a running zram device. Applying a new size means +`swapoff`, which pages every byte held in the device back into RAM -- on a host +under memory pressure that is precisely the wrong moment, and it is how applying +a config change OOMs a build box. A changed size is written to the config and +takes effect at the next reboot. + +## Verifying + + swapon --show + zramctl + sysctl vm.swappiness diff --git a/ansible/roles/zram_swap/defaults/main.yml b/ansible/roles/zram_swap/defaults/main.yml new file mode 100644 index 0000000..213fda7 --- /dev/null +++ b/ansible/roles/zram_swap/defaults/main.yml @@ -0,0 +1,37 @@ +--- +# zram swap. +# +# This exists for ONE reason: cgroup memory throttling is implemented as +# reclaim. `MemoryHigh` does not block an allocation, it pushes the cgroup into +# reclaim and lets it continue. On a host with NO swap the only reclaimable +# memory is page cache, so once a build's ANONYMOUS memory passes the line +# there is nothing left to reclaim and the throttle stops being a slowdown and +# becomes a stall. +# +# zram makes anonymous pages reclaimable again by compressing them in RAM. It +# is not extra capacity and it is not a swap tier -- it is somewhere for the +# throttle to push, so a build over its budget degrades instead of wedging. +# That is why a few GB is the right size and a disk-backed swap file is not the +# same thing. + +zram_swap_enabled: true + +# MiB, evaluated by the generator against this host's own RAM. Capped at 8 GiB +# because the point is a reclaim target, not capacity: 246 GB gets 8192, +# 32 GB gets 4096, 8 GB gets 1024. +zram_swap_size: "min(ram / 8, 8192)" + +# zstd compresses harder than lzo-rle for a little more CPU, which is the right +# trade here -- the device is small and the host has cores to spare. +zram_swap_compression: zstd + +# 180, not the stock 60. Swappiness balances reclaiming page cache against +# reclaiming anonymous memory, and it was tuned for swap that costs a disk seek. +# zram costs a memcpy and a compress, so anonymous memory is the CHEAPER thing +# to reclaim and the default has the balance backwards. Values above 100 need +# kernel 5.8 or newer. +# +# This is correct for the zram-ONLY swap this role sets up. Set it to "" to +# leave the host's swappiness alone -- which is what you want if the box also +# has disk swap, because 180 would then push anonymous pages onto a disk. +zram_swap_swappiness: 180 diff --git a/ansible/roles/zram_swap/tasks/main.yml b/ansible/roles/zram_swap/tasks/main.yml new file mode 100644 index 0000000..c4b544d --- /dev/null +++ b/ansible/roles/zram_swap/tasks/main.yml @@ -0,0 +1,178 @@ +--- +# zram swap, so cgroup memory throttling has somewhere to reclaim to. +# See defaults/main.yml for why this is not "add some swap". + +- name: Set up zram swap + when: + - zram_swap_enabled | bool + - ansible_facts['system'] == 'Linux' + block: + - name: Install the zram generator (Ubuntu) + ansible.builtin.apt: + name: systemd-zram-generator + state: present + become: true + when: ansible_facts['distribution'] == 'Ubuntu' + + # zram-generator, NOT zram-generator-defaults: that subpackage exists only + # to ship a default config, and this role ships its own. + - name: Install the zram generator (Fedora) + ansible.builtin.dnf: + name: zram-generator + state: present + become: true + when: ansible_facts['distribution'] == 'Fedora' + + - name: Deploy the zram generator configuration + ansible.builtin.template: + src: zram-generator.conf.j2 + dest: /etc/systemd/zram-generator.conf + owner: root + group: root + mode: '0644' + become: true + register: zram_swap_config + + - name: Reload the unit generators so the config is picked up + ansible.builtin.systemd: + daemon_reload: true + become: true + when: + - zram_swap_config is changed + - not ansible_check_mode + + # `started`, never `restarted`. Restarting means swapoff, which pages every + # byte held in the device back into RAM -- on a host under memory pressure + # that is how applying a config change OOMs a build box. A size change + # therefore takes effect at the next reboot, which the README states. + - name: Start the zram device + ansible.builtin.systemd: + name: systemd-zram-setup@zram0.service + state: started + become: true + when: not ansible_check_mode + + # One swapon answers both questions below: whether zram came up, and + # whether the host has swap this role does not own. + - name: Read the active swap devices + ansible.builtin.command: + cmd: swapon --show=NAME,SIZE,USED --noheadings + changed_when: false + failed_when: false + check_mode: false + register: zram_swap_verify + + # Skipped entirely where the host has swap this role did not create: 180 is + # right for zram, and pushes anonymous pages onto a disk anywhere else. + - name: Set swappiness for zram-backed swap + ansible.builtin.copy: + dest: /etc/sysctl.d/zzz-50-hyperi-zram.conf + owner: root + group: root + mode: '0644' + content: | + # Deployed by hyperi-developer (zram_swap role). Edits are overwritten. + # zram reclaim costs a compress, not a disk seek, so anonymous memory + # is cheaper to reclaim than page cache and the stock 60 has the + # balance backwards. Above 100 needs kernel 5.8+. + vm.swappiness = {{ zram_swap_swappiness }} + become: true + register: zram_swap_sysctl + when: + - zram_swap_swappiness | string | length > 0 + - zram_swap_verify.stdout_lines | default([]) + | reject('search', '^/dev/zram') | select('search', '\S') + | list | length == 0 + + # This file only. `sysctl --system` re-applies every drop-in on the host, + # which would revert an admin's live tuning of something unrelated. + - name: Load the swappiness setting + ansible.builtin.command: + cmd: sysctl -p /etc/sysctl.d/zzz-50-hyperi-zram.conf + become: true + changed_when: true + when: + - zram_swap_sysctl is changed + - not ansible_check_mode + + # The role's whole purpose is that a throttled cgroup has somewhere to + # reclaim to. If no device came up that is not true, and the governor's + # memory limits behave as if this role had never run. + - name: Report the zram swap state + ansible.builtin.debug: + msg: "{{ zram_swap_verify.stdout_lines | default(['no swap active']) }}" + + - name: Record that zram did not come up + # noqa: var-naming[no-role-prefix] -- deploy_warnings is the shared + # accumulator reported by playbooks/main.yml post_tasks. + ansible.builtin.set_fact: + deploy_warnings: >- + {{ deploy_warnings | default([]) + + ['zram_swap: no zram device is active after the run. Memory' + ~ ' throttling on this host still has nothing to reclaim but' + ~ ' page cache, so MemoryHigh stalls rather than slows.'] }} + when: + - not ansible_check_mode + - zram_swap_verify.stdout_lines | default([]) + | select('search', '^/dev/zram') | list | length == 0 + + rescue: + - name: Record that zram swap did not deploy + # noqa: var-naming[no-role-prefix] -- deploy_warnings is the shared + # accumulator reported by playbooks/main.yml post_tasks. + ansible.builtin.set_fact: + deploy_warnings: >- + {{ deploy_warnings | default([]) + + ['zram swap: ' ~ (ansible_failed_task.name | default('unknown task')) + ~ ' -- ' ~ (ansible_failed_result.msg | default('no message'))] }} + +# ============================================================================ +# Opt-out tombstone -- a host that turns zram off gets it removed, not merely +# stopped, so the next boot does not bring it back. +# ============================================================================ + +- name: Remove zram swap (opt-out) + when: + - not (zram_swap_enabled | bool) + - ansible_facts['system'] == 'Linux' + become: true + block: + - name: Stop the zram device + ansible.builtin.systemd: + name: systemd-zram-setup@zram0.service + state: stopped + failed_when: false + when: not ansible_check_mode + + - name: Remove the zram configuration + ansible.builtin.file: + path: /etc/systemd/zram-generator.conf + state: absent + register: zram_swap_conf_removed + + # Two tasks rather than one loop: the deploy path leaves swappiness alone on + # a host with disk swap, so the opt-out may only reset it where the drop-in + # actually existed -- otherwise it would clobber an admin's own setting. + - name: Remove the swappiness drop-in + ansible.builtin.file: + path: /etc/sysctl.d/zzz-50-hyperi-zram.conf + state: absent + register: zram_swap_sysctl_removed + + # Removing the drop-in does not lower the running value -- sysctl re-applies + # files that exist, it never resets one nothing sets. `-w` rather than + # another drop-in, so opting out leaves no file behind. + - name: Restore the kernel default swappiness + ansible.builtin.command: + cmd: sysctl -w vm.swappiness=60 + changed_when: true + when: + - zram_swap_sysctl_removed is changed + - not ansible_check_mode + + - name: Reload the unit generators after removing the configuration + ansible.builtin.systemd: + daemon_reload: true + when: + - zram_swap_conf_removed is changed + - not ansible_check_mode diff --git a/ansible/roles/zram_swap/templates/zram-generator.conf.j2 b/ansible/roles/zram_swap/templates/zram-generator.conf.j2 new file mode 100644 index 0000000..4fdc236 --- /dev/null +++ b/ansible/roles/zram_swap/templates/zram-generator.conf.j2 @@ -0,0 +1,10 @@ +# Deployed by hyperi-developer (zram_swap role). Edits are overwritten. +# +# The generator reads the FIRST file it finds across /run, /etc, /usr/local/lib +# and /usr/lib, so this file REPLACES the distro's default rather than merging +# with it. That is deliberate: Fedora's zram-generator-defaults ships a +# `host-memory-limit`, and a host above that limit gets NO zram device at all +# and no error saying so. Not setting the key here means no such ceiling. +[zram0] +zram-size = {{ zram_swap_size }} +compression-algorithm = {{ zram_swap_compression }} diff --git a/docs/rust-build-governor.md b/docs/rust-build-governor.md new file mode 100644 index 0000000..2192919 --- /dev/null +++ b/docs/rust-build-governor.md @@ -0,0 +1,84 @@ +# Bounding concurrent Rust builds + +`hyperi-rust-govern` is installed by the `developer-rust` role as +`~/.local/bin/cargo`, ahead of the real cargo on PATH, so a developer or an +agent who knows none of this runs `cargo build` and is governed. It places the +build in `rust-build.slice` and holds one of N build slots for the build's +lifetime. + +## How N is chosen + +**N is a memory semaphore, not a queue length.** Memory is spent per *crate*, +not per job: one enormous crate compiles as a single `rustc` however high `-j` +goes, so what is worth bounding is how many builds are resident at once. + + N = min(MemoryHigh / per-build allowance, cores-minus-reserve / 2) + jobs = cores-minus-reserve / N (floored at 2) + +Both are computed by the shim at run time, from the cgroup memory limit where +one is set and `/proc/meminfo` otherwise, reproducing the arithmetic systemd +does for `MemoryHigh=%` against the same total. The slot count and the +memory ceiling therefore cannot drift, and a resized VM needs no re-converge. + +| host RAM | MemoryHigh (50%) | N | jobs each (32 cores) | +|---|---|---|---| +| 256 GB | 128G | 8 | 3 | +| 128 GB | 64G | 4 | 7 | +| 64 GB | 32G | 2 | 15 | +| 32 GB | 16G | 1 | 30 | + +The 32 GB row is a global mutex -- what this was before it was a semaphore -- +and is the check that the model reproduces behaviour known to work. +`rust_governor_slots: 1` pins that everywhere. + +The per-build allowance (`rust_governor_build_allowance_gb`, 14 GB) is the +largest single `rustc` observed plus headroom, taken from one workspace. A +codebase whose memory scales with the job count rather than with one huge crate +wants a different number. + +The CPU reserve comes off the top *before* the remainder is divided, so the +desktop keeps its share at every slot count. No CPU quota is set anywhere: a +quota wastes cores whenever a session idles, while `CPUWeight` gives +proportional share under contention and the whole box when nothing competes. + +## At saturation + +**A build degrades, it is not released.** A build that cannot get a slot within +`rust_governor_lock_wait_seconds` proceeds at the floor job count -- and on +Linux with a live user manager, inside the slice under its own `MemoryMax` of +one allowance. Waiting the full timeout +and then building unbounded would drop the limit at exactly the moment +contention is highest. + +**On Linux a slot is released by the kernel, not by cleanup.** It is an +`flock` on an open file description, so a killed or OOM-killed build frees its +slot with no reaper and no stale-lock detection. The fd is closed for the build +itself (`9>&-`), so a daemon the build starts cannot inherit the slot. macOS +ships no `flock(1)`, so there a slot is a directory held by an exit trap with a +liveness check on the recorded pid: a SIGKILLed holder is reclaimed by the next +arrival rather than by the kernel. + +A build that starts alone keeps the crowded job count: `CARGO_BUILD_JOBS` is +fixed when the process starts. For a known-solo run, pass `CARGO_BUILD_JOBS` +yourself -- the shim respects a caller's value over its own. + +## What it needs + +**Swap.** `MemoryHigh` throttles by reclaim, and on a swapless host the only +reclaimable memory is page cache -- so a build past the line stalls rather than +slows. The `zram_swap` role is the other half, and a converge onto a swapless +host says so in its warnings. + +**Incremental compilation left alone, by default.** sccache refuses to cache +any `rustc` call carrying `-C incremental`, so `rust_governor_no_incremental` +can turn incremental off to make builds cacheable -- but the win is narrower +than it looks. Cargo never builds *dependencies* incrementally, so those were +always cacheable; what this buys is caching of *workspace* crates, and it pays +by losing incremental on those same crates. Edit a large single-`rustc` crate +and sccache misses on the changed source with nothing to fall back on. + +Turn it on for a build box, a CI runner, or a workstation running many sessions +against the same workspaces, where builds start from clean trees and there is no +incremental state to lose. Opting out by hand is +`HYPERI_RUST_GOVERN_NO_INCREMENTAL=0`, not `CARGO_INCREMENTAL=1` -- the latter +makes sccache refuse the build outright. diff --git a/tools/ci/run-tests.sh b/tools/ci/run-tests.sh index f3983c6..cd9a4fe 100755 --- a/tools/ci/run-tests.sh +++ b/tools/ci/run-tests.sh @@ -55,7 +55,8 @@ fi if have shellcheck; then ran=$((ran + 1)) echo "==> shellcheck" - if shellcheck install.sh tools/ci/run-tests.sh tools/hyperi-doctor; then + if shellcheck install.sh tools/ci/run-tests.sh tools/hyperi-doctor \ + ansible/roles/developer-rust/files/hyperi-rust-govern; then echo " ok" else echo " FAILED" From dd2d7fe521b32408b73bae64605b9c7ea966ac44 Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 4 Sep 2026 08:43:53 +1000 Subject: [PATCH 2/4] fix(tests): snapshot-reset harness for Proxmox VMs Three playbooks under tests/proxmox/, driven by hand, not CI. create.yml clones the base template into the test range, sizes it to the lab's other test machines, gives it a static address or DHCP, boots it once to bring every package current, then stops and snapshots the VM as `clean`. hyperi-developer never runs there: the snapshot is the state every test starts from. reset.yml rolls a VM back to that snapshot and starts it, or with start=false leaves it stopped, which is the resting state of a test VM. delete.yml removes one. All three refuse a vmid outside the range, which is the safety mechanism .env.sample had promised and nothing had implemented. A static VM is reached by its address and a DHCP VM by the name the lab DNS registers for it, so nothing waits on a DNS entry that may not exist yet. The free-vmid picker now sorts: difference() is a set operation and its first element was arbitrary. Authentication is the API token hyperi-infra's tools use, not a root password, and the modules are community.proxmox: the community.general proxmox_* modules are deprecated and removed at 15.0.0, which the existing floor reaches. Login to the VM is by key, because cloud images ship sshd with password login off. The shared loader in tests/common/setup_env.yml ran `source .env` under /bin/sh, which is dash on Debian and Ubuntu and has no `source`. The failure was silent - the pipeline's exit status is jq's - so `env` returned whatever PROXMOX_* the operator's shell already carried, and a token from the environment quietly stood in for the file's. It now runs under bash with set -euo pipefail. provision.yml, which shares the loader, only ever worked by the same accident. create.yml adopts a VM of the same name so a run that stopped partway is finished rather than doubled. Proven against Proxmox: a VM created, rolled back, started, converged and returned to its snapshot stopped. --- ansible/requirements.yml | 6 + ansible/tests/.env.sample | 58 +++- ansible/tests/common/setup_env.yml | 8 + ansible/tests/proxmox/README.md | 61 +++++ ansible/tests/proxmox/create.yml | 247 ++++++++++++++++++ ansible/tests/proxmox/delete.yml | 58 ++++ ansible/tests/proxmox/reset.yml | 145 ++++++++++ .../tests/proxmox/tasks/require_test_vmid.yml | 12 + 8 files changed, 592 insertions(+), 3 deletions(-) create mode 100644 ansible/tests/proxmox/README.md create mode 100644 ansible/tests/proxmox/create.yml create mode 100644 ansible/tests/proxmox/delete.yml create mode 100644 ansible/tests/proxmox/reset.yml create mode 100644 ansible/tests/proxmox/tasks/require_test_vmid.yml diff --git a/ansible/requirements.yml b/ansible/requirements.yml index 4920387..2408bc7 100644 --- a/ansible/requirements.yml +++ b/ansible/requirements.yml @@ -19,5 +19,11 @@ collections: # Required by the molecule docker driver, not by any role. - name: community.docker version: ">=5.0.0" + + # tests/proxmox/create.yml and reset.yml. The proxmox_* modules left + # community.general for this collection and are removed from it at 15.0.0, + # which the floor above will reach. + - name: community.proxmox + version: ">=1.0.0" - name: ansible.posix version: ">=2.0.0" diff --git a/ansible/tests/.env.sample b/ansible/tests/.env.sample index 195f3d4..c1d3fd5 100644 --- a/ansible/tests/.env.sample +++ b/ansible/tests/.env.sample @@ -7,10 +7,18 @@ # ============================================================================= # Your Proxmox server connection details PROXMOX_HOST=proxmox.example.com -PROXMOX_USER=root@pam -PROXMOX_PASSWORD= # Optional - leave empty if using SSH keys PROXMOX_NODE=pve -PROXMOX_API_VERIFY_SSL=false +PROXMOX_API_VERIFY_SSL=true + +# An API token, the same one hyperi-infra's tools use: user@realm!name plus +# its secret, made under Datacenter > Permissions > API Tokens. The playbooks +# in tests/proxmox/ authenticate with these two and nothing else. +PROXMOX_TOKEN_ID=hyperi-vm@pve!cli-token +PROXMOX_TOKEN_SECRET= + +# root@pam password. Only the older provision.yml still reads these. +PROXMOX_USER=root@pam +PROXMOX_PASSWORD= # Fedora Test VM PROXMOX_FEDORA_VMID=2005 @@ -28,6 +36,50 @@ PROXMOX_UBUNTU_USER=dfe PROXMOX_UBUNTU_PASSWORD=dfe # Optional - omit from inventory if using SSH keys PROXMOX_UBUNTU_SSH_PORT=22 +# ----------------------------------------------------------------------------- +# TEST VMS (tests/proxmox/create.yml and reset.yml) +# ----------------------------------------------------------------------------- +# Snapshot-reset testing: create.yml clones a base template into the range +# below, sizes it, brings it current, stops it and snapshots it. reset.yml +# rolls any VM in the range back to that snapshot. delete.yml removes one. +# All three refuse a vmid outside the range, so a typo cannot roll back, +# overwrite or delete a real machine. +PROXMOX_TEST_VMID_MIN=8100 +PROXMOX_TEST_VMID_MAX=8199 + +# The base (not desktop) template: a clean cloud image with cloud-init and +# nothing else. The desktop template bakes hyperi-developer in, which is the +# thing under test. +PROXMOX_TEST_TEMPLATE_VMID=9041 + +# Size and placement of a VM create.yml makes, matched to the lab's other test +# machines. STORAGE is where the full clone lands; the disk is grown to DISK +# and cloud-init grows the root filesystem at boot. Leave one empty to keep +# the template's value. +PROXMOX_TEST_CORES=8 +PROXMOX_TEST_MEMORY=16384 +PROXMOX_TEST_DISK=100G +PROXMOX_TEST_STORAGE= + +# The cloud-init search domain. The lab DNS registers . for +# every DHCP lease, which is how a VM made without -e ip is reached. +PROXMOX_TEST_DOMAIN=example.com + +# Static addressing, used when create.yml is given -e ip=/. +PROXMOX_TEST_GATEWAY=192.0.2.1 +PROXMOX_TEST_NAMESERVER= + +# Login that create.yml sets through cloud-init, and that reset.yml writes +# into the inventory. A VM made some other way passes its own with +# -e test_user= -e test_password= -e snapshot=. +PROXMOX_TEST_USER=hyperi +PROXMOX_TEST_PASSWORD= + +# The private key Ansible connects with; its .pub is what cloud-init installs +# for that user. Cloud images ship sshd with password login off, so the +# password above serves the console and sudo only. An absolute path. +PROXMOX_TEST_SSH_KEY=/home/you/.ssh/devex-ssh + # ----------------------------------------------------------------------------- # CLONE-TEST (emulates the auto desktop image cycle: clone -> test -> delete) # ----------------------------------------------------------------------------- diff --git a/ansible/tests/common/setup_env.yml b/ansible/tests/common/setup_env.yml index fa42e7c..8cbe692 100644 --- a/ansible/tests/common/setup_env.yml +++ b/ansible/tests/common/setup_env.yml @@ -30,12 +30,20 @@ Press Enter when done editing... when: not env_file_stat.stat.exists +# bash, not /bin/sh: on Debian and Ubuntu /bin/sh is dash, which has no +# `source`. The failure was silent -- the pipeline's exit status is jq's -- so +# `env` returned whatever PROXMOX_* the ambient shell already carried, and a +# token from the operator's environment quietly stood in for the file's. +# `set -e` and pipefail make a missing or unreadable .env fail the task. - name: Load environment variables from .env (shell format) ansible.builtin.shell: | + set -euo pipefail set -a source "{{ tests_dir }}/.env" set +a env | grep -E '^(PROXMOX|ANSIBLE)_' | jq -R 'split("=") | {(.[0]): .[1:] | join("=")}' | jq -s 'add' + args: + executable: /bin/bash register: env_raw changed_when: false diff --git a/ansible/tests/proxmox/README.md b/ansible/tests/proxmox/README.md new file mode 100644 index 0000000..976d87f --- /dev/null +++ b/ansible/tests/proxmox/README.md @@ -0,0 +1,61 @@ +# Snapshot-reset testing against Proxmox VMs + +Run hyperi-developer against a real VM from a known clean state, as many times +as needed, by hand. Not CI: it needs a Proxmox endpoint and credentials. + +Configuration is the `TEST VMS` block in `tests/.env` (copy +`tests/.env.sample`). Authentication is the same Proxmox API token +hyperi-infra's tools use -- `PROXMOX_TOKEN_ID` and `PROXMOX_TOKEN_SECRET`, +never a root password. Login to the VM itself is by key: cloud-init installs +the `.pub` of `PROXMOX_TEST_SSH_KEY` for the test user, because cloud images +ship sshd with password login off. Everything here runs from `ansible/`. + +The modules need `proxmoxer` on the machine running the playbook, for the +Python that Ansible uses: `sudo apt install python3-proxmoxer` on Debian and +Ubuntu, or pip inside whatever venv runs Ansible. Then +`ansible-galaxy collection install -r requirements.yml` for `community.proxmox`. + +## Make a clean VM once + + ansible-playbook tests/proxmox/create.yml -e vm_name=ubuntu-test.example.com \ + -e vmid=8101 -e ip=192.0.2.51/24 + +Clones the base template, sizes it from `.env` (cores, memory, disk, +storage), gives it a static address or DHCP, boots it once to bring every +package current, **stops it**, and snapshots it as `clean`. hyperi-developer +never runs here: the snapshot is the state every test starts from. + +The VM name is what Proxmox and the lab DNS see, so name a static VM by its +FQDN. Leave off `-e vmid` for the lowest free id in the range and `-e ip` for +DHCP, where the lab DNS names the VM `.`. + +The base template, not the desktop one: the desktop template bakes +hyperi-developer in, which is the thing under test. + +## Test + + ansible-playbook tests/proxmox/reset.yml -e vmid=8101 + ansible-playbook -i tests/proxmox/inventory_proxmox.yml playbooks/main.yml --tags developer-rust + ansible-playbook tests/proxmox/reset.yml -e vmid=8101 -e start=false + +`reset.yml` stops the VM, rolls it back to `clean`, starts it, waits for sshd, +and writes `inventory_proxmox.yml` for it. The last line is the resting state: +a test VM is off except while a test runs, so finish by rolling it back again +and leaving it stopped. + +A VM made some other way -- the Fedora box, say -- passes its own snapshot +name and login: `-e snapshot=initial_build -e test_user=dfe -e test_password=dfe`. + +## Remove + + ansible-playbook tests/proxmox/delete.yml -e vmid=8102 + +## Safety + +All three playbooks refuse a vmid outside `PROXMOX_TEST_VMID_MIN..MAX`. A +rollback or delete discards everything on the machine, so that range is the +only place it can happen. + +`provision.yml` and `test_all.yml` predate this and target a fixed +Fedora+Ubuntu pair; `test_all.yml` also installs from a branch that no longer +exists. Prefer the playbooks above. diff --git a/ansible/tests/proxmox/create.yml b/ansible/tests/proxmox/create.yml new file mode 100644 index 0000000..f7eb189 --- /dev/null +++ b/ansible/tests/proxmox/create.yml @@ -0,0 +1,247 @@ +--- +# Create a clean test VM from a base template and snapshot it, so reset.yml +# can return it to this exact state before every hyperi-developer run. +# +# The VM is cloned, sized, booted once to bring its packages current, then +# STOPPED and snapshotted. hyperi-developer never runs here: the snapshot IS +# the "before" state. +# +# ansible-playbook tests/proxmox/create.yml -e vm_name=ubuntu-test.example.com \ +# -e vmid=8101 -e ip=192.0.2.51/24 +# ansible-playbook tests/proxmox/create.yml -e vm_name=scratch +# +# Without -e ip the VM takes DHCP. Size, storage, gateway and nameserver come +# from the TEST VMS block in tests/.env -- see tests/.env.sample. + +- name: Clone a base template into the test range + hosts: localhost + connection: local + gather_facts: false + vars: + snapshot: clean + vmid: "" + ip: "" + module_defaults: + community.proxmox.proxmox_vm_info: &api + api_host: "{{ env.PROXMOX_HOST }}" + # A token, never a password. PROXMOX_TOKEN_ID is user@realm!name as + # hyperi-infra stores it; the modules take the two halves separately. + api_user: "{{ env.PROXMOX_TOKEN_ID.split('!')[0] }}" + api_token_id: "{{ env.PROXMOX_TOKEN_ID.split('!')[1] }}" + api_token_secret: "{{ env.PROXMOX_TOKEN_SECRET }}" + validate_certs: "{{ env.PROXMOX_API_VERIFY_SSL | default('true') | bool }}" + community.proxmox.proxmox_kvm: *api + community.proxmox.proxmox_disk: *api + community.proxmox.proxmox_snap: *api + tasks: + - name: Require a VM name + ansible.builtin.assert: + that: vm_name is defined and vm_name | length > 0 + fail_msg: "pass -e vm_name=" + quiet: true + + - name: Load tests/.env + ansible.builtin.include_tasks: + file: ../common/setup_env.yml + + # A static VM is reached by its address, so nothing here waits on DNS. A + # DHCP VM is reached by the name the lab DNS registers for its lease: the + # VM name with the domain appended, unless it is already a FQDN. + - name: Resolve the address Ansible reaches the VM on + ansible.builtin.set_fact: + vm_host: >- + {{ ip | regex_replace('/.*$', '') if ip | length > 0 else + (vm_name if '.' in vm_name else vm_name ~ '.' ~ env.PROXMOX_TEST_DOMAIN) }} + + - name: List the VMs on the node + community.proxmox.proxmox_vm_info: + node: "{{ env.PROXMOX_NODE }}" + register: proxmox_existing + + # A VM of this name already on the node is a run that stopped partway: the + # clone succeeded and something after it did not. Finish it rather than + # clone a second one beside it. + - name: Adopt a VM of this name if one exists + ansible.builtin.set_fact: + proxmox_adopted: >- + {{ proxmox_existing.proxmox_vms | selectattr('name', 'eq', vm_name) + | map(attribute='vmid') | map('int') | list }} + proxmox_used: "{{ proxmox_existing.proxmox_vms | map(attribute='vmid') | map('int') | list }}" + + # difference() is a set operation and returns its result unordered, so the + # sort is what makes "the lowest free id" true. + - name: Pick the vmid + ansible.builtin.set_fact: + vmid: >- + {{ proxmox_adopted[0] if proxmox_adopted | length > 0 else + (vmid if vmid | string | length > 0 else + (range(env.PROXMOX_TEST_VMID_MIN | int, env.PROXMOX_TEST_VMID_MAX | int + 1) | list + | difference(proxmox_used) | sort | first)) }} + + # A vmid passed by hand is held to the same range, so a typo cannot land + # on a real machine. + - name: Refuse a vmid outside the test range or already in use + ansible.builtin.assert: + that: + - vmid | int >= env.PROXMOX_TEST_VMID_MIN | int + - vmid | int <= env.PROXMOX_TEST_VMID_MAX | int + - proxmox_adopted | length > 0 or vmid | int not in proxmox_used + fail_msg: >- + vmid {{ vmid }} is outside {{ env.PROXMOX_TEST_VMID_MIN }}-{{ env.PROXMOX_TEST_VMID_MAX }} + or already in use + quiet: true + + - name: Clone the base template + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + clone: "{{ env.PROXMOX_TEST_TEMPLATE_VMID }}" + vmid: "{{ env.PROXMOX_TEST_TEMPLATE_VMID }}" + newid: "{{ vmid }}" + name: "{{ vm_name }}" + full: true + storage: "{{ env.PROXMOX_TEST_STORAGE | default(omit, true) }}" + timeout: 600 + state: present + when: proxmox_adopted | length == 0 + + # The base template carries no user, no network and the smallest size that + # boots. A key as well as a password: cloud images ship sshd with password + # login off, so the password serves the console and sudo, and the key is + # how Ansible gets in. Applied to an adopted VM too -- it is idempotent, + # and cloud-init picks a changed config up at the next boot. + - name: Set the size, cloud-init user, key and network + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + cores: "{{ env.PROXMOX_TEST_CORES | default(omit, true) }}" + memory: "{{ env.PROXMOX_TEST_MEMORY | default(omit, true) }}" + ostype: l26 + ciuser: "{{ env.PROXMOX_TEST_USER }}" + cipassword: "{{ env.PROXMOX_TEST_PASSWORD }}" + sshkeys: "{{ lookup('file', env.PROXMOX_TEST_SSH_KEY ~ '.pub') }}" + nameservers: "{{ [env.PROXMOX_TEST_NAMESERVER] if env.PROXMOX_TEST_NAMESERVER | default('') | length > 0 else omit }}" + searchdomains: ["{{ env.PROXMOX_TEST_DOMAIN }}"] + ipconfig: + ipconfig0: "{{ 'ip=' ~ ip ~ ',gw=' ~ env.PROXMOX_TEST_GATEWAY if ip | length > 0 else 'ip=dhcp' }}" + update: true + register: proxmox_cloudinit + + - name: Read the disk size + community.proxmox.proxmox_vm_info: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + config: current + register: proxmox_config + + # Grown, never shrunk: the API refuses a shrink, so an adopted VM already + # at size is left alone. cloud-init grows the root filesystem at boot. + - name: Grow the disk + community.proxmox.proxmox_disk: + vmid: "{{ vmid }}" + disk: scsi0 + size: "{{ env.PROXMOX_TEST_DISK }}" + state: resized + when: + - env.PROXMOX_TEST_DISK | default('') | length > 0 + - proxmox_config.proxmox_vms[0].config.scsi0 is not search('size=' ~ env.PROXMOX_TEST_DISK ~ '(,|$)') + register: proxmox_disk + + # A changed cloud-init config applies only at boot, so a running adopted VM + # is bounced. A fresh clone is already stopped and this is a no-op. + - name: Stop the VM so a cloud-init change applies at boot + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + state: stopped + force: true + timeout: 300 + when: proxmox_cloudinit is changed or proxmox_disk is changed + + - name: Start the VM + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + state: started + + # Gated on sshd answering. The timeout is a backstop for a VM that never + # came up, not a timer to race. + - name: Wait for SSH + ansible.builtin.wait_for: + host: "{{ vm_host }}" + port: 22 + timeout: 600 + + - name: Register the VM for the upgrade play + ansible.builtin.add_host: + name: "{{ vm_host }}" + groups: new_vm + ansible_user: "{{ env.PROXMOX_TEST_USER }}" + ansible_ssh_private_key_file: "{{ env.PROXMOX_TEST_SSH_KEY }}" + ansible_become_password: "{{ env.PROXMOX_TEST_PASSWORD }}" + # A fresh clone has a fresh host key every time. IdentitiesOnly: the + # client would otherwise offer every agent key first and exhaust + # sshd's MaxAuthTries before reaching this one. + ansible_ssh_common_args: -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes + +- name: Bring the base image current + hosts: new_vm + become: true + gather_facts: true + tasks: + # sshd answers before cloud-init has finished its own package work on the + # first boot, and the two would fight over the dpkg lock. This blocks on + # cloud-init's own completion signal. rc 2 is "done with recoverable + # errors", which a fresh image reports for warnings that do not matter here. + - name: Wait for cloud-init to finish + ansible.builtin.command: cloud-init status --wait + register: proxmox_cloud_init + changed_when: false + failed_when: proxmox_cloud_init.rc not in [0, 2] + + - name: Upgrade every package (Debian family) + ansible.builtin.apt: + update_cache: true + upgrade: dist + when: ansible_facts['os_family'] == 'Debian' + + # The whole point of this task is "everything to latest", which is the one + # case the package-latest rule exists to question. + - name: Upgrade every package (RedHat family) # noqa: package-latest + ansible.builtin.dnf: + name: "*" + state: latest + when: ansible_facts['os_family'] == 'RedHat' + +- name: Stop and snapshot the clean state + hosts: localhost + connection: local + gather_facts: false + # Play vars do not cross plays; -e still overrides both. + vars: + snapshot: clean + module_defaults: + community.proxmox.proxmox_kvm: *api + community.proxmox.proxmox_snap: *api + tasks: + # Stopped first. A snapshot of a running VM without vmstate is a crash + # image, and every rollback to it is then a hard reset. + - name: Stop the VM + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + state: stopped + force: true + timeout: 300 + + - name: Snapshot the clean state + community.proxmox.proxmox_snap: + vmid: "{{ vmid }}" + snapname: "{{ snapshot }}" + vmstate: false + state: present + + - name: Report + ansible.builtin.debug: + msg: >- + {{ vm_name }} is vmid {{ vmid }}, stopped, snapshot '{{ snapshot }}'. + Reset it with: ansible-playbook tests/proxmox/reset.yml -e vmid={{ vmid }} diff --git a/ansible/tests/proxmox/delete.yml b/ansible/tests/proxmox/delete.yml new file mode 100644 index 0000000..e3cd842 --- /dev/null +++ b/ansible/tests/proxmox/delete.yml @@ -0,0 +1,58 @@ +--- +# Delete a test VM and its disks. Refuses a vmid outside the test range. +# +# ansible-playbook tests/proxmox/delete.yml -e vmid=8102 + +- name: Delete a test VM + hosts: localhost + connection: local + gather_facts: false + module_defaults: + community.proxmox.proxmox_vm_info: &api + api_host: "{{ env.PROXMOX_HOST }}" + # A token, never a password. PROXMOX_TOKEN_ID is user@realm!name as + # hyperi-infra stores it; the modules take the two halves separately. + api_user: "{{ env.PROXMOX_TOKEN_ID.split('!')[0] }}" + api_token_id: "{{ env.PROXMOX_TOKEN_ID.split('!')[1] }}" + api_token_secret: "{{ env.PROXMOX_TOKEN_SECRET }}" + validate_certs: "{{ env.PROXMOX_API_VERIFY_SSL | default('true') | bool }}" + community.proxmox.proxmox_kvm: *api + tasks: + - name: Require a vmid + ansible.builtin.assert: + that: vmid is defined and vmid | string | length > 0 + fail_msg: "pass -e vmid=" + quiet: true + + - name: Load tests/.env + ansible.builtin.include_tasks: + file: ../common/setup_env.yml + + - name: Refuse a vmid outside the test range + ansible.builtin.include_tasks: + file: tasks/require_test_vmid.yml + + - name: Look up the VM + community.proxmox.proxmox_vm_info: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + register: proxmox_vm + + - name: Refuse a vmid that does not exist + ansible.builtin.assert: + that: proxmox_vm.proxmox_vms | length == 1 + fail_msg: "no VM with vmid {{ vmid }} on {{ env.PROXMOX_NODE }}" + quiet: true + + # force: a running VM is stopped first rather than refused. + - name: Delete the VM + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + state: absent + force: true + timeout: 300 + + - name: Report + ansible.builtin.debug: + msg: "vmid {{ vmid }} ({{ proxmox_vm.proxmox_vms[0].name }}) deleted." diff --git a/ansible/tests/proxmox/reset.yml b/ansible/tests/proxmox/reset.yml new file mode 100644 index 0000000..f98865a --- /dev/null +++ b/ansible/tests/proxmox/reset.yml @@ -0,0 +1,145 @@ +--- +# Return a test VM to its clean snapshot, start it, and write an inventory for +# it, so hyperi-developer can be run against a known state and then the run +# thrown away. +# +# ansible-playbook tests/proxmox/reset.yml -e vmid=8101 +# ansible-playbook -i tests/proxmox/inventory_proxmox.yml playbooks/main.yml --tags developer-rust +# ansible-playbook tests/proxmox/reset.yml -e vmid=8101 -e start=false +# +# A test VM is off except while a test runs. The last line returns it to the +# snapshot and leaves it stopped. +# +# Snapshot name, login user and password default to the TEST VMS block in +# tests/.env and can be overridden per VM: +# +# ansible-playbook tests/proxmox/reset.yml -e vmid=8100 -e snapshot=initial_build \ +# -e test_user=dfe -e test_password=dfe + +- name: Reset a test VM to its clean snapshot + hosts: localhost + connection: local + gather_facts: false + vars: + start: true + module_defaults: + community.proxmox.proxmox_vm_info: &api + api_host: "{{ env.PROXMOX_HOST }}" + # A token, never a password. PROXMOX_TOKEN_ID is user@realm!name as + # hyperi-infra stores it; the modules take the two halves separately. + api_user: "{{ env.PROXMOX_TOKEN_ID.split('!')[0] }}" + api_token_id: "{{ env.PROXMOX_TOKEN_ID.split('!')[1] }}" + api_token_secret: "{{ env.PROXMOX_TOKEN_SECRET }}" + validate_certs: "{{ env.PROXMOX_API_VERIFY_SSL | default('true') | bool }}" + community.proxmox.proxmox_kvm: *api + community.proxmox.proxmox_snap: *api + tasks: + - name: Require a vmid + ansible.builtin.assert: + that: vmid is defined and vmid | string | length > 0 + fail_msg: "pass -e vmid=" + quiet: true + + - name: Load tests/.env + ansible.builtin.include_tasks: + file: ../common/setup_env.yml + + - name: Resolve the per-VM settings + ansible.builtin.set_fact: + snapshot: "{{ snapshot | default('clean', true) }}" + test_user: "{{ test_user | default(env.PROXMOX_TEST_USER, true) }}" + test_password: "{{ test_password | default(env.PROXMOX_TEST_PASSWORD, true) }}" + + - name: Refuse a vmid outside the test range + ansible.builtin.include_tasks: + file: tasks/require_test_vmid.yml + + - name: Look up the VM + community.proxmox.proxmox_vm_info: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + config: current + register: proxmox_vm + + - name: Refuse a vmid that does not exist + ansible.builtin.assert: + that: proxmox_vm.proxmox_vms | length == 1 + fail_msg: "no VM with vmid {{ vmid }} on {{ env.PROXMOX_NODE }}" + quiet: true + + # A static VM is reached by its address, so nothing here waits on DNS. A + # DHCP VM is reached by the name the lab DNS registers for its lease: the + # VM name with the domain appended, unless it is already a FQDN. + - name: Resolve the address Ansible reaches the VM on + ansible.builtin.set_fact: + vm_host: >- + {{ proxmox_static[0] if proxmox_static | length > 0 else + (proxmox_name if '.' in proxmox_name else proxmox_name ~ '.' ~ env.PROXMOX_TEST_DOMAIN) }} + vars: + proxmox_name: "{{ proxmox_vm.proxmox_vms[0].name }}" + proxmox_static: "{{ proxmox_vm.proxmox_vms[0].config.ipconfig0 | default('') | regex_findall('ip=([0-9.]+)/') }}" + + # Stopped before rollback. A rollback of a running VM to a snapshot taken + # without vmstate is a hard reset, and the stop makes the ordering explicit + # rather than incidental. + - name: Stop the VM + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + state: stopped + force: true + timeout: 300 + + - name: Roll back to the snapshot + community.proxmox.proxmox_snap: + vmid: "{{ vmid }}" + snapname: "{{ snapshot }}" + state: rollback + + - name: Start the VM + community.proxmox.proxmox_kvm: + node: "{{ env.PROXMOX_NODE }}" + vmid: "{{ vmid }}" + state: started + when: start | bool + + - name: Wait for SSH + ansible.builtin.wait_for: + host: "{{ vm_host }}" + port: 22 + timeout: 600 + when: start | bool + + - name: Drop the inventory of a VM left stopped + ansible.builtin.file: + path: "{{ playbook_dir }}/inventory_proxmox.yml" + state: absent + when: not start | bool + + # Overwritten on every reset, and gitignored: it carries the login. + - name: Write the inventory + when: start | bool + ansible.builtin.copy: + dest: "{{ playbook_dir }}/inventory_proxmox.yml" + mode: '0600' + content: | + [test_vm] + {{ vm_host }} + + [test_vm:vars] + ansible_user={{ test_user }} + ansible_ssh_private_key_file={{ env.PROXMOX_TEST_SSH_KEY }} + ansible_become_password={{ test_password }} + # A rolled-back VM presents the same host key every time, but a + # recreated one does not, and this file is thrown away with it. + # IdentitiesOnly: the client would otherwise offer every agent key + # first and exhaust sshd's MaxAuthTries before reaching this one. + ansible_ssh_common_args=-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o IdentitiesOnly=yes + + - name: Report + ansible.builtin.debug: + msg: >- + vmid {{ vmid }} ({{ proxmox_vm.proxmox_vms[0].name }}) is back at + '{{ snapshot }}' and + {{ 'running at ' ~ vm_host ~ '. Next: ansible-playbook -i tests/proxmox/inventory_proxmox.yml playbooks/main.yml' + if start | bool else 'stopped.' }} diff --git a/ansible/tests/proxmox/tasks/require_test_vmid.yml b/ansible/tests/proxmox/tasks/require_test_vmid.yml new file mode 100644 index 0000000..a026d65 --- /dev/null +++ b/ansible/tests/proxmox/tasks/require_test_vmid.yml @@ -0,0 +1,12 @@ +--- +# The playbooks that include this roll back or delete a VM. The declared test +# range is the only place they may do that. +- name: Refuse a vmid outside the test range + ansible.builtin.assert: + that: + - vmid | int >= env.PROXMOX_TEST_VMID_MIN | int + - vmid | int <= env.PROXMOX_TEST_VMID_MAX | int + fail_msg: >- + vmid {{ vmid }} is outside the test range + {{ env.PROXMOX_TEST_VMID_MIN }}-{{ env.PROXMOX_TEST_VMID_MAX }} + quiet: true From 7d17f23dca4294fcc12a375ddc3f2ca27479c8e8 Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 4 Sep 2026 10:25:28 +1000 Subject: [PATCH 3/4] fix(ansible): load the shared vars for every inventory hyperi_github_token, hyperi_github_headers, hyperi_github_env and hyperi_core_versions lived in inventories/localhost/group_vars, so playbooks/main.yml only worked with that one inventory. Against any other, a test VM or a fleet host, the first task to reference them failed undefined - the Rust role's build-environment step, on a clean VM, after thirty tasks had run. The variables belong to the playbook, not to one inventory: they are read from the control node's environment and name the versions the roles install. Moved to playbooks/group_vars, which Ansible loads beside the playbook for whatever inventory is passed. The localhost install path is unchanged. --- ansible/molecule/existing-host/molecule.yml | 2 +- ansible/{inventories/localhost => playbooks}/group_vars/all.yml | 0 ansible/roles/developer/defaults/main.yml | 2 +- 3 files changed, 2 insertions(+), 2 deletions(-) rename ansible/{inventories/localhost => playbooks}/group_vars/all.yml (100%) diff --git a/ansible/molecule/existing-host/molecule.yml b/ansible/molecule/existing-host/molecule.yml index d4aedb0..f87b771 100644 --- a/ansible/molecule/existing-host/molecule.yml +++ b/ansible/molecule/existing-host/molecule.yml @@ -58,7 +58,7 @@ provisioner: hosts: inventory.yml # The version SSoT. Without it any task reading hyperi_core_versions # (the rustup pin, node_major) is undefined under this scenario. - group_vars: ${MOLECULE_PROJECT_DIRECTORY}/inventories/localhost/group_vars + group_vars: ${MOLECULE_PROJECT_DIRECTORY}/playbooks/group_vars scenario: name: existing-host diff --git a/ansible/inventories/localhost/group_vars/all.yml b/ansible/playbooks/group_vars/all.yml similarity index 100% rename from ansible/inventories/localhost/group_vars/all.yml rename to ansible/playbooks/group_vars/all.yml diff --git a/ansible/roles/developer/defaults/main.yml b/ansible/roles/developer/defaults/main.yml index de589af..98ada28 100644 --- a/ansible/roles/developer/defaults/main.yml +++ b/ansible/roles/developer/defaults/main.yml @@ -54,7 +54,7 @@ docker_fedora_supported_releases: # NodeSource publishes a repo PER MAJOR (node_24.x, node_22.x) and both provide # the package `nodejs`, so a box can hold exactly one system-wide major. That is # the n slot; fnm covers n-1 per user. Majors come from the SSoT -- see -# hyperi_core_versions in inventories/localhost/group_vars/all.yml. +# hyperi_core_versions in playbooks/group_vars/all.yml. # # The deb and rpm repos are signed by DIFFERENT keys, so both URLs are carried. node_major: "{{ hyperi_core_versions.node_major }}" From 75c95b99e2b8aaeecc3958b038e961594259cbf4 Mon Sep 17 00:00:00 2001 From: Derek Date: Fri, 4 Sep 2026 13:24:09 +1000 Subject: [PATCH 4/4] fix(developer-ai): opt-in Codex agent tooling New opt-in role installing the OpenAI Codex CLI as a second opinion alongside Claude Code, plus OpenAI's Codex plugin for Claude Code. Claude Code stays the driver. Selected by --tags developer-ai, the ai group tag, or the codex and codex-plugin tool tags; never pulled by a bare install. Dependencies are probed on the target, per user, rather than declared. Claude Code comes only from soe, so a meta dependency would drag org policy onto a machine that asked for a review tool, and node comes from the developer base rather than developer-node, so the dependency that looks right supplies nothing. A miss skips the component and names the tag that fixes it. Codex installs via the official installer, which verifies the release tarball against codex-package_SHA256SUMS; the thin codex- assets carry no published digest at all. macOS takes the cask, there being no formula. Sign-in stays interactive and is not automated. Splits the Claude Code binary install from the HyperI managed settings, which move behind a claude-policy tag only soe selects. --tags claude used to write /etc/claude-code/ on any machine, which is org policy arriving through a tag that says nothing about policy. install.sh no longer reports success unconditionally. A failed optional component records a warning and the run continues by design, so the exit code is 0; the banner now reads the warning count from the applied-state stamp, and a --check run says nothing was changed. Headroom was built and removed before shipping. It works by pointing ANTHROPIC_BASE_URL at a local proxy, and Claude Code gates capabilities on api.anthropic.com: behind a custom base URL it stops sending the context-1m header and accounts against 200k, disables on-demand tool loading, and disables Remote Control. See docs/install-matrix.md. Verified on Ubuntu: codex-cli 0.153.2 and plugin codex@openai-codex 1.0.6 installed per user, applied-state stamp reports zero warnings. --- README.md | 6 + ansible/playbooks/main.yml | 12 + ansible/roles/developer-ai/defaults/main.yml | 71 +++++ ansible/roles/developer-ai/meta/main.yml | 35 +++ ansible/roles/developer-ai/tasks/codex.yml | 200 ++++++++++++ .../developer-ai/tasks/codex_plugin_cc.yml | 296 ++++++++++++++++++ ansible/roles/developer-ai/tasks/init.yml | 118 +++++++ ansible/roles/developer-ai/tasks/main.yml | 84 +++++ ansible/roles/developer-ai/tasks/verify.yml | 66 ++++ ansible/roles/developer-ai/vars/macos.yml | 10 + .../files/update/hyperi-update-linux.sh | 57 ++++ .../files/update/hyperi-update-macos.sh | 31 +- ansible/roles/soe/tasks/claude.yml | 59 +--- ansible/roles/soe/tasks/claude_policy.yml | 54 ++++ ansible/roles/soe/tasks/main.yml | 12 + docs/install-matrix.md | 112 ++++++- install.sh | 37 ++- 17 files changed, 1199 insertions(+), 61 deletions(-) create mode 100644 ansible/roles/developer-ai/defaults/main.yml create mode 100644 ansible/roles/developer-ai/meta/main.yml create mode 100644 ansible/roles/developer-ai/tasks/codex.yml create mode 100644 ansible/roles/developer-ai/tasks/codex_plugin_cc.yml create mode 100644 ansible/roles/developer-ai/tasks/init.yml create mode 100644 ansible/roles/developer-ai/tasks/main.yml create mode 100644 ansible/roles/developer-ai/tasks/verify.yml create mode 100644 ansible/roles/developer-ai/vars/macos.yml create mode 100644 ansible/roles/soe/tasks/claude_policy.yml diff --git a/README.md b/README.md index 87811c7..f41eb10 100644 --- a/README.md +++ b/README.md @@ -169,6 +169,11 @@ under `become` -- so treat it exactly as you would the playbook. ## What Gets Installed +The summary below is the shape of it. [docs/install-matrix.md](docs/install-matrix.md) +is the single source of truth for the detail: every role, group, persona and +tool, which tag selects it, how each one is fetched, and which of them verify a +digest. Read that before changing a role or adding a tool. + **Default** (`./install.sh`) - a lightweight generic CLI dev base, nothing HyperI-specific: - Docker (Engine on Linux, CLI-only via Homebrew on macOS, no Docker Desktop, bring your own daemon) @@ -183,6 +188,7 @@ under `become` -- so treat it exactly as you would the playbook. - `infrastructure`: OpenTofu + OpenBao (the OSS forks, no HashiCorp BUSL tools), AWS CLI v2, checkov, and terraform-docs for generating IaC module reference docs (engine-agnostic -- it reads `.tf` whichever binary runs it, and OpenTofu has no native `tofu docs`). Under `k8s`: kubectl + kubectx + kubens + k9s + kind + argocd + dive + kustomize + kubeconform + kube-linter. helm is NOT in that group -- it sits in `cloud`, so plain `--tags infrastructure` gets it whether or not you select `k8s`. The `data` group: clickhouse-client, rpk, valkey-cli, vector. The `cloudflare` group: flarectl + wrangler (flarectl builds from source on both platforms -- Cloudflare ships no binary -- so Linux needs `developer-go`) - `contributor`: hyperi-ci and the tools its checks drive (semgrep, alint), gitleaks, trivy, hadolint, pip-audit, ansible-lint, pre-commit, act, and git-scrub for rewriting AI residue or a leaked secret out of git history -- gitleaks scans FULL history, so a secret removed from HEAD still fails the gate. macbash is here too: it flags the GNU-only bash constructs that break on macOS, and CONTRIBUTING.md asks for it on every shell change - `soe` / `soe-gui`: HyperI org policy: VPN clients, Claude Code, Slack, LibreOffice, RDP client, telemetry-disable, auto-updates, GNOME taskbar +- `developer-ai` (off by default): the OpenAI Codex CLI as a second opinion alongside Claude Code rather than a replacement for it, plus OpenAI's Codex plugin FOR Claude Code, so `/codex:review` and `/codex:adversarial-review` are things Claude asks Codex for. The plugin is skipped -- with a warning naming the tag that fixes it -- unless claude, codex and a new enough node are all present for that user, because it installs happily without them and then throws on every invocation. Sign-in stays the person's: `codex login --device-auth` on a box with no browser - `power-profile` (off by default, and deliberately not in `soe`): sleep, idle and lid policy, selected per machine. `always-on` (the default profile) never idle-suspends on mains power and does not sleep when the lid shuts -- for a repurposed laptop doing build work, or a desktop that has to answer ssh. `vm` never sleeps or suspends at all, for an unattended RDP guest that nobody can walk over and wake. Battery behaviour stays stock under `always-on`, because a machine that will not sleep in a bag cooks itself. Profiles are data files, so adding one is adding a file -- see [roles/power-profile/README.md](ansible/roles/power-profile/README.md) - `zram` (off by default, and deliberately not in `soe`): a small compressed-RAM swap device, sized from the host's RAM and capped at 8 GiB. It exists because cgroup `MemoryHigh` throttles by reclaim, and on a swapless host the only reclaimable memory is page cache -- so a build past its budget stalls rather than slows. Pairs with the Rust build governor, which warns at converge time when it lands on a swapless host. Never restarts a running device, so a size change waits for a reboot -- see [roles/zram_swap/README.md](ansible/roles/zram_swap/README.md) - `arcane` (off by default): [Arcane](https://getarcane.app), a web UI for the containers on the box. Enable it with `-e soe_arcane_enabled=true` and you get a daemon on `http://localhost:3552` that comes back after a reboot and keeps itself updated. Works against docker-ce on Linux and colima on macOS. Bound to loopback because it holds the Docker socket, so whatever reaches that port owns the machine. Login is whatever Arcane seeds -- `arcane` / `arcane-admin` as upstream documents it. The role sets neither, and only clears the forced first-login password prompt, which it does by re-submitting that seeded password so the credentials stay unchanged. That needs the password policy relaxed to `basic` (`soe_arcane_password_policy`), because upstream's default `strong` policy rejects its own seeded password. There is still a login -- auto-login sits behind a `buildables` Go build tag that no published image is compiled with, so zero-auth is not available without building your own image. The login lasts about a day by default, which is a prompt every morning on a dev box -- `-e soe_arcane_long_session=true` stretches it to a year from each login. It is off by default because Arcane holds the Docker socket, and it takes one log-out and log-in to take effect, since the session expiry is stamped at login diff --git a/ansible/playbooks/main.yml b/ansible/playbooks/main.yml index 52684f5..ad365c7 100644 --- a/ansible/playbooks/main.yml +++ b/ansible/playbooks/main.yml @@ -302,6 +302,18 @@ become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" tags: ['soe-gui', 'never'] + # AI coding-agent tooling (opt-in): the Codex CLI as a second opinion + # alongside Claude Code, plus OpenAI's Codex plugin FOR Claude Code. + # + # AFTER soe ON PURPOSE. Claude Code is a hard dependency of the Codex + # plugin and is installed by soe/tasks/claude.yml, so a single + # `--tags claude,codex-plugin` run has to reach that task first. Listed + # earlier, the plugin's dependency probe would run before Claude Code + # existed and skip the plugin the same run that installed its dependency. + - role: developer-ai + become: "{{ ansible_facts['distribution'] != 'MacOSX' }}" + tags: ['developer-ai', 'never'] + # macOS modern bash (opt-in, macOS-only). - role: bash-modern tags: ['bash-modern', 'never'] diff --git a/ansible/roles/developer-ai/defaults/main.yml b/ansible/roles/developer-ai/defaults/main.yml new file mode 100644 index 0000000..e6c1d52 --- /dev/null +++ b/ansible/roles/developer-ai/defaults/main.yml @@ -0,0 +1,71 @@ +--- +# ============================================================================ +# SHARED ENVIRONMENT +# ============================================================================ +# One PATH for every probe and every install in this role, covering both +# platforms so the tasks do not each carry their own copy. +# +# `{{ user_home }}/.local/bin` is the load-bearing entry and the reason this is +# not just the inherited PATH: claude, codex and uv-installed tools all land +# there, PER USER. `ansible_facts['env'].PATH` is the CONNECTING user's, which +# on a fleet machine is a service account that has none of them -- it is +# appended for the system binaries, never relied on for the agent CLIs. +developer_ai_env: + PATH: >- + /opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:{{ + user_home }}/.local/bin:{{ ansible_facts['env'].PATH }} + +# ============================================================================ +# CODEX CLI +# ============================================================================ +# The official installer, which is the same route `learn.chatgpt.com/docs/codex/cli` +# documents. chatgpt.com/codex/install.sh is a 302 to releases.openai.com; the +# redirect is followed rather than hardcoded so a move upstream does not need a +# change here. +# +# Why the script rather than a get_url of the release asset: the asset is not a +# bare binary. `codex-package-.tar.gz` is a ~120 MB tree that has to be +# staged into $CODEX_HOME/packages/standalone/-/ and symlinked +# twice, and the script verifies its SHA256 against the published +# codex-package_SHA256SUMS before it does so. Reimplementing that here would buy +# nothing on integrity and would leave us owning a layout upstream versions. +# +# The thin `codex-.tar.gz` assets are NOT in codex-package_SHA256SUMS -- +# they have no published digest at all. That is the trap to avoid: they are the +# obvious thing to reach for and the only unverifiable option. +developer_ai_codex_installer_url: https://chatgpt.com/codex/install.sh + +# No version pin, per the repo's one install mode: every tool resolves its +# version when the installer runs (docs/install-matrix.md). Codex ships several +# stable releases a WEEK, so this moves faster than most -- `CODEX_RELEASE` is +# the upstream knob if a host ever needs to be held back, and it belongs in +# local-config/vars.yml on that host rather than here. +developer_ai_codex_release: "" + +# ============================================================================ +# CODEX PLUGIN FOR CLAUDE CODE +# ============================================================================ +# Names come from the repo's .claude-plugin/marketplace.json: the marketplace is +# `openai-codex` and the plugin inside it is `codex`. Both are needed -- the +# install id is plugin@marketplace. +developer_ai_codex_plugin_marketplace: openai-codex +developer_ai_codex_plugin_repo: openai/codex-plugin-cc +developer_ai_codex_plugin_name: codex + +# User scope, so the plugin follows the developer rather than a checkout. This +# is also `claude plugin install`'s own default; stated explicitly because the +# whole point of this role is that user scope means the TARGET user, not +# whoever Ansible connected as. +developer_ai_codex_plugin_scope: user + +# The plugin's hooks are `node