CI hardening v2: drastically stricter gates + massive test-suite expansion

[Xiddoc]

Summary

Priority: HIGH. The current CI (.github/workflows/ci.yml) is already a solid baseline — ruff, mypy strict + pyright, 100% line+branch coverage, interrogate, radon, vulture, pip-audit, shellcheck, hadolint, docker smoke build, docs --strict, SHA-pinned actions. This task is the next ratchet: close the remaining strictness gaps and massively expand the test suite beyond line coverage into behavior, property, mutation, and environment coverage.

Per CLAUDE.md, 100% line+branch coverage is necessary but not sufficient — feat/configurable-ports shipped a real bug at 100% coverage. The theme of this task: gates that catch bugs coverage can't.

Part 1 — Stricter CI gates

Test-runner strictness

• Warnings as errors: add filterwarnings = ["error"] to [tool.pytest.ini_options] (allowlist specific upstream deprecations individually, with a comment each). Today a DeprecationWarning from pydantic/PyYAML sails through silently.
• Order-randomized tests: add pytest-randomly so inter-test state leaks (registry/tmp-dir bleed-through between isolated_registry consumers) fail loudly. Seed is printed for reproduction.
• Per-test timeout: add pytest-timeout with a tight default (e.g. 30s) so a hung subprocess/flock test fails instead of stalling the job for 6h.
• -p no:cacheprovider in CI so pass/fail never depends on a stale .pytest_cache.

Mutation testing (the big one)

• Add mutmut (or cosmic-ray) over src/beetroot/ as a CI job. Start with a survival-rate threshold on the most logic-dense modules (ports.py, config.py, registry.py, snapshot.py), ratchet downward over time. Mutation score is the only automated gate that distinguishes "executed by a test" from "asserted on by a test". Run it on PRs against changed files only (or nightly full-run via schedule:) to keep wall-time sane.

Toolchain & supply-chain strictness

• uv lock --check job: fail when pyproject.toml and uv.lock drift (today --frozen installs would fail confusingly instead of pointing at the lockfile).
• ruff format --check in CI (formatting is currently "optional; no hard policy" — make it a hard policy so diffs stop churning).
• actionlint + zizmor jobs: lint the workflows themselves (expression typos, injection-prone ${{ }} interpolation, missing permissions:). The workflow files are load-bearing and currently unlinted.
• deptry: fail on undeclared/unused dependencies in pyproject.toml.
• codespell over src/, docs/, README.md, CHANGELOG.md.
• Packaging gate: uv build + twine check dist/* + install the built wheel into a clean venv and run beetroot --help. Catches missing templates/compose.yaml wheel-data regressions that editable installs hide (paths.bundled_compose_file() via importlib.resources is exactly the kind of thing that breaks only in the wheel).
• yamllint on src/beetroot/templates/compose.yaml, .github/workflows/*.yml, examples/.
• shfmt -d alongside shellcheck for docker/*.sh; consider raising shellcheck to -S style with targeted disables.

Environment matrix

• Run the test job on a matrix: ubuntu-latest + macos-latest (researchers run the CLI on macOS; fcntl.flock, path handling, and subprocess behavior differ) × Python 3.13 + 3.14-dev (early warning, continue-on-error: true for the dev lane).
• Upload coverage as a job artifact and add a coverage xml + diff-cover step so PR review sees exactly which changed lines each test exercises.

Part 2 — Massive test-suite expansion

The suite has ~52 files and 100% coverage; expansion targets composition and adversarial input, per the "Behavior tests, not just line coverage" rule in CLAUDE.md:

• End-to-end verb flows against a fake docker: a stub executable on PATH that records argv and replays canned compose ps JSON. Drive create → apply → up → status → down → destroy and assert on the artifacts: the rendered .env dict, the exact compose argv (-p, -f, --project-directory, --env-file), the registry JSON, the freed port index. One test per verb-chain, not per function.
• Compose template render validation: for a spread of beetroot.yaml inputs, render .env and run docker compose config (or a pure-Python ${VAR} substitution check in unit scope) asserting every ${VAR} in the template has a binding and no binding is orphaned — both directions, so render_env()/template drift fails CI (extends test_compose_template_envs.py).
• Property-based expansion (Hypothesis): grow test_property_ports.py / test_property_registry.py / test_property_render_env.py into: arbitrary interleavings of allocate/free on the port allocator (no collisions, lowest-free-reuse invariant); snapshot pack→unpack round-trip equality for arbitrary valid configs; YAML→InstanceConfig→.env→ no port self-collisions for arbitrary partial overrides (the exact fix/ports-resolver-self-collision class of bug).
• Adversarial config corpus: a tests/corpus/ of hostile beetroot.yaml files — wrong types, api_version 1/2/3/5/garbage, unicode names, path-traversal instance names, 10k-entry denylists, duplicate keys — asserting every one yields a friendly error: ... + exit 1, never a traceback (extends test_cli_error_contract.py).
• Concurrency tests: real multi-process fcntl.flock contention on the registry (N processes racing create/destroy; assert no lost updates, no duplicate port indexes) — extends test_registry_race.py from threads to processes.
• Boot-script tests: lift the docker/*.sh helpers into a harness that runs them under sh with a stubbed magisk/getprop on PATH (POSIX-sh compatible, no Docker needed). Assert: bounded-wait behavior, a failing module install doesn't abort the entrypoint under set -e (A single failing magisk --install-module aborts boot under set -e, so the container exits without launching Frida #13, magisk-config.sh waits for the Magisk daemon in an unbounded loop with no timeout #14 are exactly this class of bug — these tests would have caught both), env-var override paths (BEETROOT_*) are honored.
• Snapshot/restore round-trip suite: pack → mutate nothing → restore under a new name → assert byte-identical data/, re-rendered .env ports, registry row, and manifest schema (pydantic) — across legacy api_version inputs.
• CLI help/UX contract tests: snapshot-test --help output for every verb so flag renames (--as → --name, see snapshot restore error messages tell the user to use the deprecated --as flag instead of --name #16) can't silently leave stale strings; grep-test that error messages never reference deprecated flags.

Acceptance criteria

• All new CI jobs green on main; every third-party action SHA-pinned (existing T3 policy).
• filterwarnings = error, random order, and timeouts active with zero blanket suppressions.
• Mutation-score gate wired (changed-files on PR or nightly full) with a documented threshold + ratchet plan in CLAUDE.md.
• Test count and assertion depth meaningfully expanded per Part 2 (each checkbox shipped as its own reviewed slice — this issue can be executed as a series of small PRs).
• 100% line+branch coverage still holds; no # pragma: no cover added.
• CLAUDE.md "Development workflow"/CI section updated to describe the new gates (docs are part of every feature).

Notes

• Keep CI wall-time in check: new lint-ish jobs are cheap and parallel; mutation testing is the only expensive one — scope it to changed files on PRs and full-run on schedule:.
• Out of scope: booting real redroid in CI (tracked separately in Stealth PR7: CI fingerprint audit #6, blocked on a KVM-capable runner story).

[Xiddoc]

claimed by 1a2uf6 at 2026-06-11T05:29:22Z

---

Generated by Claude Code

[Xiddoc]

progress by 1a2uf6 at 2026-06-11T06:05:00Z — Part 1 strictness slice in active development on branch claude/1a2uf6/issue-21-ci-hardening (4 commits staged locally: pytest strictness, format gate, workflow lint jobs, packaging gate); push + PR expected shortly.

---

Generated by Claude Code

[Xiddoc]

progress by 1a2uf6 at 2026-06-11T09:21:30Z — resumed after session-limit pause; continuation agent finishing the Part 1 CI-gate slice on claude/1a2uf6/issue-21-ci-hardening (4 commits local: ruff-format pass, shfmt normalization, pytest strictness, codespell fixes; workflow jobs + packaging gate in progress). Will rebase onto current master (post #30–#33) before the PR.

---

Generated by Claude Code