diff --git a/BACKLOG.md b/BACKLOG.md index 098bdb2..e64e545 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1163,8 +1163,8 @@ on the hot tier (`/dev/vg_microshift/lv_nas`) — a stable LVM path, deliberatel So the numbers cannot move at assembly. The caution below still stands for anyone editing that file — writing wrong `ARRAY` lines is its own way to lose an array — - but the work was done and never ticked. Found in the 2026-09-20 orphan sweep (§7.y). - Note the two entries carry different `name=` forms, `odroid-nas.wares.com:1` versus + but the work was done and never ticked. Found by the §7.y orphan sweep. Note the + two entries carry different `name=` forms, `odroid-nas.wares.com:1` versus `odroid-nas:0`; cosmetic, since assembly matches on UUID. ### 1.5 ~~Nothing ever trims the cold-sec copy repo~~ — **FIXED, confirmed 2026-08-21** @@ -1485,9 +1485,27 @@ fresh-host seed. Do not "fix" the divergence by templating the file. touch it. Leaving it undecided is how it got here. - [ ] `secret/lab/pihole` version 1 (created 2026-07-17 by the `root` actor, before the root token was revoked) held a key named `password` — plaintext, presumably for - `tofu/dns`'s `TF_VAR_pihole_password`. It survives in kv2 history. Establish whether - anything reads it, then decide whether a plaintext Pi-hole password should still be - in a Vault that serves HTTP in the clear (§2.6). + `tofu/dns`'s `TF_VAR_pihole_password`. It survives in kv2 history. + + **"Establish whether anything reads it" — answered 2026-09-20: nothing does.** + + - Version 1 holds exactly one key: `['password']`. Read back by key NAME only, + never by value. + - The only live reader of `secret/lab/pihole` is `ansible/playbooks/dns.yml`, and + it reads **`web-pwhash`** (lines 79, 112) — a different key entirely. + - The presumed consumer, `tofu/dns`, does not read Vault at all. It takes + `TF_VAR_pihole_password` from the environment (`variables.tf:7`, + `terraform.tf:45`). + - And that module **cannot be applied**. `tofu/dns/main.tf:37` records why: the + `ryanwholey/pihole` provider is Pi-hole v5 only and the lab runs v6. The live + record is set by `ingress_vip` in Ansible instead. + + So the plaintext is read by nothing, and its only plausible consumer has been + unusable since the v6 upgrade. **What remains is a decision, not an investigation:** + `vault kv destroy -versions=1 secret/lab/pihole` removes it from kv2 history and is + irreversible, which is why it is not done here. The §2.6 question — whether a + plaintext password belongs in a Vault that serves HTTP in the clear — is the same + decision viewed from the other end. - [x] **Done — merged as #485**, `fix(dns): match pihole config modes to the host; git was looser than reality`. The `--check` caught the playbook wanting to *widen* two permissions the hosts had tighter (`/etc/pihole` 0755→0775, `lab-clients.sql` @@ -3012,16 +3030,16 @@ all**. The check had two ways to produce a zero and only one was accounted for. `k3s_server:k3s_agents`, then verifies by what is *listening* rather than by what the package manager said, and warns by name if a host process holds 9100 on a cluster node. -- [x] **Verified 2026-09-20 at the source.** The H4's exporter is producing - **45** `node_systemd_unit_state` series matching `backup-*`, read from +- [x] **Verified 2026-09-20 at the source.** The H4's exporter is producing **45** + `node_systemd_unit_state` series matching `backup-*`, read from `localhost:9100/metrics` on the box itself. Those feed `LabBackupUnitFailed` on the one host where backups run. Read at the exporter rather than in Prometheus deliberately: a Prometheus query - answers "is the target up *and* scraping *and* retaining", three questions at once, + answers "is the target up *and* scraping *and* retaining" — three questions at once, and §3.14 has already shown retention here is shorter than configured. The exporter - answers the one that was asked — are the series being produced at all — and a - non-zero count is a number only the work actually happening can produce. + answers the one that was asked, are the series produced at all, and a non-zero count + is a number only the work actually happening can produce. ### 3.6 `.github/workflows/sync-check.yml` performs no sync check `:17-30` — named `Post-merge notice`, does nothing but `echo`, because hosted runners @@ -5203,12 +5221,21 @@ items in the sweep, and none of them was work anyone could do: **Open, and not about caching:** -- [ ] **Does `immich-machine-learning:v2.7.5` have an arm64 manifest at all?** It is cached - only on h4-core (amd64) and has therefore never run on an opi5pro. Nothing constrains - it by architecture, so if it reschedules onto an arm64 node and no manifest exists it - fails regardless of what is cached. `home-assistant` and `immich-server` are proven - multi-arch by their presence on opi5pro-2. Answer it with `k3s ctr images pull` on an - arm64 node. +- [x] **Yes — answered 2026-09-20 by asking the registry, not a node.** + `ghcr.io/immich-app/immich-machine-learning:v2.7.5` publishes: + + linux/amd64 + linux/arm64 + unknown/unknown ×2 (buildx attestations — provenance and SBOM, not platforms) + + So it can schedule on an opi5pro. The architecture worry is closed; what remains is + the generic caching point this entry is about — the image is not *present* there, so + a reschedule needs a working pull. + + Answered against the registry rather than `k3s ctr images pull` on an arm64 node, as + the entry suggested, because a pull would also have **cached it** — changing the + thing being measured. An anonymous ghcr token and a manifest-list `Accept` header + read the index without moving a byte of image data. - [ ] **`busybox:1.38` keeps turning up.** It is the Home Assistant init container, it is the workload exploiting §4.10's Kyverno resource-limits hole, and it is cached on one node. Three separate entries, one container. @@ -5470,13 +5497,13 @@ and a reminder that a citation landing does not make the sentence around it true sda 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 sdb 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 sdc 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 - sdd 199 UDMA_CRC_Error_Count 0x0032 … - 0 + sdd 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 The entry offered two possibilities — "either resolved and undocumented, or a lost thread." It was the first. Checked all four rather than the `sda sdb` the entry named, since the cold tier is two mirrors, not one. - The cost of leaving this open was not the disks; it was that "unresolved H4 CRC + The cost of leaving this open was never the disks. It was that "unresolved H4 CRC fault" sat in `docs/STANDUP.md` as an open hardware concern for weeks, and one command ends it. @@ -5820,57 +5847,89 @@ rather than a count: Eight of sixteen are a single command away. They have been open for weeks because nobody had a list saying so. -**Round 2, same day — five of those eight run, four closed:** +**Round 2, same day — all eight run, seven closed:** ``` -§1.4 mdadm ARRAY UUIDs already pinned, both arrays -> done -§6.12 UDMA_CRC all four disks raw 0, 200/200 -> done -§3.15 node_exporter series 45 backup-* series at :9100/metrics -> done -§2.13 token in history grep -c = 0 in both histories -> done -§2.1 pihole app_pwhash present on BOTH resolvers -> confirmed, still open +§1.4 mdadm ARRAY UUIDs already pinned, both arrays -> done +§6.12 UDMA_CRC all four disks raw 0, 200/200 -> done +§3.15 node_exporter series 45 backup-* series at :9100/metrics -> done +§2.13 token in history grep -c = 0 in both histories -> done +§4.17 immich arm64 linux/arm64 published; worry closed -> done +§4.17 busybox:1.38 one reference, home-assistant:53 -> done +§2.1 kv v1 `password` read by NOTHING; tofu/dns can't run -> done +§2.1 pihole app_pwhash present on BOTH resolvers -> confirmed, open ``` -Four of the five were **already true and never recorded**. Nobody had run the command, +**Seven of the eight were already true and never recorded.** Nobody had run the command, so the entries aged as though the work were outstanding — `docs/STANDUP.md` carried -"unresolved H4 CRC fault" as an open hardware concern for weeks, and the disks had zero -errors the whole time. +"unresolved H4 CRC fault" as an open hardware concern for weeks while all four disks sat +at zero errors. + +Two answered better than the entry proposed. §4.17 suggested `k3s ctr images pull` on an +arm64 node — which would have **cached the image**, changing the thing being measured; the +registry index answers it without moving a byte. §2.13 said `grep -n`, which prints the +token it is looking for; `grep -c` answers the same question without displaying it. + +The eighth is the instructive one. Measuring `app_pwhash` *confirmed* the problem rather +than closing it, because the checkbox was phrased as a **finding, not an action** — no +amount of checking could ever tick it. Rewritten as the decision it needs. **A backlog +item phrased as an observation cannot be completed**, which is the §4.17 conditions +lesson in a subtler form and worth watching for in what remains. + +**This is now checked by script, not by hand: `scripts/backlog-audit.py`.** The counting +below was done with it, and the numbers in this entry are its output rather than +arithmetic. It also resolves the miscount noted above — §5 and §9 keep their items as +bullets directly under `## N.` with no `### N.M` entry, so a parser tracking the last +heading misattributes them; the script reports those as `unowned` rather than filing them +under whichever entry happened to precede. + +``` +orphans 11 open boxes inside entries titled as complete +unowned 3 checkboxes with no owning entry (the §5/§9 shape) +unboxed 17 section-level prose bullets that can never be ticked +``` -The fifth is the instructive one. Measuring `app_pwhash` confirmed the problem rather -than closing it: the checkbox was phrased as a *finding*, not an action, so no amount of -checking could tick it. Rewritten as the decision it actually needs. **A backlog item -phrased as an observation cannot be completed** — that is the §4.17 lesson again, in a -subtler form, and worth watching for in the remaining twelve. +**That 11 was 15 until the script was corrected, and the correction is worth recording.** +`SWEPT` was in the tool's closed-marker list, so this entry — titled SWEPT because a +sweep happened — counted as complete, and its four legitimate follow-ups were reported +as orphans. The prose here said 13 while the script said 15, which is how it was caught: +**an entry that disagrees with its own auditing tool is the defect this entry is about, +committed inside it.** -Orphans: 16 → **12**. Of those, three still need one command (§3.9 chart audit, -§4.17 immich arm64 manifest, §2.1 Vault kv v1), and `busybox:1.38` resolved to a single -reference — `gitops/workloads/home-assistant/deployment.yaml:53`. +The rule the tool now carries: *every marker must describe the entry's subject, not an +activity performed on it.* A missed orphan is cheaper than a false one, because a false +one teaches people to ignore the report. -**COUNT THE FILE CAREFULLY — a naive parser reports 17, and three of those are phantom.** -Reconciling round 2's result against the file disagreed, so the difference was chased -rather than rounded: +**Kept from the hand-counted version, because it is the worked example that justifies the +script.** Before `backlog-audit.py` existed, reconciling round 2 by hand produced a +disagreement that had to be chased rather than rounded: ``` -raw parser count 17 - less §7.y's own two follow-ups (new work, not orphans) -2 - less three MISATTRIBUTED items -3 -true orphans remaining 12 +raw hand count 17 + less §7.y's own follow-ups (new work, not orphans) -2 + less three MISATTRIBUTED items -3 +true orphans 12 ``` -The misattribution is a real defect in how this file is structured, not in the counting. -**§5 and §9 hold their items as bullets directly under `## N.` with no `### N.M` entry.** -Any parser tracking "the last `###` seen" therefore assigns them to whichever entry -happened to come before — here §5's two landed under §4.17, and §9's one under §3.12. -They are not orphans and never were. - -Two consequences. Anyone auditing this file by script will chase three items that do not -exist, in entries that do not contain them. And §5 and §9's contents are invisible to any -per-entry view — which is the same problem as the 17 un-checkboxed prose bullets already -noted below, from the other direction: the items exist but have nowhere to belong. - -- [ ] **Give §5 and §9 proper `### N.M` entries**, so every checkbox has an owning entry. - Cheaper than it sounds and it makes the file scriptable, which — on the evidence of - this sweep, where a five-line script found 25 stale items no one had noticed — is - worth more than the tidiness. +Those three were not orphans and never were. **§5 and §9 hold their items as bullets +directly under `## N.` with no `### N.M` entry**, so any parser tracking "the last `###` +seen" assigns them to whichever entry came before — §5's two landed under §4.17, §9's one +under §3.12. Anyone auditing by eye would have gone looking for items in entries that do +not contain them. + +The script reports that class as `unowned` instead of misfiling it, which is why its +count and this entry's now agree. The underlying structural problem is unchanged and is +the first follow-up below: §5 and §9's contents are invisible to any per-entry view, +which is the same defect as the 17 un-checkboxed bullets seen from the other side — those +items have a box and no owner, these have an owner and no box. + +- [ ] **Give §5 and §9 proper `### N.M` entries**, so every checkbox has an owning entry + and the `unowned` count goes to zero. Cheap, and it makes this file auditable — + which, on the evidence that a five-line script found 25 stale items nobody had + noticed, is worth more than the tidiness. +- [ ] **Wire `backlog-audit.py --strict` into CI** once `unowned` is zero. It already + exits 1 on orphans or unowned items; the only reason not to gate on it today is + that the file would fail. - [ ] **Give §7 and §9's prose bullets checkboxes** — 17 items with no box at all, invisible to any count and impossible to mark done. §9 is titled "Small, live, diff --git a/scripts/backlog-audit.py b/scripts/backlog-audit.py new file mode 100644 index 0000000..92c23ae --- /dev/null +++ b/scripts/backlog-audit.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python3 +"""Audit BACKLOG.md's own bookkeeping. + +WHY THIS EXISTS +--------------- +On 2026-09-20 a five-line throwaway script counted the checkboxes in BACKLOG.md +and found 49 open / 69 done — of which **25 open items sat inside entries whose +heading announced completion** (RESOLVED, FIXED, ACCEPTED, or struck through). +Reading the headings said the work was finished; the boxes said otherwise. + +Sweeping those 25 closed nine items. Seven of them needed no work at all: they +were already true and nobody had run the command. `docs/STANDUP.md` carried +"unresolved H4 CRC fault" as an open hardware concern for weeks while all four +disks reported zero CRC errors. + +That is the same failure this file spends five thousand lines documenting in the +infrastructure — a green summary over unfinished work — occurring in the document +that records it. So the check is permanent now rather than a thing someone +remembers to do. See BACKLOG §7.y. + +WHAT IT REPORTS +--------------- +orphan An open `- [ ]` inside an entry whose heading claims completion. + The high-value category: these are usually done-and-unticked, or real + work hiding behind a reassuring title. + +unowned A checkbox with no `### N.M` entry above it inside its section. §5 and + §9 keep their items as bullets directly under `## N.`, so a parser that + tracks "the last ### seen" silently attributes them to whichever entry + came before — on 2026-09-20 that inflated the orphan count from 12 to + 17 and sent the reader looking for items in entries that never held + them. Reported as its own category rather than misfiled. + +unboxed A prose bullet in a section that otherwise uses checkboxes. It can + never be marked done, so it is invisible to any progress count. §7 and + §9 hold 17 of these, and §9 is titled "Small, live, cheap" — precisely + the work that should close in minutes. + +EXIT CODES +---------- +0 clean (or no --strict) +1 --strict and at least one orphan or unowned item + +Usage: + scripts/backlog-audit.py # summary + scripts/backlog-audit.py --list # every finding, with line numbers + scripts/backlog-audit.py --strict # non-zero exit if orphans exist + scripts/backlog-audit.py --file X.md +""" + +from __future__ import annotations + +import argparse +import re +import sys +from pathlib import Path + +# A heading claims completion if it is struck through or carries one of these. +# Kept explicit rather than clever: a regex that tried to infer "doneness" from +# prose would be the kind of check that cannot say why it decided something. +# +# EVERY MARKER MUST DESCRIBE THE ENTRY'S SUBJECT, NOT AN ACTIVITY PERFORMED ON +# IT. "SWEPT" was in this list for one revision and was wrong: §7.y is titled +# SWEPT because a sweep happened, not because its subject is finished, so its +# legitimate follow-ups were reported as orphans. Two of them, which is how the +# defect was noticed — the entry's own text said 13 and its own script said 15. +# If a future marker is ambiguous in that way, leave it out; a missed orphan is +# cheaper than a false one, because a false one teaches people to ignore the +# report. +CLOSED_MARKERS = ( + "RESOLVED", "FIXED", "APPLIED", "ANSWERED", + "ACCEPTED", "CLOSED", "DONE", +) + +RE_SECTION = re.compile(r"^## (?!#)(.+)$") +RE_ENTRY = re.compile(r"^### (.+)$") +RE_BOX = re.compile(r"^\s*- \[([ xX])\]") +RE_BULLET = re.compile(r"^- (?!\[)") + + +def heading_claims_done(text: str) -> bool: + return "~~" in text or any(m in text for m in CLOSED_MARKERS) + + +def short(text: str, n: int = 58) -> str: + text = re.sub(r"\s+", " ", text).strip() + return text if len(text) <= n else text[: n - 1] + "…" + + +def audit(path: Path) -> dict: + section = entry = None + entry_done = False + sections: dict[str, dict] = {} + orphans: list[tuple[int, str, str]] = [] + unowned: list[tuple[int, str, str]] = [] + unboxed: list[tuple[int, str, str]] = [] + + for n, line in enumerate(path.read_text(encoding="utf-8").split("\n"), 1): + m = RE_SECTION.match(line) + if m: + section = m.group(1).strip() + sections.setdefault(section, {"open": 0, "done": 0, "boxes": 0}) + entry, entry_done = None, False + continue + + m = RE_ENTRY.match(line) + if m: + entry = m.group(1).strip() + entry_done = heading_claims_done(entry) + continue + + if section is None: + continue + + m = RE_BOX.match(line) + if m: + s = sections[section] + s["boxes"] += 1 + if m.group(1) == " ": + s["open"] += 1 + if entry is None: + unowned.append((n, section, line)) + elif entry_done: + orphans.append((n, entry, line)) + else: + s["done"] += 1 + continue + + # A prose bullet with NO owning entry, in a section that uses checkboxes + # elsewhere — i.e. a section-level item that looks like work but cannot + # be ticked. The `entry is None` test is load-bearing: without it this + # matched every narrative bullet inside every entry and reported 95, + # which is noise, not a finding. §7 and §9 are the real cases. + if entry is None and RE_BULLET.match(line) and "~~" not in line: + unboxed.append((n, section, line)) + + boxed_sections = {s for s, v in sections.items() if v["boxes"] > 0} + unboxed = [u for u in unboxed if u[1] in boxed_sections] + + return { + "sections": sections, + "orphans": orphans, + "unowned": unowned, + "unboxed": unboxed, + } + + +def main() -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n")[0]) + ap.add_argument("--file", default="BACKLOG.md", type=Path) + ap.add_argument("--list", action="store_true", help="show every finding") + ap.add_argument("--strict", action="store_true", + help="exit 1 if any orphan or unowned item exists") + args = ap.parse_args() + + if not args.file.exists(): + print(f"no such file: {args.file}", file=sys.stderr) + return 2 + + r = audit(args.file) + secs = r["sections"] + tot_o = sum(v["open"] for v in secs.values()) + tot_d = sum(v["done"] for v in secs.values()) + + print(f"{args.file}\n") + print(f" {'section':<48} {'open':>5} {'done':>5}") + print(f" {'-' * 48} {'-' * 5} {'-' * 5}") + for s, v in secs.items(): + if v["boxes"]: + print(f" {short(s, 48):<48} {v['open']:>5} {v['done']:>5}") + print(f" {'-' * 48} {'-' * 5} {'-' * 5}") + print(f" {'TOTAL':<48} {tot_o:>5} {tot_d:>5}\n") + + print(f" orphans {len(r['orphans']):>3} open items inside entries titled as complete") + print(f" unowned {len(r['unowned']):>3} checkboxes with no ### entry in their section") + print(f" unboxed {len(r['unboxed']):>3} prose bullets that can never be marked done") + + if args.list: + for name, rows in (("ORPHANS", r["orphans"]), + ("UNOWNED", r["unowned"]), + ("UNBOXED", r["unboxed"])): + if not rows: + continue + print(f"\n{name}") + for n, owner, line in rows: + print(f" L{n:<6} {short(owner, 40):<40} {short(line, 62)}") + elif r["orphans"] or r["unowned"]: + print("\n (--list to see them)") + + if args.strict and (r["orphans"] or r["unowned"]): + print("\nFAIL: --strict and the file has orphaned or unowned items.") + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main())