From 2b013e3bbc2a0caf8181afadd7ffe36a6ab3f3ec Mon Sep 17 00:00:00 2001 From: swares Date: Wed, 23 Sep 2026 12:41:07 -0600 Subject: [PATCH] vault: fix the snapshot token, which was built to die monthly Fifteen days with no Vault snapshot - last one Sep 8, failing nightly with 403 invalid token. Three stacked defects, all in the six-line one-time recipe at the top of backup-vault.yml. 1. No -orphan, so the token was a child of token-admin - which docs/OPS.md and 1.11 say is deliberately allowed to lapse. Vault revokes children when the parent expires. token-admin is minted at 720h; one issued around 08-09 expires 09-08, and the last good snapshot is 09-08. 2. -period=87600h was never honoured. Vault caps period at the effective max_ttl of 768h and warns at creation. A comment promising ten years described a 32-day token. 3. Nothing renewed it. A periodic token lives one period at a time, so vault token renew -self now runs before every snapshot and fails the unit if it cannot. The monitoring was not the failure: LabBackupUnitFailed fired nightly and ntfy delivered it, to a phone that had been stolen. One subscriber, so losing the device lost the channel. Records 2.13's second occurrence - a live token pasted while diagnosing this, because the command I suggested printed it. The fault is in suggesting a command that can print a secret, not in the pasting; the header recipe now uses -field with a redirect. Also fixes a second marker bug in backlog-audit.py: markers match as substrings, so NOT YET APPLIED contains APPLIED and the tool read 1.14's heading as a completion claim. Strips 'not yet ' before matching; verified against seven headings. --- BACKLOG.md | 126 +++++++++++++++++++++++++++++ ansible/playbooks/backup-vault.yml | 57 ++++++++++++- scripts/backlog-audit.py | 15 ++++ 3 files changed, 197 insertions(+), 1 deletion(-) diff --git a/BACKLOG.md b/BACKLOG.md index d07efca..10ed1f7 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -1396,6 +1396,97 @@ the primary from the secondary. Pin it. --- +### 1.14 Vault had no snapshot for fifteen days — **CAUSE FOUND 2026-09-23; FIX WRITTEN, NOT YET APPLIED** +`ansible/playbooks/backup-vault.yml:4-7` (the header), and the script it deploys + +``` +/mnt/cold-8t/vault-snapshots/ newest = vault-snap-20260908-023115.snap (Sep 8) +backup-vault.service failed, every night since +``` + +Fifteen days with no snapshot of the store holding the restic repository passwords, the +Cloudflare R2 credentials, the lldap bind password, and the fleet login password. Vault's +raft data lived only on `rpi5`'s SD card for that window. + +``` +vault-snapshot.sh: Error taking the snapshot: Error making API request. +URL: GET http://127.0.0.1:8200/v1/sys/storage/raft/snapshot +Code: 403 · permission denied · invalid token +``` + +**Three stacked defects, each sufficient alone**, all of them in the six-line "one-time, +manual" recipe at the top of the playbook: + +**1. The token had no `-orphan`, so it was a child of `token-admin`.** Vault revokes a +token's children when the parent expires. `docs/OPS.md` and §1.11 record that +`token-admin` is *deliberately allowed to lapse* — "nothing automated uses it, so an +expired one is expected, not an incident." Something automated did use it: this. The +backup was built to die on a ~30-day cycle, and the arithmetic fits — `token-admin` is +minted at 720h, one issued around 08-09 expires 09-08, and the last good snapshot is +**09-08**. + +**2. `-period=87600h` was never honoured.** Vault caps period at the effective `max_ttl` +and says so at creation: + +``` +period of "87600h" exceeded the effective max_ttl of "768h"; period value is capped +``` + +A comment promising ten years described a 32-day token. Nobody re-read the creation +output; there was no reason to, because it said `Success!` underneath. + +**3. Nothing renewed it.** A periodic token lives one period at a time and must be +renewed inside it. `vault token renew -self` now runs before every snapshot, so the +period rolls forward daily — which is what "periodic" is for. + +**The monitoring was not the failure. Delivery was.** `LabBackupUnitFailed` fired +correctly, nightly, from the first failure. ntfy delivered it, as it had before — to a +phone that had been stolen. There is exactly one subscriber, so losing the device lost +the channel, and a correct alert went nowhere for fifteen days. + +- [x] `vault token renew -self` added to the snapshot script, failing the unit if it + cannot renew. +- [x] Header recipe corrected: `-orphan`, `-period=768h`, and + `-field=token | sudo tee … >/dev/null` so the value is never displayed. +- [ ] **Mint the replacement token and confirm a snapshot lands.** Not done here because + it needs an admin token: + + # on rpi5 + vault token create -orphan -display-name=vault-backup -policy=vault-snapshot \ + -period=768h -field=token | sudo tee /etc/vault.d/backup-token >/dev/null + sudo chmod 600 /etc/vault.d/backup-token + sudo systemctl start backup-vault.service + # then, on the H4 — the only evidence that counts: + ls -lt /mnt/cold-8t/vault-snapshots/ | head -3 + +- [ ] **Audit every other unattended Vault token for `orphan=False`.** One command, and + it prints accessors rather than tokens: + + for a in $(vault list -format=json auth/token/accessors \ + | python3 -c 'import json,sys;print(" ".join(json.load(sys.stdin)))'); do + vault token lookup -accessor -format=json "$a" | python3 -c ' + import json,sys; d=json.load(sys.stdin)["data"] + print(d.get("display_name"), "orphan=", d.get("orphan"), d.get("policies"))' + done + + On 2026-09-23 this showed only two tokens — `token-vault-backup` (orphan=False) and + `token-token-admin` (orphan=True) — so the blast radius was one job. Any future + unattended token with `orphan=False` has the same timer running. + +- [ ] **Give alert delivery a second endpoint.** A laptop browser subscriber or a desktop + ntfy topic costs nothing and would have surfaced this on 09-09. Redundancy belongs + on the thing that tells you, not only on the thing being watched — the same argument + the lab already accepts for four DNS resolvers. + +- [ ] **`rpi5`'s journal retains about four days.** `SystemMaxUse=100M` with 99.5M in use, + so `journalctl --since -90d` showed a first failure of **Sep 19** when the snapshot + evidence says **Sep 9**. Ten days of history had already rotated away. Same shape as + §3.14 — retention well inside what the configuration implies — and it cost a wrong + answer about when this started. Either raise the cap on the host that runs Vault, or + stop trusting its journal for anything older than a week. + +--- + ## 2. Security ### 2.1 ~~Pi-hole admin UI deploys unauthenticated~~ — **RESOLVED 2026-08-28; the primary really was, the secondary never was** @@ -2055,6 +2146,29 @@ it under debugging pressure are different things — the durable fix is not typi command lines at all, which is why `--token-file` was reached for in the first place. That it is unsupported on the restore path (§ Drill 2c) is an upstream gap worth knowing. +**SECOND OCCURRENCE — 2026-09-23, and the process note above predicted it exactly.** While +diagnosing §1.14, a live Vault token with the `vault-snapshot` policy was pasted into a +transcript, because the suggested command was `vault token create …` with no `-field=token` +and no redirection — so Vault printed the token, as it is documented to. That policy reads +`sys/storage/raft/snapshot`, i.e. a dump of the entire store. + +``` +vault token revoke -accessor col4IdpIEg232TgP2tRU61aI +Success! Revoked token (if it existed) +``` + +Revoked within minutes, and a replacement minted with +`-field=token | sudo tee … >/dev/null`. + +**Note where the fault was: in the command that was suggested, not in the pasting.** "Do +not paste secrets" asks a person to notice a secret in output they were told to produce. +The durable version is that any command handed over during diagnosis must not be capable +of printing one — `-field` plus a redirect, `grep -c` instead of `grep -n`, an accessor +instead of a token. Both occurrences of this entry happened while gathering evidence under +pressure, which is the condition in which nobody re-reads output before pasting it. +`backup-vault.yml`'s header now carries the safe recipe; the one it replaced printed the +token on purpose. + ### 2.12 Vault is never restarted, so it silently runs an old binary — **mechanism open; the seven-week instance was resolved 08-16** **Heading corrected 2026-08-21** — it described an instance that the entry's own first sentence says is fixed. The mechanism is what remains, and both defects are intact: @@ -5979,6 +6093,18 @@ The rule the tool now carries: *every marker must describe the entry's subject, activity performed on it.* A missed orphan is cheaper than a false one, because a false one teaches people to ignore the report. +**Second marker bug, 2026-09-23, same lesson from the other direction.** Markers are +matched as substrings, so **"NOT YET APPLIED" contains "APPLIED"** — §1.14's heading says +its fix is *not* applied and the tool read that as a completion claim, turning four honest +open items into four phantom orphans. §2.14 has carried *"FIXED …, not yet applied"* for +weeks and escaped only because it is struck through as well. + +Fixed by stripping `not yet ` before matching, which is deliberately dumber than +parsing negation: that is the phrase this repo actually writes. Verified against seven +headings including both real cases. **Both marker bugs were found the same way** — the +prose and the script disagreed, and the difference was chased instead of rounded. That is +now twice that the tool's value came from contradicting the person using it. + **Kept from the hand-counted version, because it is the worked example that justifies the script.** Before `backlog-audit.py` existed, reconciling round 2 by hand produced a disagreement that had to be chased rather than rounded: diff --git a/ansible/playbooks/backup-vault.yml b/ansible/playbooks/backup-vault.yml index 4f87bcd..15b1b5e 100644 --- a/ansible/playbooks/backup-vault.yml +++ b/ansible/playbooks/backup-vault.yml @@ -3,8 +3,40 @@ # # Prerequisites (one-time, manual on rpi5): # vault policy write vault-snapshot (path sys/storage/raft/snapshot, read) +# vault token create -orphan -display-name=vault-backup -policy=vault-snapshot \ +# -period=768h -field=token | sudo tee /etc/vault.d/backup-token >/dev/null +# sudo chmod 600 /etc/vault.d/backup-token +# +# THE PREVIOUS VERSION OF THOSE THREE LINES CAUSED A FIFTEEN-DAY BACKUP OUTAGE. +# It read: +# # vault token create -display-name=vault-backup -period=87600h -policy=vault-snapshot -# echo "" | sudo tee /etc/vault.d/backup-token && sudo chmod 600 /etc/vault.d/backup-token +# echo "" | sudo tee /etc/vault.d/backup-token && sudo chmod 600 … +# +# Three defects, each sufficient on its own. See BACKLOG §1.14. +# +# 1. NO `-orphan`. The token was a CHILD of whatever token created it — an admin +# token. Vault revokes children when the parent expires or is revoked, and +# `docs/OPS.md` plus §1.11 record that `token-admin` is *deliberately* allowed +# to lapse. So this backup was built to die on the admin token's ~30-day +# cycle. Last good snapshot 2026-09-08; token-admin's 720h TTL from ~08-09 +# expired the same day. +# +# 2. `-period=87600h` WAS NEVER HONOURED. Vault caps period at the effective +# max_ttl — 768h here — and warns on creation: +# period of "87600h" exceeded the effective max_ttl of "768h"; period +# value is capped accordingly +# A comment promising ten years described a 32-day token. Ask for 768h so the +# number in the repo is the number Vault uses. +# +# 3. NOTHING RENEWED IT. A periodic token lives one period at a time. The +# snapshot script now runs `vault token renew -self` before each snapshot; +# see the comment there. +# +# `-field=token | sudo tee … >/dev/null` is also deliberate: the earlier recipe +# printed the token to the terminal, and on 2026-09-23 a live one was pasted into +# a transcript while diagnosing exactly this failure. That is §2.13's second +# occurrence. Never print a token you are about to write to a file. # - name: Configure Vault raft snapshot backups hosts: vault @@ -54,6 +86,29 @@ set -euo pipefail export VAULT_ADDR="{{ vault_addr }}" export VAULT_TOKEN="$(cat {{ vault_backup_token_file }})" + + # RENEW BEFORE USE — added 2026-09-23 after fifteen days with no snapshot. + # + # This token is PERIODIC, and a periodic token does not live forever: it + # lives for one period and must be renewed inside it. Nothing renewed + # this one. The header of this playbook said `-period=87600h` (ten + # years), but Vault caps period at the effective max_ttl, which here is + # **768h**, and says so on creation: + # + # period of "87600h" exceeded the effective max_ttl of "768h"; + # period value is capped accordingly + # + # So the documented ten-year token was a 32-day token all along, and the + # job that depends on it never asked for more time. Renewing on every run + # makes the period roll forward daily, which is the whole point of a + # periodic token. + # + # NOT prefixed with `-` and not `|| true`: if the renew fails, the token + # is already dead or dying and the snapshot is the thing that matters. + # Failing here fails the unit, which trips LabBackupUnitFailed — the alert + # that did fire for fifteen days while nobody was able to read it. + vault token renew -self >/dev/null + SNAP_DIR=/tmp/vault-snapshots mkdir -p "$SNAP_DIR" SNAP="$SNAP_DIR/vault-snap-$(date +%Y%m%d-%H%M%S).snap" diff --git a/scripts/backlog-audit.py b/scripts/backlog-audit.py index 7824316..e2580de 100644 --- a/scripts/backlog-audit.py +++ b/scripts/backlog-audit.py @@ -95,7 +95,22 @@ RE_BULLET = re.compile(r"^- (?!\[)") +# Markers are matched as substrings, so a NEGATED marker would match its own +# negation: "NOT YET APPLIED" contains "APPLIED". That is not hypothetical — it +# happened on 2026-09-23, on §1.14, whose heading says the fix is *not* applied +# and which the tool therefore counted as complete, turning four honest open items +# into four phantom orphans. §2.14 has carried "FIXED …, not yet applied" for weeks +# and escaped only because that one is also struck through. +# +# Stripping the negation before matching is deliberately dumber than parsing it: +# the phrase this repo actually writes is "not yet ", so that is the phrase +# removed. A heading needing more nuance than this should be reworded, not +# accommodated. +RE_NEGATED = re.compile(r"\bnot\s+yet\s+\w+", re.IGNORECASE) + + def heading_claims_done(text: str) -> bool: + text = RE_NEGATED.sub("", text) return "~~" in text or any(m in text for m in CLOSED_MARKERS)