ansible: codify workstation SSH keys after a fleet-wide lockout

BACKLOG.md

@@ -156,10 +156,34 @@ physical console, and the OPi Zero 2Ws and RPi 3B do not have one — item 8 is
 route back in. `ansible/playbooks/break-glass-key.yml` installs the public half on every
 Linux host; the private half is generated offline and lives on paper only.
 
-- [ ] **Generate the pair offline** on a machine outside the lab, commit only
-      `ansible/files/break-glass.pub`, run the playbook, then **test it** —
-      `ssh -i ./break-glass -o IdentitiesOnly=yes <user>@192.168.1.160 'id'` — before
-      shredding the file copy. An untested break-glass key is worse than none.
+- [x] **TESTED 2026-09-23, under a real lockout, and it worked.** This item has said
+      *"an untested break-glass key is worse than none"* since August. It stopped being
+      untested the day it was the only way in.
+
+      §2.15 disabled `PasswordAuthentication` fleet-wide on 09-20. On 09-23 the Windows
+      laptop could not log in anywhere:
+
+          no supported authentication method. server sent: public key
+
+      The laptop had never had a key on those hosts — password auth had been carrying it.
+      No shell was open on the H4, so there was no second route. The break-glass key was
+      the only one left, and it answered on all three hosts tried:
+
+          ssh -i ./break-glass -o IdentitiesOnly=yes swares@192.168.1.160 id
+          ssh -i ./break-glass -o IdentitiesOnly=yes swares@192.168.1.42  id
+          ssh -i ./break-glass -o IdentitiesOnly=yes swares@192.168.1.21  id
+          -> uid=1000(swares) … 27(sudo) …   on each
+
+      Three hosts including two control-plane nodes, so this is not one lucky machine.
+      **Key auth is unaffected by `PasswordAuthentication no`**, which is why the
+      break-glass path survived the change that caused the lockout.
+
+      The file copy was shredded afterwards, which is the rest of this item. Worth
+      recording that it existed at all: it was sitting in the Windows home directory as
+      `break-glass.txt` — Notepad's extension — and that file is `swares` plus
+      `NOPASSWD: ALL` (`bootstrap.yml:41`) on every Linux host. Effectively root on the
+      fleet, at rest, on a portable machine. It is also the only reason today ended well,
+      which is the tension this entry lives inside.
 - [ ] Verify the paper transcription by typing it back **from the paper**. Diffing against
       the file proves the file is right, not the paper, and the paper is what you will be
       holding.
@@ -2236,6 +2260,25 @@ needs normalising and its synonyms accepting, and the dry run is what surfaced t
       `n150-1` and `n150-2` were `passwordauthentication yes` for as long as anyone can
       establish, while the task claiming to disable it reported `ok`.
 
+      **IT LOCKED THE WINDOWS LAPTOP OUT OF THE ENTIRE FLEET — 2026-09-23.**
+      `PasswordAuthentication no` removed the only authentication method that laptop had.
+      It had never been given an SSH key; password auth had been carrying it silently, and
+      `ansible/files/` held exactly one public key — the break-glass one — with no
+      mechanism for a workstation key at all.
+
+      This entry checks what the change does to `sshd` in considerable detail and never
+      asks **who was authenticating with a password**. That question is the one that
+      mattered, it costs one command (`grep -c 'sshd.*Accepted password' /var/log/auth.log`
+      or `lastlog`), and nobody asked it — including in review.
+
+      Recovered with the break-glass key, which is now tested for the first time since
+      August (§1.2). Fixed properly by `ansible/playbooks/admin-keys.yml` and
+      `ansible/files/admin-keys/`, so workstation keys are in git and land on any host that
+      is rebuilt or returns from the dead.
+
+      **Before hardening an authentication path, enumerate who is using the method you are
+      about to remove.** Not what the config says — who is actually authenticating.
+
       **One host silently missed the rollout, and that is the part to remember.**
       `opi-zero2w-2` went UNREACHABLE mid-run — `Connection timed out during banner
       exchange` — and the play carried on through the remaining three hosts and
@@ -2482,6 +2525,17 @@ Against: 2 GB RAM, a 2014 SoC, eMMC of unknown wear, and `LAB-DESIGN.md` calls t
 flaky. The Zero 2W spares are 4 GB, 64-bit and on current Debian — but they are WiFi,
 which is the whole problem.
 
+- [ ] **Remove the workstation key that today's recovery pushed to it.** On 2026-09-23 the
+      ad-hoc admin-key push targeted `all:!x86_nodes:!embedded`, which includes `xu3-1`, so
+      the Windows laptop's key is now in `~swares/.ssh/authorized_keys` there.
+      `admin-keys.yml` excludes the host going forward, but exclusion does not remove what
+      is already present:
+
+          ansible xu3-1 -i inventory/hosts.yml -m ansible.posix.authorized_key \
+            -a "user=swares state=absent key='$(cat files/admin-keys/wares-windows.pub)'"
+
+      A key granting `swares` — and therefore `NOPASSWD: ALL` — on a host running OpenSSL
+      1.0.2g is the one place in the lab not to put one.
 - [ ] **Decide: retire, or rebuild onto current Armbian with a real job.** Deferred
       deliberately 2026-09-20 — worth considering a use before powering it off, and the
       DNS-host question above is the one that might justify it.
@@ -3931,6 +3985,19 @@ Grep patterns run under `sudo` land in the journal they are searching.
       2.4 GHz actually helps on that board before putting it in Ansible, or the repo
       acquires a fix nobody verified.
 
+      **SECOND OCCURRENCE, DIFFERENT BOARD — 2026-09-23.** `opi-zero2w-1` was
+      `UNREACHABLE` with **`No route to host`** on `192.168.1.184` during the admin-key
+      push, so it is the only host in the fleet without the new workstation key. Note the
+      error differs from `-2`'s: a banner-exchange timeout is a slow link, *no route* is a
+      box that is off, hung, or off the network entirely.
+
+      That is **two of the four Zero 2W boards dropping out of fleet-wide changes in four
+      days**, and `opi-zero2w-1` is the DNS **tertiary** resolver. The `dns-*` blackbox
+      probes from §3.3 should have caught it — checking whether they did is the free test.
+      `ansible/playbooks/admin-keys.yml` exists partly because of this: a key pushed by an
+      ad-hoc command leaves a downed host permanently behind, where a committed playbook
+      catches it up whenever it returns.
+
       **FIRST OPERATIONAL COST — 2026-09-20.** Until now this entry rested on signal
       measurements: `-63 dBm`, `87.8 Mbit/s (MCS 2)`, `38/168 ms`. Today the link did
       something.

CLAUDE.md

@@ -100,6 +100,20 @@ Read this before acting. Full context is in `docs/` (start with `ARCHITECTURE.md
 - Never read aloud, echo, or commit: `/etc/restic/password`, `ansible/.vault_pass`,
   `ansible/files/pull-secret.json`, any kubeconfig or k3s token. These are
   `.gitignore`d — keep it that way.
+- **SSH is key-only now, fleet-wide, and that has bitten once.** §2.15 set
+  `PasswordAuthentication no` everywhere on 2026-09-20; three days later it locked the
+  Windows laptop out of all fourteen hosts, because that laptop had never been given a key
+  and password auth had been carrying it silently. Recovery was the break-glass key, which
+  worked and is finally tested (§1.2). **Workstation keys live in
+  `ansible/files/admin-keys/*.pub`, installed by `ansible/playbooks/admin-keys.yml`** — add
+  them there, never by hand, or a rebuilt host or one returning from downtime will be
+  missing them. `xu3-1` is excluded deliberately (§2.16). Before removing any
+  authentication method, **enumerate who is actually using it**, not what the config says.
+- The break-glass private key belongs offline, in the envelope — not on a workstation. On
+  2026-09-23 it was found in a Windows home directory as `break-glass.txt`, which is
+  `swares` plus `NOPASSWD: ALL` (`bootstrap.yml:41`) on every Linux host. It was also the
+  only way back in that day, so the answer is a tested key plus a working workstation key,
+  not a shredded key and no route.
 
 ## Verify before asserting
 

ansible/files/admin-keys/wares-windows.pub

@@ -0,0 +1 @@
+ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILPgYHjT8hul/EcyKgn+Iz075voR2Jxzh9i8dzCQyRus wares@DESKTOP-SL9D1QM

ansible/playbooks/admin-keys.yml

@@ -0,0 +1,124 @@
+---
+# Install workstation admin SSH public keys on every managed Linux host.
+#
+# WHY THIS EXISTS — 2026-09-23
+#
+# §2.15 disabled `PasswordAuthentication` fleet-wide on 2026-09-20. Two days
+# later the Windows laptop could not log in anywhere:
+#
+#     no supported authentication method. server sent: public key
+#
+# The hardening was right. What was missing is that the laptop had never had a
+# key on those hosts — password auth had been carrying it, and nobody asked who
+# was authenticating with a password before removing the option. `ansible/files/`
+# held exactly one public key, the break-glass one, and no mechanism existed for
+# a workstation key at all.
+#
+# Recovery was the break-glass key, which worked — see BACKLOG §1.2, whose
+# "test it" checkbox had been open since August and was finally answered by a
+# real lockout. The laptop key was then pushed by an ad-hoc `ansible -m
+# authorized_key` command, which is not reproducible and left `opi-zero2w-1`
+# (down at the time) permanently one key behind. This play is that command, made
+# reproducible.
+#
+# TO ADD A WORKSTATION: drop its public key in `ansible/files/admin-keys/` as
+# `<name>.pub` and run this. Public keys are not secrets; committing them is the
+# point, so a rebuilt or returning host gets every key without anyone
+# remembering.
+#
+# Run:
+#   ansible-playbook -i inventory/hosts.yml playbooks/admin-keys.yml --check
+#   ansible-playbook -i inventory/hosts.yml playbooks/admin-keys.yml
+#
+# To remove a key, delete the file and re-run with -e admin_keys_prune=true.
+# That is opt-in on purpose: a play that silently prunes authorized_keys is a
+# lockout waiting for a typo in a filename.
+- name: Install workstation admin SSH keys
+  # xu3-1 is deliberately EXCLUDED, decided 2026-09-23. It is in the usual
+  # pattern and did receive this key during the ad-hoc recovery push, but it runs
+  # Ubuntu 16.04 with OpenSSL 1.0.2g and is pending a retire-or-rebuild decision
+  # (§2.16). A key granting `swares` — which `bootstrap.yml` gives NOPASSWD sudo
+  # — on the least maintained host in the lab is the one place not to put it.
+  # The recovery copy was removed by hand; see §2.16.
+  hosts: "{{ admin_key_hosts | default('all:!x86_nodes:!embedded:!xu3-1') }}"
+  become: true
+  gather_facts: false
+
+  vars:
+    admin_key_dir: "{{ playbook_dir }}/../files/admin-keys"
+    admin_key_user: "{{ ansible_user | default('swares') }}"
+    admin_keys_prune: false
+
+  tasks:
+    - name: Collect the public keys to install
+      ansible.builtin.find:
+        paths: "{{ admin_key_dir }}"
+        patterns: "*.pub"
+      delegate_to: localhost
+      become: false
+      run_once: true
+      check_mode: false   # read-only; must run under --check or the asserts below are vacuous
+      register: admin_key_files
+
+    # An empty key directory would otherwise produce a play that loops over
+    # nothing and reports success on every host — the exact failure this play was
+    # written after. See the next assert for the same lesson one level down.
+    - name: There must be at least one key to install
+      ansible.builtin.assert:
+        that:
+          - admin_key_files.files | length > 0
+        fail_msg: >-
+          No *.pub files in {{ admin_key_dir }}. Refusing to run: a play that
+          installs nothing and reports success is worse than one that fails.
+        quiet: true
+      run_once: true
+
+    # THE EMPTY-KEY LESSON, and why this assert exists.
+    #
+    # On 2026-09-23 the recovery push ran with an unset shell variable, so
+    # `ansible.posix.authorized_key` received `key: ""`. It did not error. It
+    # reported **SUCCESS on thirteen hosts** with `changed: false`, and anyone
+    # reading that output would conclude the key was deployed. It was not.
+    #
+    # So every key is validated as a key before anything is installed — shape
+    # checked, not merely non-empty, because `key: "hello"` would also have been
+    # accepted silently.
+    - name: Every file must contain a public key
+      ansible.builtin.assert:
+        that:
+          - lookup('file', item) is match('^(ssh-ed25519|ssh-rsa|ecdsa-sha2-[a-z0-9-]+|sk-ssh-ed25519@openssh\.com) AAAA')
+        fail_msg: >-
+          {{ item | basename }} does not look like an SSH public key. Refusing to
+          install any key: authorized_key accepts an empty or malformed value and
+          reports success, so nothing downstream would tell you.
+        quiet: true
+      loop: "{{ admin_key_files.files | map(attribute='path') | list }}"
+      loop_control:
+        label: "{{ item | basename }}"
+      run_once: true
+
+    - name: Install each admin key for {{ admin_key_user }}
+      ansible.posix.authorized_key:
+        user: "{{ admin_key_user }}"
+        key: "{{ lookup('file', item) }}"
+        state: present
+        exclusive: false
+      loop: "{{ admin_key_files.files | map(attribute='path') | list }}"
+      loop_control:
+        label: "{{ item | basename }}"
+
+    # `exclusive: false` above means removing a file does not remove the key from
+    # the hosts. Pruning is a separate, opt-in pass so that a renamed or
+    # accidentally deleted file cannot lock anyone out. It keeps the break-glass
+    # key and the Ansible control-node key, neither of which lives in this
+    # directory.
+    - name: Report that pruning was not attempted
+      ansible.builtin.debug:
+        msg: >-
+          Keys in {{ admin_key_dir }} are installed. Keys already on the hosts but
+          NOT in that directory were left alone — including the break-glass key
+          (§1.2) and the control-node key from bootstrap.yml. Re-run with
+          -e admin_keys_prune=true to remove a key you have deleted from the
+          directory, and read `authorized_keys` on one host first.
+      run_once: true
+      when: not admin_keys_prune | bool