From 5d8adccd6c3dda58b37f1597e8decfab9d0bc1ab Mon Sep 17 00:00:00 2001 From: swares Date: Wed, 23 Sep 2026 11:21:53 -0600 Subject: [PATCH] ansible: codify workstation SSH keys after a fleet-wide lockout 2.15 set PasswordAuthentication no everywhere on 09-20. On 09-23 the Windows laptop could not log in anywhere: it had never been given a key, and password auth had been carrying it silently. ansible/files/ held one public key - the break-glass one - and no mechanism for a workstation key existed. That entry examines what the change does to sshd in detail and never asks who was authenticating with a password. Before removing an authentication method, enumerate who is actually using it. Recovery was the break-glass key, which worked on .160, .42 and .21 - so 1.2's 'test it' checkbox, open since August, is answered by a real lockout rather than a drill. Key auth is unaffected by PasswordAuthentication no, which is why that path survived the change that caused the outage. admin-keys.yml replaces the ad-hoc recovery push. It asserts at least one key exists and that each file matches an SSH public-key shape, because the recovery command ran with an unset variable, handed authorized_key an empty string, and got SUCCESS on thirteen hosts with changed: false. Regex verified against the real keys and against empty, garbage, a bare key type, and a private key pasted by mistake. Excludes xu3-1 deliberately (2.16): a key granting NOPASSWD sudo on a host running OpenSSL 1.0.2g. Records that the recovery push already put one there and must be removed. Also records opi-zero2w-1 down with No route to host - the second Zero 2W in four days to silently miss a fleet-wide change, and the DNS tertiary. --- BACKLOG.md | 75 ++++++++++++- CLAUDE.md | 14 +++ ansible/files/admin-keys/wares-windows.pub | 1 + ansible/playbooks/admin-keys.yml | 124 +++++++++++++++++++++ 4 files changed, 210 insertions(+), 4 deletions(-) create mode 100644 ansible/files/admin-keys/wares-windows.pub create mode 100644 ansible/playbooks/admin-keys.yml diff --git a/BACKLOG.md b/BACKLOG.md index d52105a..d07efca 100644 --- a/BACKLOG.md +++ b/BACKLOG.md @@ -156,10 +156,34 @@ physical console, and the OPi Zero 2Ws and RPi 3B do not have one — item 8 is route back in. `ansible/playbooks/break-glass-key.yml` installs the public half on every Linux host; the private half is generated offline and lives on paper only. -- [ ] **Generate the pair offline** on a machine outside the lab, commit only - `ansible/files/break-glass.pub`, run the playbook, then **test it** — - `ssh -i ./break-glass -o IdentitiesOnly=yes @192.168.1.160 'id'` — before - shredding the file copy. An untested break-glass key is worse than none. +- [x] **TESTED 2026-09-23, under a real lockout, and it worked.** This item has said + *"an untested break-glass key is worse than none"* since August. It stopped being + untested the day it was the only way in. + + §2.15 disabled `PasswordAuthentication` fleet-wide on 09-20. On 09-23 the Windows + laptop could not log in anywhere: + + no supported authentication method. server sent: public key + + The laptop had never had a key on those hosts — password auth had been carrying it. + No shell was open on the H4, so there was no second route. The break-glass key was + the only one left, and it answered on all three hosts tried: + + ssh -i ./break-glass -o IdentitiesOnly=yes swares@192.168.1.160 id + ssh -i ./break-glass -o IdentitiesOnly=yes swares@192.168.1.42 id + ssh -i ./break-glass -o IdentitiesOnly=yes swares@192.168.1.21 id + -> uid=1000(swares) … 27(sudo) … on each + + Three hosts including two control-plane nodes, so this is not one lucky machine. + **Key auth is unaffected by `PasswordAuthentication no`**, which is why the + break-glass path survived the change that caused the lockout. + + The file copy was shredded afterwards, which is the rest of this item. Worth + recording that it existed at all: it was sitting in the Windows home directory as + `break-glass.txt` — Notepad's extension — and that file is `swares` plus + `NOPASSWD: ALL` (`bootstrap.yml:41`) on every Linux host. Effectively root on the + fleet, at rest, on a portable machine. It is also the only reason today ended well, + which is the tension this entry lives inside. - [ ] Verify the paper transcription by typing it back **from the paper**. Diffing against the file proves the file is right, not the paper, and the paper is what you will be holding. @@ -2236,6 +2260,25 @@ needs normalising and its synonyms accepting, and the dry run is what surfaced t `n150-1` and `n150-2` were `passwordauthentication yes` for as long as anyone can establish, while the task claiming to disable it reported `ok`. + **IT LOCKED THE WINDOWS LAPTOP OUT OF THE ENTIRE FLEET — 2026-09-23.** + `PasswordAuthentication no` removed the only authentication method that laptop had. + It had never been given an SSH key; password auth had been carrying it silently, and + `ansible/files/` held exactly one public key — the break-glass one — with no + mechanism for a workstation key at all. + + This entry checks what the change does to `sshd` in considerable detail and never + asks **who was authenticating with a password**. That question is the one that + mattered, it costs one command (`grep -c 'sshd.*Accepted password' /var/log/auth.log` + or `lastlog`), and nobody asked it — including in review. + + Recovered with the break-glass key, which is now tested for the first time since + August (§1.2). Fixed properly by `ansible/playbooks/admin-keys.yml` and + `ansible/files/admin-keys/`, so workstation keys are in git and land on any host that + is rebuilt or returns from the dead. + + **Before hardening an authentication path, enumerate who is using the method you are + about to remove.** Not what the config says — who is actually authenticating. + **One host silently missed the rollout, and that is the part to remember.** `opi-zero2w-2` went UNREACHABLE mid-run — `Connection timed out during banner exchange` — and the play carried on through the remaining three hosts and @@ -2482,6 +2525,17 @@ Against: 2 GB RAM, a 2014 SoC, eMMC of unknown wear, and `LAB-DESIGN.md` calls t flaky. The Zero 2W spares are 4 GB, 64-bit and on current Debian — but they are WiFi, which is the whole problem. +- [ ] **Remove the workstation key that today's recovery pushed to it.** On 2026-09-23 the + ad-hoc admin-key push targeted `all:!x86_nodes:!embedded`, which includes `xu3-1`, so + the Windows laptop's key is now in `~swares/.ssh/authorized_keys` there. + `admin-keys.yml` excludes the host going forward, but exclusion does not remove what + is already present: + + ansible xu3-1 -i inventory/hosts.yml -m ansible.posix.authorized_key \ + -a "user=swares state=absent key='$(cat files/admin-keys/wares-windows.pub)'" + + A key granting `swares` — and therefore `NOPASSWD: ALL` — on a host running OpenSSL + 1.0.2g is the one place in the lab not to put one. - [ ] **Decide: retire, or rebuild onto current Armbian with a real job.** Deferred deliberately 2026-09-20 — worth considering a use before powering it off, and the DNS-host question above is the one that might justify it. @@ -3931,6 +3985,19 @@ Grep patterns run under `sudo` land in the journal they are searching. 2.4 GHz actually helps on that board before putting it in Ansible, or the repo acquires a fix nobody verified. + **SECOND OCCURRENCE, DIFFERENT BOARD — 2026-09-23.** `opi-zero2w-1` was + `UNREACHABLE` with **`No route to host`** on `192.168.1.184` during the admin-key + push, so it is the only host in the fleet without the new workstation key. Note the + error differs from `-2`'s: a banner-exchange timeout is a slow link, *no route* is a + box that is off, hung, or off the network entirely. + + That is **two of the four Zero 2W boards dropping out of fleet-wide changes in four + days**, and `opi-zero2w-1` is the DNS **tertiary** resolver. The `dns-*` blackbox + probes from §3.3 should have caught it — checking whether they did is the free test. + `ansible/playbooks/admin-keys.yml` exists partly because of this: a key pushed by an + ad-hoc command leaves a downed host permanently behind, where a committed playbook + catches it up whenever it returns. + **FIRST OPERATIONAL COST — 2026-09-20.** Until now this entry rested on signal measurements: `-63 dBm`, `87.8 Mbit/s (MCS 2)`, `38/168 ms`. Today the link did something. diff --git a/CLAUDE.md b/CLAUDE.md index 91d060f..6064c5a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -100,6 +100,20 @@ Read this before acting. Full context is in `docs/` (start with `ARCHITECTURE.md - Never read aloud, echo, or commit: `/etc/restic/password`, `ansible/.vault_pass`, `ansible/files/pull-secret.json`, any kubeconfig or k3s token. These are `.gitignore`d — keep it that way. +- **SSH is key-only now, fleet-wide, and that has bitten once.** §2.15 set + `PasswordAuthentication no` everywhere on 2026-09-20; three days later it locked the + Windows laptop out of all fourteen hosts, because that laptop had never been given a key + and password auth had been carrying it silently. Recovery was the break-glass key, which + worked and is finally tested (§1.2). **Workstation keys live in + `ansible/files/admin-keys/*.pub`, installed by `ansible/playbooks/admin-keys.yml`** — add + them there, never by hand, or a rebuilt host or one returning from downtime will be + missing them. `xu3-1` is excluded deliberately (§2.16). Before removing any + authentication method, **enumerate who is actually using it**, not what the config says. +- The break-glass private key belongs offline, in the envelope — not on a workstation. On + 2026-09-23 it was found in a Windows home directory as `break-glass.txt`, which is + `swares` plus `NOPASSWD: ALL` (`bootstrap.yml:41`) on every Linux host. It was also the + only way back in that day, so the answer is a tested key plus a working workstation key, + not a shredded key and no route. ## Verify before asserting diff --git a/ansible/files/admin-keys/wares-windows.pub b/ansible/files/admin-keys/wares-windows.pub new file mode 100644 index 0000000..90daea5 --- /dev/null +++ b/ansible/files/admin-keys/wares-windows.pub @@ -0,0 +1 @@ +ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILPgYHjT8hul/EcyKgn+Iz075voR2Jxzh9i8dzCQyRus wares@DESKTOP-SL9D1QM diff --git a/ansible/playbooks/admin-keys.yml b/ansible/playbooks/admin-keys.yml new file mode 100644 index 0000000..9b83bd4 --- /dev/null +++ b/ansible/playbooks/admin-keys.yml @@ -0,0 +1,124 @@ +--- +# Install workstation admin SSH public keys on every managed Linux host. +# +# WHY THIS EXISTS — 2026-09-23 +# +# §2.15 disabled `PasswordAuthentication` fleet-wide on 2026-09-20. Two days +# later the Windows laptop could not log in anywhere: +# +# no supported authentication method. server sent: public key +# +# The hardening was right. What was missing is that the laptop had never had a +# key on those hosts — password auth had been carrying it, and nobody asked who +# was authenticating with a password before removing the option. `ansible/files/` +# held exactly one public key, the break-glass one, and no mechanism existed for +# a workstation key at all. +# +# Recovery was the break-glass key, which worked — see BACKLOG §1.2, whose +# "test it" checkbox had been open since August and was finally answered by a +# real lockout. The laptop key was then pushed by an ad-hoc `ansible -m +# authorized_key` command, which is not reproducible and left `opi-zero2w-1` +# (down at the time) permanently one key behind. This play is that command, made +# reproducible. +# +# TO ADD A WORKSTATION: drop its public key in `ansible/files/admin-keys/` as +# `.pub` and run this. Public keys are not secrets; committing them is the +# point, so a rebuilt or returning host gets every key without anyone +# remembering. +# +# Run: +# ansible-playbook -i inventory/hosts.yml playbooks/admin-keys.yml --check +# ansible-playbook -i inventory/hosts.yml playbooks/admin-keys.yml +# +# To remove a key, delete the file and re-run with -e admin_keys_prune=true. +# That is opt-in on purpose: a play that silently prunes authorized_keys is a +# lockout waiting for a typo in a filename. +- name: Install workstation admin SSH keys + # xu3-1 is deliberately EXCLUDED, decided 2026-09-23. It is in the usual + # pattern and did receive this key during the ad-hoc recovery push, but it runs + # Ubuntu 16.04 with OpenSSL 1.0.2g and is pending a retire-or-rebuild decision + # (§2.16). A key granting `swares` — which `bootstrap.yml` gives NOPASSWD sudo + # — on the least maintained host in the lab is the one place not to put it. + # The recovery copy was removed by hand; see §2.16. + hosts: "{{ admin_key_hosts | default('all:!x86_nodes:!embedded:!xu3-1') }}" + become: true + gather_facts: false + + vars: + admin_key_dir: "{{ playbook_dir }}/../files/admin-keys" + admin_key_user: "{{ ansible_user | default('swares') }}" + admin_keys_prune: false + + tasks: + - name: Collect the public keys to install + ansible.builtin.find: + paths: "{{ admin_key_dir }}" + patterns: "*.pub" + delegate_to: localhost + become: false + run_once: true + check_mode: false # read-only; must run under --check or the asserts below are vacuous + register: admin_key_files + + # An empty key directory would otherwise produce a play that loops over + # nothing and reports success on every host — the exact failure this play was + # written after. See the next assert for the same lesson one level down. + - name: There must be at least one key to install + ansible.builtin.assert: + that: + - admin_key_files.files | length > 0 + fail_msg: >- + No *.pub files in {{ admin_key_dir }}. Refusing to run: a play that + installs nothing and reports success is worse than one that fails. + quiet: true + run_once: true + + # THE EMPTY-KEY LESSON, and why this assert exists. + # + # On 2026-09-23 the recovery push ran with an unset shell variable, so + # `ansible.posix.authorized_key` received `key: ""`. It did not error. It + # reported **SUCCESS on thirteen hosts** with `changed: false`, and anyone + # reading that output would conclude the key was deployed. It was not. + # + # So every key is validated as a key before anything is installed — shape + # checked, not merely non-empty, because `key: "hello"` would also have been + # accepted silently. + - name: Every file must contain a public key + ansible.builtin.assert: + that: + - lookup('file', item) is match('^(ssh-ed25519|ssh-rsa|ecdsa-sha2-[a-z0-9-]+|sk-ssh-ed25519@openssh\.com) AAAA') + fail_msg: >- + {{ item | basename }} does not look like an SSH public key. Refusing to + install any key: authorized_key accepts an empty or malformed value and + reports success, so nothing downstream would tell you. + quiet: true + loop: "{{ admin_key_files.files | map(attribute='path') | list }}" + loop_control: + label: "{{ item | basename }}" + run_once: true + + - name: Install each admin key for {{ admin_key_user }} + ansible.posix.authorized_key: + user: "{{ admin_key_user }}" + key: "{{ lookup('file', item) }}" + state: present + exclusive: false + loop: "{{ admin_key_files.files | map(attribute='path') | list }}" + loop_control: + label: "{{ item | basename }}" + + # `exclusive: false` above means removing a file does not remove the key from + # the hosts. Pruning is a separate, opt-in pass so that a renamed or + # accidentally deleted file cannot lock anyone out. It keeps the break-glass + # key and the Ansible control-node key, neither of which lives in this + # directory. + - name: Report that pruning was not attempted + ansible.builtin.debug: + msg: >- + Keys in {{ admin_key_dir }} are installed. Keys already on the hosts but + NOT in that directory were left alone — including the break-glass key + (§1.2) and the control-node key from bootstrap.yml. Re-run with + -e admin_keys_prune=true to remove a key you have deleted from the + directory, and read `authorized_keys` on one host first. + run_once: true + when: not admin_keys_prune | bool