Fix BIND's broken DNSSEC validation found while prepping OS upgrade

Found while investigating flaky third-party apt repo resolution
(download.docker.com, ppa.launchpad.net): BIND's own recursive
resolver was returning SERVFAIL for essentially any DNSSEC-signed
external domain, root-caused via dnssec.log to a stale/inconsistent
root trust-anchor state, likely dating to when full local recursion
was first enabled on 2026-07-19. Reset managed-keys.bind to force a
clean RFC 5011 re-bootstrap; verified previously-failing domains now
resolve correctly and the authoritative jayfield.org zone is
unaffected.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

TODO.md

@@ -3,6 +3,47 @@
 Outstanding hardening items (see `README.md` for the full service overview).
 Nothing here is urgent; all are low-risk, no-downtime changes.
 
+- [x] **BIND's own DNSSEC validation was broadly broken for signed domains** — found and fixed 2026-07-26
+      while investigating why third-party apt repos (`download.docker.com`, `ppa.launchpad.net`) kept
+      failing to resolve ("Temporary failure resolving") during prep for an OS release upgrade.
+      `dig`-direct queries and `apt`/`curl` (via the real NSS/`systemd-resolved` path,
+      `127.0.0.53` → forwards to local BIND on `127.0.0.1`) gave inconsistent results for the
+      same hostname, which pointed at BIND's own recursion rather than a network flake.
+      Confirmed with a quick sample: `github.com`, `launchpad.net`, `ppa.launchpad.net`,
+      `deb.debian.org`, and `download.docker.com` **all** returned `SERVFAIL` when queried
+      directly against `127.0.0.1`, every time (4/4 reproductions), while unsigned zones
+      (e.g. `archive.ubuntu.com`) resolved fine — i.e. broken specifically for DNSSEC-signed
+      domains, not domain-specific and not transient. Ruled out fragmentation/router
+      interference: forcing `+tcp`, `+bufsize=512`, and `+noedns` against BIND all still
+      SERVFAILed, while the same lookups against `8.8.8.8` over the same network path
+      succeeded fine (`NOERROR`) — so it wasn't a UDP/EDNS size or FritzBox path issue.
+      `dnssec.log` (one of the per-category log files already split out in
+      `named.conf.logging`) gave the exact cause: `validating com/DS: no valid signature
+      found`. `rndc managed-keys status` showed two root trust-anchor keys — key id `20326`
+      "trusted since" 2022-03-17 and key id `38696` "trusted since" 2025-02-10 (consistent
+      with the real 2025 ICANN root KSK rollover) — both looked superficially fine (neither
+      marked for removal), but validation against the live root zone was failing regardless.
+      `bind9`/`bind9-libs`/`dns-root-data` were already at the latest available jammy
+      versions, ruling out a simple stale-package fix. Root cause is almost certainly that
+      this BIND instance only started doing full local recursive resolution on 2026-07-19 (see
+      the forwarders-removed entry below) — before that it was forwarding to the ISP's
+      resolver, so it was never actively tracking/validating the root DNSSEC chain itself,
+      and its RFC 5011 managed-keys state was stale/inconsistent by the time real validation
+      started happening.
+      Fixed with the standard recovery procedure for this class of problem: stopped `named`,
+      backed up `/var/cache/bind/managed-keys.bind`(`.jnl`) to
+      `/root/removed-configs-backup/bind-managed-keys-reset-20260726131408/`, deleted the
+      live copies (forcing BIND to rebuild its trust anchor from scratch via RFC 5011 on next
+      start, exactly as a fresh install would), restarted `named`. Verified: all 5 previously
+      SERVFAILing domains above now return `NOERROR`; `jayfield.org`'s own authoritative zone
+      still resolves correctly (unaffected by the restart); `rndc managed-keys status` now
+      shows both root keys freshly re-trusted as of the restart timestamp, confirming a real
+      re-validation happened rather than a fluke.
+      **Why this mattered beyond the immediate apt problem**: this box uses BIND as its
+      general OS resolver (via `systemd-resolved` → `127.0.0.1`), so this was silently
+      breaking DNSSEC validation for *any* signed external domain something on this host
+      might need to reach — not just apt/PPA lookups.
+
 - [ ] **Mail server IP (`89.58.8.149`) listed on Barracuda's BRBL** — found
       2026-07-25 (user report: "jayfield.org is on the Barracuda MX
       blacklist"). Confirmed via `dig +short 149.8.58.89.b.barracudacentral.org A`