Complete 22.04->24.04 release upgrade

Ran do-release-upgrade detached and non-interactive after the specific
failure mode that killed the 2026-07-18 attempt (an unattended
interactive prompt), validated with a full test reboot first. Caught
and fixed 12 more leftover packages from the original incomplete
upgrade that an earlier cleanup pass had missed. Completed successfully
into Ubuntu 24.04.4 / kernel 6.8.0-136.

Two real post-upgrade problems found and fixed same day: fail2ban
failed to start due to a genuine gap in the noble package (a
[DEFAULT] value referenced but never defined), fixed with an owned
jail.d drop-in; rspamd was fully removed (not just held back like
PHP), its service masked as a safety measure, its apt source migrated
to the new deb822 format and disabled - reinstalled from a re-enabled
noble-pointed repo and verified it's genuinely scanning again, not
just a phantom "active" service.

Full verification pass confirms mail, DNS/DNSSEC, dyndns, Docker,
and SSH hardening all intact. MariaDB auto-upgraded again to noble's
native 10.11.14 as part of this run; vmail data verified unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

TODO.md

@@ -106,8 +106,84 @@ Nothing here is urgent; all are low-risk, no-downtime changes.
       (`postmap -q ... mysql:/etc/postfix/sql/{domains,accounts}.cf`) resolve correctly against
       the upgraded database. No galera clustering was actually in use (just an unused compiled-in
       dependency), so the `galera-3`→`4` swap needed no further action.
-      The 22.04→24.04 release upgrade itself is still pending, now that this release is
-      actually in a clean, consistent state to upgrade *from*.
+
+- [x] **22.04 → 24.04 (Noble) release upgrade** — completed 2026-07-26, once the box was in a
+      clean, consistent 22.04 state (all the items above). Confirmed beforehand that the
+      dyndns app's PHP 7.4 dependency (already EOL) still has a `noble` build in Ondřej's PPA
+      (initial check of the plain `Packages` file gave a false "not available" — that file
+      404s uncompressed, only `Packages.gz` is actually published; the correct check confirmed
+      7.4 through 8.5 all build for `noble`), so no PHP migration was needed first.
+      **How it was run, given exactly what killed the *previous* upgrade attempt** (the
+      2026-07-18 one documented above, stuck forever at an unanswered dpkg prompt): launched
+      `do-release-upgrade -m server -f DistUpgradeViewNonInteractive` fully detached
+      (`setsid`/`nohup`, stdin from `/dev/null`, logged to a file), so it can't hang waiting on
+      an interactive answer and survives regardless of what happens to the SSH session. Did a
+      full test reboot first (not just service restarts) to confirm the box comes back up
+      cleanly on its own — it did, DNS/DNSSEC/mail/Docker all fine — before relying on the same
+      mechanism for the upgrade's own final reboot.
+      First attempt refused to start ("Please install all available updates for your release
+      before upgrading"): 12 more packages turned out to be left over from the *original*
+      2026-07-18 incomplete upgrade that the earlier cleanup pass had missed (`fuse`→`fuse3`,
+      `fwupd`, `open-vm-tools`, `passwd`, `mailutils`, etc. — all needing package replacement,
+      same pattern as the `mariadb-server` transition). `apt full-upgrade --dry-run` confirmed
+      a clean transition (11 upgraded, 20 newly installed, only 5 old-focal-era libraries
+      removed, e.g. `libmailutils6`→`8`), applied it, then relaunched.
+      Second run completed the package transition successfully (confirmed via
+      `/etc/os-release` → `Ubuntu 24.04.4 LTS`), but not before an alarming few minutes where
+      SSH was completely unreachable (`Connection refused`, not a timeout) — turned out to be
+      `openssh-server` itself being replaced partway through installing/configuring hundreds
+      of packages, not an actual reboot or network failure (`ssh.service`'s own "active since"
+      timestamp landed right in the middle of that window). The non-interactive frontend did
+      hit a real conffile prompt this time (`/etc/fail2ban/jail.conf`) and — unlike the
+      2026-07-18 attempt — auto-answered it instead of hanging, proving the fix works. Also
+      correctly recognized `php7.4`/`php8.0`/`rspamd` as "foreign" (third-party-sourced,
+      repos disabled during the upgrade) and left them alone rather than deleting them as
+      obsolete — except it did delete `rspamd` anyway (see below). Rebooted into the new
+      kernel (`6.8.0-136-generic`) the same way as the earlier test reboot; came back up
+      cleanly again.
+      **Two real problems found post-reboot, both fixed same day:**
+      - `fail2ban` failed to start: `Bad value substitution ... 'ignorecommands_dir' which is
+        not a valid option name` in the `apache-fakegooglebot` jail. Root cause: the
+        non-interactive frontend kept our existing `jail.conf` at the earlier conffile prompt
+        (we've never touched it directly — all our real customization lives in
+        `jail.d/*.local` drop-ins), but the *upgraded* `fail2ban` package expects a
+        `[DEFAULT]` value for `ignorecommands_dir` that isn't defined anywhere in either the
+        old or the freshly-`--force-confnew`-reinstalled `jail.conf` — a genuine gap in the
+        `noble` package (the referenced script directory,
+        `/etc/fail2ban/filter.d/ignorecommands/`, does exist on disk from the package's own
+        file list, just nothing points `ignorecommands_dir` at it). Fixed with a small owned
+        drop-in, `jail.d/ignorecommands-dir.local`, defining it — same convention as our other
+        fail2ban customizations, no package-owned file touched. Verified: all 19 jails present,
+        custom `findtime=1d` sshd/dovecot hardening intact.
+      - `rspamd` was **fully removed** (`dpkg -l` showed `rc`, not just held back like
+        `php7.4`/`php8.0` were) — its `systemd` unit was also masked (symlinked to `/dev/null`)
+        by the upgrade tool as a safety measure for a package it could no longer account for.
+        Its `.list` source was gone entirely too (unlike Docker/PHP, whose disabled entries
+        were left commented-out) — the upgrade tool had migrated it to the newer deb822
+        `.sources` format instead (`rspamd.sources`, `Enabled: no`, already correctly
+        re-pointed at `noble`). Docker's source, meanwhile, was gone with *no* `.sources`
+        replacement at all (had to recreate `docker.list` from scratch, keyring was still
+        intact). Fixed: enabled the `rspamd.sources` and `ondrej-ubuntu-php-*.sources` files,
+        recreated `docker.list` pointed at `noble`, confirmed all three fetch cleanly, then
+        `apt install rspamd` (candidate correctly resolved to the same `4.1.3` build, not
+        Ubuntu's own older native `3.8.1-1ubuntu3` noble package). Verified for real this
+        time, not just "service active" (which had been a false positive from a legacy SysV
+        init shim silently succeeding with no binary behind it): real daemon process,
+        listening on `11332`/`11333`/`11334`, `rspamadm configtest` clean, and
+        `rspamc symbols` against a test message showing genuine scanning (DMARC checks, ARC
+        signing with the real `jayfield.org`/`2022` key, full symbol scoring) — not just an
+        empty process. Note for later: `milter_default_action = accept` in Postfix meant mail
+        kept flowing the whole time rspamd was down, just completely unfiltered and unsigned
+        — not a full outage, but worth knowing that gap exists.
+      Full verification pass after all fixes: SSH hardening intact
+      (`PermitRootLogin no`/`PasswordAuthentication no`), DNSSEC still validating
+      (`github.com` → `NOERROR`), `jayfield.org` zone resolving correctly, all 5 Docker
+      containers up without disruption, all 7 Apache vhosts responding as expected, the dyndns
+      app verified end-to-end (real per-host-credentialed update via `vpn`'s login → correct
+      TSIG key → BIND → `dig` confirms). `mariadb-server` was upgraded again automatically as
+      part of this same run, `10.6.23`→`10.11.14` (noble's native version) — verified `vmail`
+      schema intact (1 domain, 2 accounts, 4 aliases, unchanged), `doveadm`/Postfix MySQL
+      lookups both still correct.
 
 - [ ] **Mail server IP (`89.58.8.149`) listed on Barracuda's BRBL** — found
       2026-07-25 (user report: "jayfield.org is on the Barracuda MX