diff --git a/README.md b/README.md index ff81a39..486b8d0 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,7 @@ # alpha.jayfield.org — Web Services Overview -Host: `alpha.jayfield.org` (`89.58.8.149`), Ubuntu 22.04 LTS. +Host: `alpha.jayfield.org` (`89.58.8.149`), Ubuntu 24.04 LTS (upgraded from +22.04 on 2026-07-26 — see `TODO.md` for the full rollout). SSH: `ssh -p 10022 jens@alpha.jayfield.org` All administration in this repo's notes is performed remotely against @@ -16,18 +17,21 @@ Portainer). Apache reverse-proxies into the Docker layer where needed. | Service | Role | |---|---| -| `apache2` (2.4.52) | Terminates HTTP/HTTPS for all vhosts, reverse-proxies to backends (Docker containers, rspamd, Portainer) | +| `apache2` (2.4.58) | Terminates HTTP/HTTPS for all vhosts, reverse-proxies to backends (Docker containers, rspamd, Portainer) | | `bind9` | Authoritative DNS for the `jayfield.org` zone (`ns1`/`ns2.jayfield.org` are both this host); also the box's own recursive resolver — see below | | `docker` | Runs the containerized web services (see below) | -| `certbot` (apt, 1.21.0) | Let's Encrypt cert issuance/renewal, via `certbot.timer` (systemd) | +| `certbot` (apt, 2.9.0) | Let's Encrypt cert issuance/renewal, via `certbot.timer` (systemd) | | Postfix (`master`) | SMTP, ports 25/587, on the public IP + localhost | | Dovecot | IMAP/IMAPS (143/993) + ManageSieve (4190) | -| rspamd | Spam filtering for the mail stack, localhost-only (11332–11334), proxied at `mail.jayfield.org/rspamd` | +| rspamd (4.1.3, `rspamd.com` PPA — not the OS's own older native package) | Spam filtering for the mail stack, localhost-only (11332–11334), proxied at `mail.jayfield.org/rspamd` | | redis-server | Localhost-only (6379); backs rspamd's Bayes classifier, history, and reputation caching | +| `mariadb` (10.11.14, noble-native) | Holds `vmail` (Postfix/Dovecot virtual accounts) — see `TODO.md` for the `10.3`→`10.6`→`10.11` migration history | -Python: `python3.10.12` (fixed from a stalled `python3.8` leftover of an -incomplete 20.04→22.04 upgrade — this was breaking apt's certbot via a -`pytz`/`zoneinfo` incompatibility). +Python: `python3.12.3` (noble's default). Previously `python3.10.12` on +22.04, itself fixed from a stalled `python3.8` leftover of the original +incomplete 20.04→22.04 upgrade (breaking apt's certbot via a +`pytz`/`zoneinfo` incompatibility) — that whole incomplete-upgrade saga is +now fully resolved, see `TODO.md`. ## System users diff --git a/TODO.md b/TODO.md index 3259773..5de73bf 100644 --- a/TODO.md +++ b/TODO.md @@ -106,8 +106,84 @@ Nothing here is urgent; all are low-risk, no-downtime changes. (`postmap -q ... mysql:/etc/postfix/sql/{domains,accounts}.cf`) resolve correctly against the upgraded database. No galera clustering was actually in use (just an unused compiled-in dependency), so the `galera-3`→`4` swap needed no further action. - The 22.04→24.04 release upgrade itself is still pending, now that this release is - actually in a clean, consistent state to upgrade *from*. + +- [x] **22.04 → 24.04 (Noble) release upgrade** — completed 2026-07-26, once the box was in a + clean, consistent 22.04 state (all the items above). Confirmed beforehand that the + dyndns app's PHP 7.4 dependency (already EOL) still has a `noble` build in Ondřej's PPA + (initial check of the plain `Packages` file gave a false "not available" — that file + 404s uncompressed, only `Packages.gz` is actually published; the correct check confirmed + 7.4 through 8.5 all build for `noble`), so no PHP migration was needed first. + **How it was run, given exactly what killed the *previous* upgrade attempt** (the + 2026-07-18 one documented above, stuck forever at an unanswered dpkg prompt): launched + `do-release-upgrade -m server -f DistUpgradeViewNonInteractive` fully detached + (`setsid`/`nohup`, stdin from `/dev/null`, logged to a file), so it can't hang waiting on + an interactive answer and survives regardless of what happens to the SSH session. Did a + full test reboot first (not just service restarts) to confirm the box comes back up + cleanly on its own — it did, DNS/DNSSEC/mail/Docker all fine — before relying on the same + mechanism for the upgrade's own final reboot. + First attempt refused to start ("Please install all available updates for your release + before upgrading"): 12 more packages turned out to be left over from the *original* + 2026-07-18 incomplete upgrade that the earlier cleanup pass had missed (`fuse`→`fuse3`, + `fwupd`, `open-vm-tools`, `passwd`, `mailutils`, etc. — all needing package replacement, + same pattern as the `mariadb-server` transition). `apt full-upgrade --dry-run` confirmed + a clean transition (11 upgraded, 20 newly installed, only 5 old-focal-era libraries + removed, e.g. `libmailutils6`→`8`), applied it, then relaunched. + Second run completed the package transition successfully (confirmed via + `/etc/os-release` → `Ubuntu 24.04.4 LTS`), but not before an alarming few minutes where + SSH was completely unreachable (`Connection refused`, not a timeout) — turned out to be + `openssh-server` itself being replaced partway through installing/configuring hundreds + of packages, not an actual reboot or network failure (`ssh.service`'s own "active since" + timestamp landed right in the middle of that window). The non-interactive frontend did + hit a real conffile prompt this time (`/etc/fail2ban/jail.conf`) and — unlike the + 2026-07-18 attempt — auto-answered it instead of hanging, proving the fix works. Also + correctly recognized `php7.4`/`php8.0`/`rspamd` as "foreign" (third-party-sourced, + repos disabled during the upgrade) and left them alone rather than deleting them as + obsolete — except it did delete `rspamd` anyway (see below). Rebooted into the new + kernel (`6.8.0-136-generic`) the same way as the earlier test reboot; came back up + cleanly again. + **Two real problems found post-reboot, both fixed same day:** + - `fail2ban` failed to start: `Bad value substitution ... 'ignorecommands_dir' which is + not a valid option name` in the `apache-fakegooglebot` jail. Root cause: the + non-interactive frontend kept our existing `jail.conf` at the earlier conffile prompt + (we've never touched it directly — all our real customization lives in + `jail.d/*.local` drop-ins), but the *upgraded* `fail2ban` package expects a + `[DEFAULT]` value for `ignorecommands_dir` that isn't defined anywhere in either the + old or the freshly-`--force-confnew`-reinstalled `jail.conf` — a genuine gap in the + `noble` package (the referenced script directory, + `/etc/fail2ban/filter.d/ignorecommands/`, does exist on disk from the package's own + file list, just nothing points `ignorecommands_dir` at it). Fixed with a small owned + drop-in, `jail.d/ignorecommands-dir.local`, defining it — same convention as our other + fail2ban customizations, no package-owned file touched. Verified: all 19 jails present, + custom `findtime=1d` sshd/dovecot hardening intact. + - `rspamd` was **fully removed** (`dpkg -l` showed `rc`, not just held back like + `php7.4`/`php8.0` were) — its `systemd` unit was also masked (symlinked to `/dev/null`) + by the upgrade tool as a safety measure for a package it could no longer account for. + Its `.list` source was gone entirely too (unlike Docker/PHP, whose disabled entries + were left commented-out) — the upgrade tool had migrated it to the newer deb822 + `.sources` format instead (`rspamd.sources`, `Enabled: no`, already correctly + re-pointed at `noble`). Docker's source, meanwhile, was gone with *no* `.sources` + replacement at all (had to recreate `docker.list` from scratch, keyring was still + intact). Fixed: enabled the `rspamd.sources` and `ondrej-ubuntu-php-*.sources` files, + recreated `docker.list` pointed at `noble`, confirmed all three fetch cleanly, then + `apt install rspamd` (candidate correctly resolved to the same `4.1.3` build, not + Ubuntu's own older native `3.8.1-1ubuntu3` noble package). Verified for real this + time, not just "service active" (which had been a false positive from a legacy SysV + init shim silently succeeding with no binary behind it): real daemon process, + listening on `11332`/`11333`/`11334`, `rspamadm configtest` clean, and + `rspamc symbols` against a test message showing genuine scanning (DMARC checks, ARC + signing with the real `jayfield.org`/`2022` key, full symbol scoring) — not just an + empty process. Note for later: `milter_default_action = accept` in Postfix meant mail + kept flowing the whole time rspamd was down, just completely unfiltered and unsigned + — not a full outage, but worth knowing that gap exists. + Full verification pass after all fixes: SSH hardening intact + (`PermitRootLogin no`/`PasswordAuthentication no`), DNSSEC still validating + (`github.com` → `NOERROR`), `jayfield.org` zone resolving correctly, all 5 Docker + containers up without disruption, all 7 Apache vhosts responding as expected, the dyndns + app verified end-to-end (real per-host-credentialed update via `vpn`'s login → correct + TSIG key → BIND → `dig` confirms). `mariadb-server` was upgraded again automatically as + part of this same run, `10.6.23`→`10.11.14` (noble's native version) — verified `vmail` + schema intact (1 domain, 2 accounts, 4 aliases, unchanged), `doveadm`/Postfix MySQL + lookups both still correct. - [ ] **Mail server IP (`89.58.8.149`) listed on Barracuda's BRBL** — found 2026-07-25 (user report: "jayfield.org is on the Barracuda MX