From 6587c797e86d1a7a19d34e24150e8253639b3e45 Mon Sep 17 00:00:00 2001 From: Jens Ahrensfeld Date: Sun, 26 Jul 2026 14:33:24 +0200 Subject: [PATCH] Add automated post-boot sanity check Codifies every manual verification done throughout today's 22.04->24.04 upgrade work into a single script: package consistency, SSH hardening, DNS/DNSSEC, full mail stack (including a live rspamd scan test - would have caught today's phantom-active rspamd immediately), MariaDB/vmail, fail2ban jails and custom hardening, all public vhosts, and Docker containers. Runs automatically once per boot via a systemd oneshot unit, since every significant change so far has ended in a reboot anyway; also safe to run manually. Emails a pass/warn/fail summary to jens@jayfield.org via the local mail stack. Tested live: manual run and an actual reboot both produced identical clean results (44 pass, 1 warn - pending routine package updates, 0 fail). Co-Authored-By: Claude Sonnet 5 --- README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/README.md b/README.md index 486b8d0..5f0291b 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,32 @@ incomplete 20.04→22.04 upgrade (breaking apt's certbot via a `pytz`/`zoneinfo` incompatibility) — that whole incomplete-upgrade saga is now fully resolved, see `TODO.md`. +## Post-boot sanity check + +`/usr/local/sbin/sanity-check.sh` (added 2026-07-26, after the 24.04 +upgrade turned up two real post-upgrade breakages — see `TODO.md`) checks +that the whole stack is actually healthy, not just "systemd says active": +package/dpkg consistency, disk space, SSH hardening +(`PermitRootLogin`/`PasswordAuthentication`), BIND + DNSSEC validation + +the dyndns zone, the full mail stack including a live `rspamc symbols` +scan (catches a phantom-active rspamd, which is exactly what happened +during the 24.04 upgrade) and whether anything is actually listening on +Postfix's configured milter port, MariaDB/`vmail` data + Postfix's SQL +lookups, fail2ban's jail count and the custom `findtime` hardening, all 6 +public vhosts by expected HTTP status code, and all 5 Docker containers. + +Runs automatically once per boot via `sanity-check.service` (a oneshot +unit, `systemctl enable`d, `After=` all the relevant services plus a 10s +settle delay before running) — covers OS upgrades, kernel updates, and +any other change that ends in a reboot, which in practice has been all of +them so far. Also safe to run manually after a change that *doesn't* +reboot (a plain `apt upgrade`, a config edit): `sudo +/usr/local/sbin/sanity-check.sh`. Each run logs to +`/var/log/sanity-check/.log` and emails a pass/warn/fail +summary to `jens@jayfield.org` via the local mail stack (no external +dependency). Exits non-zero if anything actually failed (warnings, like +routine pending package upgrades, don't fail the run). + ## System users | User | uid | Home | Purpose |