jens/docs-alpha.jayfield.org
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add automated post-boot sanity check

Browse Source 
Codifies every manual verification done throughout today's 22.04->24.04
upgrade work into a single script: package consistency, SSH hardening,
DNS/DNSSEC, full mail stack (including a live rspamd scan test - would
have caught today's phantom-active rspamd immediately), MariaDB/vmail,
fail2ban jails and custom hardening, all public vhosts, and Docker
containers. Runs automatically once per boot via a systemd oneshot
unit, since every significant change so far has ended in a reboot
anyway; also safe to run manually. Emails a pass/warn/fail summary to
jens@jayfield.org via the local mail stack. Tested live: manual run
and an actual reboot both produced identical clean results (44 pass,
1 warn - pending routine package updates, 0 fail).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
jens
2026-07-26 14:33:24 +02:00
co-authored by Claude Sonnet 5
parent 834b420756
commit 6587c797e8
1 changed files with 26 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+26
View File
@@ -33,6 +33,32 @@ incomplete 20.04→22.04 upgrade (breaking apt's certbot via a
`pytz`/`zoneinfo` incompatibility) — that whole incomplete-upgrade saga is `pytz`/`zoneinfo` incompatibility) — that whole incomplete-upgrade saga is
now fully resolved, see `TODO.md`. now fully resolved, see `TODO.md`.
## Post-boot sanity check
`/usr/local/sbin/sanity-check.sh` (added 2026-07-26, after the 24.04
upgrade turned up two real post-upgrade breakages — see `TODO.md`) checks
that the whole stack is actually healthy, not just "systemd says active":
package/dpkg consistency, disk space, SSH hardening
(`PermitRootLogin`/`PasswordAuthentication`), BIND + DNSSEC validation +
the dyndns zone, the full mail stack including a live `rspamc symbols`
scan (catches a phantom-active rspamd, which is exactly what happened
during the 24.04 upgrade) and whether anything is actually listening on
Postfix's configured milter port, MariaDB/`vmail` data + Postfix's SQL
lookups, fail2ban's jail count and the custom `findtime` hardening, all 6
public vhosts by expected HTTP status code, and all 5 Docker containers.
Runs automatically once per boot via `sanity-check.service` (a oneshot
unit, `systemctl enable`d, `After=` all the relevant services plus a 10s
settle delay before running) — covers OS upgrades, kernel updates, and
any other change that ends in a reboot, which in practice has been all of
them so far. Also safe to run manually after a change that *doesn't*
reboot (a plain `apt upgrade`, a config edit): `sudo
/usr/local/sbin/sanity-check.sh`. Each run logs to
`/var/log/sanity-check/<timestamp>.log` and emails a pass/warn/fail
summary to `jens@jayfield.org` via the local mail stack (no external
dependency). Exits non-zero if anything actually failed (warnings, like
routine pending package upgrades, don't fail the run).
## System users ## System users
| User | uid | Home | Purpose | | User | uid | Home | Purpose |