docs-alpha.jayfield.org/TODO.md

alpha.jayfield.org — TODO

Outstanding hardening items (see README.md for the full service overview). Nothing here is urgent; all are low-risk, no-downtime changes.

• Weekly attacker-summary report routing to root@jayfield.org — done 2026-07-26. attacker-check.py's report email was pointed at root@jayfield.org, but that address didn't resolve to anything (not in /etc/aliases, not in vmail.aliases) — would have bounced every week. Added it as a new vmail.aliases row → jens@jayfield.org, matching the existing postmaster/hostmaster/webmaster/wlan pattern. Also added per-run dated report files (/var/log/attacker-check/reports/<date>.txt), alongside the two existing append-only CSVs. Verified: postmap -q root@jayfield.org mysql:/etc/postfix/sql/aliases.cf resolves correctly, a live re-run delivered to jens@jayfield.org (orig_to=<root@jayfield.org> in the Postfix log) and wrote reports/2026-07-26.txt.

• BIND's own DNSSEC validation was broadly broken for signed domains — found and fixed 2026-07-26 while investigating why third-party apt repos (download.docker.com, ppa.launchpad.net) kept failing to resolve ("Temporary failure resolving") during prep for an OS release upgrade. dig-direct queries and apt/curl (via the real NSS/systemd-resolved path, 127.0.0.53 → forwards to local BIND on 127.0.0.1) gave inconsistent results for the same hostname, which pointed at BIND's own recursion rather than a network flake. Confirmed with a quick sample: github.com, launchpad.net, ppa.launchpad.net, deb.debian.org, and download.docker.com all returned SERVFAIL when queried directly against 127.0.0.1, every time (4/4 reproductions), while unsigned zones (e.g. archive.ubuntu.com) resolved fine — i.e. broken specifically for DNSSEC-signed domains, not domain-specific and not transient. Ruled out fragmentation/router interference: forcing +tcp, +bufsize=512, and +noedns against BIND all still SERVFAILed, while the same lookups against 8.8.8.8 over the same network path succeeded fine (NOERROR) — so it wasn't a UDP/EDNS size or FritzBox path issue. dnssec.log (one of the per-category log files already split out in named.conf.logging) gave the exact cause: validating com/DS: no valid signature found. rndc managed-keys status showed two root trust-anchor keys — key id 20326 "trusted since" 2022-03-17 and key id 38696 "trusted since" 2025-02-10 (consistent with the real 2025 ICANN root KSK rollover) — both looked superficially fine (neither marked for removal), but validation against the live root zone was failing regardless. bind9/bind9-libs/dns-root-data were already at the latest available jammy versions, ruling out a simple stale-package fix. Root cause is almost certainly that this BIND instance only started doing full local recursive resolution on 2026-07-19 (see the forwarders-removed entry below) — before that it was forwarding to the ISP's resolver, so it was never actively tracking/validating the root DNSSEC chain itself, and its RFC 5011 managed-keys state was stale/inconsistent by the time real validation started happening. Fixed with the standard recovery procedure for this class of problem: stopped named, backed up /var/cache/bind/managed-keys.bind(.jnl) to /root/removed-configs-backup/bind-managed-keys-reset-20260726131408/, deleted the live copies (forcing BIND to rebuild its trust anchor from scratch via RFC 5011 on next start, exactly as a fresh install would), restarted named. Verified: all 5 previously SERVFAILing domains above now return NOERROR; jayfield.org's own authoritative zone still resolves correctly (unaffected by the restart); rndc managed-keys status now shows both root keys freshly re-trusted as of the restart timestamp, confirming a real re-validation happened rather than a fluke. Why this mattered beyond the immediate apt problem: this box uses BIND as its general OS resolver (via systemd-resolved → 127.0.0.1), so this was silently breaking DNSSEC validation for any signed external domain something on this host might need to reach — not just apt/PPA lookups.

• Cleaned up an incomplete 20.04→22.04 release upgrade — found and fixed 2026-07-26, while investigating third-party apt repos ahead of a planned 22.04→24.04 upgrade. /var/log/dist-upgrade/ showed a do-release-upgrade run from 2026-07-18 that got most of the way through (core OS, Apache, BIND, SSH — matches everything else fixed this week) but stalled at an interactive dpkg conffile prompt for /etc/systemd/resolved.conf and was never resumed. Left behind: mariadb-server-10.3/mariadb-client-10.3 (and galera-3, fuse, a couple of libtss2/libsemanage libs) still installed and marked "pending removal" but never actually replaced by jammy's mariadb-server-10.6; and three third-party apt sources (docker.list, rspamd.list, ondrej-ubuntu-php-focal.list) still pointed at focal, with the PHP PPA disabled entirely (the release-upgrade tool correctly remapped its deb line to jammy but left it commented out — .distUpgrade backups of the originals were left alongside, clearly meant to be manually re-enabled after upgrade completion, which never happened). System itself was stable throughout (dpkg --audit/apt-get check both clean) — not an emergency, just deferred work. Fixed: pointed all three sources at jammy (backed up originals to /root/removed-configs-backup/apt-repo-fixes-20260726132055/ and dyndns-per-host-auth-...-adjacent dirs), confirmed with apt update that all three now fetch cleanly (only worked once the DNSSEC fix above landed — this Docker repo failure had looked like "just" a stale codename, but was actually blocked by the SERVFAIL bug too). Reviewed rspamd's actual 3.11.0→4.1.3 changelog against this host's live local.d/override.d config before upgrading — none of the documented breaking changes (ed25519 DKIM, mx_check symbol renames, neural autolearn, worker ssl=true, custom Lua task:get_urls()/orig address handling) touch anything actually configured here (stock RSA DKIM/ARC, Redis-backed Bayes, milter proxy, no custom Lua). Ran apt upgrade --dry-run first and confirmed zero removals and no mariadb/galera involvement (a plain apt upgrade correctly refuses the mariadb-server metapackage transition since it requires package replacement, not just a version bump) before applying: 77 packages upgraded, 13 newly-installed dependency libraries, docker-ce 5:28.1.1→5:29.6.2, PHP 7.4.33/8.0.30 rebuilt for jammy (still the same, EOL, PHP versions — just properly matching this OS release instead of running focal binaries), rspamd 3.11.0→4.1.3. needrestart flagged ~25 services after the upgrade, including ssh.service, named.service, mariadb.service, postfix@-.service, dovecot.service, and fail2ban.service. Restarted ssh first and deliberately (sshd -t config check, then restart, then verified with a genuinely fresh SSH connection rather than assuming the existing one survived) given it's the only remote access to this box; then the other mail/DNS/security services one at a time with functional checks after each (SMTP/IMAP banners, fail2ban-client status confirming all 19 jails and the custom findtime=1d sshd/dovecot hardening survived, live dig against BIND). Confirmed docker-ce's own postinst had already restarted the daemon automatically (not in needrestart's list) and all 5 containers (portainer, nextcloud, nextcloud-redis, nextcloud-db, web) came back up without disruption; rspamd 4.1.3 config syntax checks clean (rspamadm configtest) with only a pre-existing, non-blocking task-timeout-tuning advisory, web UI reachable. Left user@1001.service (a per-login systemd session, not a real service) unrestarted since restarting it would disrupt the active login for no real benefit. mariadb-server-10.3→10.6 migration completed 2026-07-26: took a full logical backup first (mysqldump --all-databases --routines --triggers --events --single-transaction, ~2MB, verified complete — all 4 databases including vmail present, proper "Dump completed" footer) plus a tarball of /etc/mysql, to /root/db-backups/. Confirmed via apt install --dry-run mariadb-server that the transition was exactly as expected (remove mariadb-server/client(-core)-10.3 + galera-3, install the -10.6/galera-4 equivalents, nothing else touched) before running it for real. Went smoothly: package postinst detected the old 10.3 datadir and ran the system-table upgrade automatically (confirmed via mariadb-upgrade: "already upgraded to 10.6.23-MariaDB... no need to run mysql_upgrade again"). Verified end-to-end afterward — vmail schema intact (1 domain, 2 accounts, 4 aliases, matching pre-upgrade state), doveadm user '*' correctly lists both real mailboxes (jens@/otto2022@jayfield.org), and Postfix's own MySQL lookups (postmap -q ... mysql:/etc/postfix/sql/{domains,accounts}.cf) resolve correctly against the upgraded database. No galera clustering was actually in use (just an unused compiled-in dependency), so the galera-3→4 swap needed no further action.

• 22.04 → 24.04 (Noble) release upgrade — completed 2026-07-26, once the box was in a clean, consistent 22.04 state (all the items above). Confirmed beforehand that the dyndns app's PHP 7.4 dependency (already EOL) still has a noble build in Ondřej's PPA (initial check of the plain Packages file gave a false "not available" — that file 404s uncompressed, only Packages.gz is actually published; the correct check confirmed 7.4 through 8.5 all build for noble), so no PHP migration was needed first. How it was run, given exactly what killed the previous upgrade attempt (the 2026-07-18 one documented above, stuck forever at an unanswered dpkg prompt): launched do-release-upgrade -m server -f DistUpgradeViewNonInteractive fully detached (setsid/nohup, stdin from /dev/null, logged to a file), so it can't hang waiting on an interactive answer and survives regardless of what happens to the SSH session. Did a full test reboot first (not just service restarts) to confirm the box comes back up cleanly on its own — it did, DNS/DNSSEC/mail/Docker all fine — before relying on the same mechanism for the upgrade's own final reboot. First attempt refused to start ("Please install all available updates for your release before upgrading"): 12 more packages turned out to be left over from the original 2026-07-18 incomplete upgrade that the earlier cleanup pass had missed (fuse→fuse3, fwupd, open-vm-tools, passwd, mailutils, etc. — all needing package replacement, same pattern as the mariadb-server transition). apt full-upgrade --dry-run confirmed a clean transition (11 upgraded, 20 newly installed, only 5 old-focal-era libraries removed, e.g. libmailutils6→8), applied it, then relaunched. Second run completed the package transition successfully (confirmed via /etc/os-release → Ubuntu 24.04.4 LTS), but not before an alarming few minutes where SSH was completely unreachable (Connection refused, not a timeout) — turned out to be openssh-server itself being replaced partway through installing/configuring hundreds of packages, not an actual reboot or network failure (ssh.service's own "active since" timestamp landed right in the middle of that window). The non-interactive frontend did hit a real conffile prompt this time (/etc/fail2ban/jail.conf) and — unlike the 2026-07-18 attempt — auto-answered it instead of hanging, proving the fix works. Also correctly recognized php7.4/php8.0/rspamd as "foreign" (third-party-sourced, repos disabled during the upgrade) and left them alone rather than deleting them as obsolete — except it did delete rspamd anyway (see below). Rebooted into the new kernel (6.8.0-136-generic) the same way as the earlier test reboot; came back up cleanly again. Two real problems found post-reboot, both fixed same day: - fail2ban failed to start: Bad value substitution ... 'ignorecommands_dir' which is not a valid option name in the apache-fakegooglebot jail. Root cause: the non-interactive frontend kept our existing jail.conf at the earlier conffile prompt (we've never touched it directly — all our real customization lives in jail.d/*.local drop-ins), but the upgraded fail2ban package expects a [DEFAULT] value for ignorecommands_dir that isn't defined anywhere in either the old or the freshly---force-confnew-reinstalled jail.conf — a genuine gap in the noble package (the referenced script directory, /etc/fail2ban/filter.d/ignorecommands/, does exist on disk from the package's own file list, just nothing points ignorecommands_dir at it). Fixed with a small owned drop-in, jail.d/ignorecommands-dir.local, defining it — same convention as our other fail2ban customizations, no package-owned file touched. Verified: all 19 jails present, custom findtime=1d sshd/dovecot hardening intact. - rspamd was fully removed (dpkg -l showed rc, not just held back like php7.4/php8.0 were) — its systemd unit was also masked (symlinked to /dev/null) by the upgrade tool as a safety measure for a package it could no longer account for. Its .list source was gone entirely too (unlike Docker/PHP, whose disabled entries were left commented-out) — the upgrade tool had migrated it to the newer deb822 .sources format instead (rspamd.sources, Enabled: no, already correctly re-pointed at noble). Docker's source, meanwhile, was gone with no .sources replacement at all (had to recreate docker.list from scratch, keyring was still intact). Fixed: enabled the rspamd.sources and ondrej-ubuntu-php-*.sources files, recreated docker.list pointed at noble, confirmed all three fetch cleanly, then apt install rspamd (candidate correctly resolved to the same 4.1.3 build, not Ubuntu's own older native 3.8.1-1ubuntu3 noble package). Verified for real this time, not just "service active" (which had been a false positive from a legacy SysV init shim silently succeeding with no binary behind it): real daemon process, listening on 11332/11333/11334, rspamadm configtest clean, and rspamc symbols against a test message showing genuine scanning (DMARC checks, ARC signing with the real jayfield.org/2022 key, full symbol scoring) — not just an empty process. Note for later: milter_default_action = accept in Postfix meant mail kept flowing the whole time rspamd was down, just completely unfiltered and unsigned — not a full outage, but worth knowing that gap exists. Full verification pass after all fixes: SSH hardening intact (PermitRootLogin no/PasswordAuthentication no), DNSSEC still validating (github.com → NOERROR), jayfield.org zone resolving correctly, all 5 Docker containers up without disruption, all 7 Apache vhosts responding as expected, the dyndns app verified end-to-end (real per-host-credentialed update via vpn's login → correct TSIG key → BIND → dig confirms). mariadb-server was upgraded again automatically as part of this same run, 10.6.23→10.11.14 (noble's native version) — verified vmail schema intact (1 domain, 2 accounts, 4 aliases, unchanged), doveadm/Postfix MySQL lookups both still correct.

• Mail server IP (89.58.8.149) listed on Barracuda's BRBL — found 2026-07-25 (user report: "jayfield.org is on the Barracuda MX blacklist"). Confirmed via dig +short 149.8.58.89.b.barracudacentral.org A → 127.0.0.2 (listed); clean on Spamhaus ZEN, SpamCop, and SORBS — Barracuda-specific. Checked the server for actual abuse before requesting removal: mail queue empty, outbound volume ~11 msgs over the prior week, only known SASL logins (jens@/otto2022@jayfield.org from the usual home IP), no backscatter, correct PTR (alpha.jayfield.org), SPF/DKIM/DMARC (p=reject) all already live — no sign of a compromised account or open relay, so most likely a stale/inherited reputation hit on the IP rather than active abuse from this host. Submitted a removal request via https://www.barracudacentral.org/rbl/removal-request on 2026-07-25. Confirmation number: BBR21785009436-19117-9182. Per Barracuda's response, the IP gets a temporary 48h reputation bump to "normal" while they investigate (up to ~1h to propagate), after which it either stays clear or reverts to listed. Status 2026-07-26: still clear (dig +short 149.8.58.89.b.barracudacentral.org A → NXDOMAIN, no 127.0.0.2) — matches the 1h check (routine trig_01F1RMhEtNHtAbuHb9ffe1oX, fired 2026-07-25 21:02 UTC, also clear). This is expected during the 48h "normal" bump window, not yet proof of permanent delisting. The real test is the 48h check (routine trig_019GayXSo5TW5eSxiJ99Ghrp, scheduled 2026-07-27 20:03 UTC) — if it reverts to listed, next step is a manual follow-up to Barracuda citing the confirmation number above.

• dyndns app has no real per-user auth binding — found 2026-07-20, fixed 2026-07-26. index.php's pass == '_NO_PASS_' check is a hardcoded magic string, not a real per-user credential — nothing tied the single shared Apache Basic Auth identity (jens, the only .htpasswd account) to which hostname label got updated. Confirmed live in the access log: a FritzBox ("Fritz!Box DDNS/1.0.3") was authenticating as jens while updating the uschi record — i.e. every device was already relying on this exact gap, self-policing which host it claimed via an unauthenticated hostname=/user= param. Fixed with real per-host TSIG binding, not just an app-level check: - Added a dedicated .htpasswd account for each of kack, test, uschi, vpn (jens's existing account/password untouched). - Generated 5 per-host TSIG keys (hmac-sha512, tsig-keygen), one per hostname, each in its own /etc/bind/<host>.dyndns.jayfield.org.key (root:bind 640 + setfacl grant for www-data, same pattern as the original key). - Replaced the zone's flat allow-update { key dyndns.jayfield.org; }; in named.conf.default-zones with an update-policy granting each new key permission to update only its own name (grant "<host>" name <host>. A AAAA TXT;), plus one explicit legacy fallback grant for the original shared key (subdomain rule, zone-wide) — kept temporarily so unmigrated devices don't break (see below). - nsupdate.php/nsupdate_ipv4.php/nsupdate_ipv6.php now sign with the key belonging to $_SERVER['PHP_AUTH_USER'] (the authenticated identity), not whatever hostname was requested. If they match, the new per-host key is used — BIND itself then refuses the update if that key isn't granted that name, independent of any PHP-side check. If the authenticated user is specifically the legacy jens account and doesn't match, it falls back to the old shared key (logged via syslog(LOG_WARNING, ...) for migration tracking) — every other account gets rejected outright (-1) on a mismatch, so newly-created per-host logins can never touch another host, even via the fallback. - Also fixed a pre-existing bug found in nic/update.php while testing this: it printed good <ip> unconditionally, without ever checking nsupdate's return code — so a rejected update silently looked successful to the calling device. Now prints the real dyndns2-protocol response (good <ip> / badauth). All files backed up to /root/removed-configs-backup/dyndns-per-host-auth-20260726094500/. Validated with named-checkconf/apache2ctl configtest, applied via rndc reload/systemctl reload apache2. Tested both by driving nsupdate() directly as www-data and end-to-end over HTTPS with the new vpn credentials: own-host updates succeed, cross-host attempts from new accounts are rejected (-1/badauth) with the target record provably unchanged, and the legacy jens→uschi pattern still succeeds unchanged. Test IPs used during verification were restored to the real production values (test, uschi, vpn) immediately after; live dig confirmed the final state matches the pre-change records exactly. Follow-up: the jens legacy fallback (and its .htpasswd account) should stay only until every real device — including whichever ones are still authenticating as jens to update uschi — has been reconfigured with its own dedicated login. Once dyndns legacy shared-key update stops appearing in syslog, remove the fallback grant and the jens account can be scoped down to just its own host too. vpn migrated 2026-07-26: home FritzBox reconfigured with the new vpn login and confirmed live — after a WAN IP change (real connection recycle, not a test), the access log shows a genuine Fritz!Box DDNS/1.0.3 request authenticated as vpn updating vpn.dyndns.jayfield.org to the new IP (217.229.60.187), and dig confirms the zone picked it up. First host fully off the legacy fallback. uschi's device was reconfigured too but hasn't sent a real authenticated update yet (FritzBox only re-registers on WAN IP change or manual renew, neither has happened since) — not yet confirmed live. kack/test removed 2026-07-26 instead of migrated: kack hadn't updated since Feb 2024 and test showed no real traffic in the (short) per-vhost access log history either — both looked like stale/unused entries rather than active devices, so removed outright rather than given dedicated logins. Deleted their A/AAAA/TXT records via nsupdate (using the legacy shared key), dropped their update-policy grants and include lines from named.conf.default-zones, moved their per-host TSIG key files and .htpasswd entries (htpasswd -D) out. Backed up to /root/removed-configs-backup/dyndns-remove-kack-test-20260726101439/. Validated with named-checkconf/apache2ctl configtest, applied via rndc reload/systemctl reload apache2, confirmed both names now resolve to nothing (dig empty) and jens/uschi/vpn untouched; rndc sync forced the on-disk zone file to reflect it immediately rather than waiting for BIND's next periodic dump. jens.dyndns.jayfield.org hostname removed 2026-07-26 — same story as kack: stale since Feb 2024, no device actually updating it (the only jens-authenticated traffic ever seen was the legacy fallback updating other hosts, never its own). Deleted the record, dropped its dedicated update-policy grant and key include line, moved jens.dyndns.jayfield.org.key to /root/removed-configs-backup/dyndns-remove-jenshost-20260726101844/. Important distinction from the kack/test removal: this only removed the hostname. The jens login and the legacy shared-key fallback grant (grant "dyndns.jayfield.org" subdomain dyndns.jayfield.org. ANY;) were deliberately left in place, since uschi's device still authenticates as jens through that fallback — removing the login would have broken it immediately. Re-verified after this change (direct nsupdate() call as www-data, writing uschi's own current real IP back so nothing actually changed) that the jens→uschi legacy path still returns 0. named-checkconf/ rndc reload clean throughout. Legacy fallback removed entirely 2026-07-26 — confirmed the jens fallback wasn't needed for uschi anymore (its own dedicated login/key from the original fix is what should be used going forward), and with jens's and kack/test's own hostnames already gone, nothing was left depending on it. Removed the grant "dyndns.jayfield.org" subdomain dyndns.jayfield.org. ANY; fallback grant and its key include from named.conf.default-zones; retired the original shared TSIG key itself (/etc/bind/dyndns.jayfield.org.key, in production since March 2022) to /root/removed-configs-backup/dyndns-remove-legacy-fallback-20260726102228/; removed the now-dead legacy-path branch from nsupdate.php/nsupdate_ipv4.php/nsupdate_ipv6.php — nsupdate() now unconditionally rejects (-1) any request where the authenticated user doesn't match the requested host, no exceptions. Verified: jens attempting to update uschi now fails immediately (ret=-1, no BIND round-trip at all — rejected in PHP before ever building an nsupdate command), while uschi and vpn updating themselves still succeed (ret=0, confirmed via dig against their real IPs). Per-user auth binding for dyndns is now unconditional, closing this item out completely — no more grandfathered exceptions of any kind. The jens .htpasswd account itself was left in place (not asked to remove it, and it's now inert for DNS updates anyway since it has no hostname or grant left to use). uschi migration confirmed live 2026-07-26: after the legacy fallback was removed, a real WAN IP change (connection recycle, not a test) produced a genuine Fritz!Box DDNS/1.0.3 request authenticated as uschi updating uschi.dyndns.jayfield.org to 92.252.63.190, confirmed via dig. (One wrinkle along the way: an earlier verification test of mine — writing uschi's old IP back to confirm the fix — landed moments after that device's first post-migration update and clobbered it back to the stale value; the device's next natural IP change overwrote it correctly, so no lasting effect, but a reminder to use non-colliding test values when a real device might update concurrently.) Both uschi and vpn are now fully migrated to dedicated per-host logins with no legacy exceptions remaining anywhere in the setup. vpn re-confirmed 2026-07-26: a second WAN IP change produced another clean authenticated update (87.180.9.214), this time with no test collision — TXT timestamp matches exactly. Both uschi and vpn have now each been confirmed live across a real IP change with their own dedicated login; this item is fully done, nothing further to track. jens password rotated 2026-07-26: since the account is now unused (no hostname or key left to act on) and its old password predates this whole migration, rotated it for hygiene via htpasswd -b. Old .htpasswd backed up to /root/removed-configs-backup/dyndns-jens-rotate-20260726124040/. Verified: new password authenticates fine (200), old password now gets 401, and — as expected since the fix — even a successful jens login still can't perform any DNS update (badauth). Account kept around (not asked to remove it), but fully inert either way.

• dyndns/default-vhost access logging is a no-op — done 2026-07-25 found 2026-07-20 while investigating the command injection below. dyndns.jayfield.org, jayfield.org, www.jayfield.org, mail.jayfield.org, web.jayfield.org, and cloud.jayfield.org had no CustomLog directive of their own, and access.log (only written by default-ssl.conf/000-default.conf) had 0-8 lines — looking like no real vhost was being logged at all. Root cause (2026-07-25 re-check): the fallback assumption was wrong. There's no global CustomLog access.log in apache2.conf; instead conf-enabled/other-vhosts-access-log.conf enables mod_vhost_combined, which was already writing every vhost-without-its-own-CustomLog's traffic to /var/log/apache2/other_vhosts_access.log (5,432 lines, confirmed real traffic incl. bot scans against cloud.jayfield.org). So requests were never actually un-logged — just commingled in one shared, harder-to-grep file instead of access.log. Fixed anyway, since a dedicated per-site log remains the better end state for incident review: added an explicit CustomLog ${APACHE_LOG_DIR}/<site>_access.log combined to every <VirtualHost> block for dyndns, jayfield.org, www, mail, web, cloud, and portainer (the :80 and :443/le-ssl blocks of each site share one log file). Configs backed up to /root/removed-configs-backup/vhost-customlog-20260725203959/. /etc/logrotate.d/apache2 already globs *.log, so no logrotate change needed. Validated with apache2ctl configtest, applied via systemctl reload apache2, verified live with curl against all 7 domains and confirmed each new <site>_access.log recorded the hit.

• OS command injection in the dyndns update scripts — done 2026-07-20 /var/www/dyndns/nsupdate.php, nsupdate_ipv4.php, and nsupdate_ipv6.php (used by both index.php and nic/update.php, the router/ddclient-facing endpoint) built a shell command string by directly interpolating unsanitized $_GET input (hostname/myip/ ip, which become $host/$ip/$ipv4/$ipv6) and passed it to PHP's exec(), which runs it via /bin/sh -c — no escapeshellarg()/escapeshellcmd() anywhere. A request like hostname=$(id>/tmp/pwned).dyndns.jayfield.org would have executed arbitrary shell commands as www-data (the domain-suffix check still passed after the .-split). nic/update.php had no application-level auth of its own at all. Mitigating factor: the whole dyndns.jayfield.org vhost sits behind Apache Basic Auth, so this needed valid (or leaked/brute-forced) dyndns credentials to reach — not exploitable by an anonymous visitor. Compounding factors found during the audit: PHP is 7.4.33, EOL since Nov 2022, disable_functions was empty (exec fully available); index.php's pass == '_NO_PASS_' check is a hardcoded magic string, not real per-user auth, so any valid dyndns credential can update any of the zone's 5 hostnames (jens, kack, test, uschi, vpn), not just its own — left as-is, out of scope for this fix; and the vhost's access logging turned out to be a no-op (shared access.log has 0 lines), so past exploitation couldn't be fully ruled out from logs alone — though file mtimes, www-data processes/crontab, and zone contents all looked unremarkable, consistent with unexploited. Fixed: all three files backed up to /root/removed-configs-backup/dyndns-preinjectionfix-<timestamp>/, then rewritten to build the nsupdate command list as a plain string written to proc_open()'s stdin pipe using the array form of the command (array("/usr/bin/nsupdate", "-k", ...)) — this never invokes a shell at all, so no value can be interpreted as shell syntax regardless of content. Added strict preg_match validation requiring $host/$domain to be shaped like a valid hostname (letters/digits/hyphens, dot-separated labels) and filter_var(..., FILTER_VALIDATE_IP, FILTER_FLAG_IPV4/IPV6) on the IP before use, as defense in depth on top of removing the shell. Regression found and fixed along the way: functional testing showed legitimate updates were already silently failing (update failed: REFUSED, permission denied reading the TSIG key) — turned out www-data was never granted read access to /etc/bind/dyndns.jayfield.org.key when its permissions were tightened to 640 root:bind in the item below, so the dyndns web update feature had been broken for real users since 2026-07-19, independent of this vulnerability. Fixed with a narrowly-scoped POSIX ACL (setfacl -m u:www-data:r /etc/bind/dyndns.jayfield.org.key, acl package installed) rather than adding www-data to the bind group — the group also owns rndc.key (full remote BIND control), which would have been a much bigger privilege grant than dyndns needs. Verified via sudo -u www-data php driving the real functions directly (same code path as the web UI, without needing the Basic Auth password): two injection payloads ($(...) and ;-chained) both rejected (-1, no /tmp side-effect file created), an invalid IP rejected (-1), and a legitimate update succeeded (0), confirmed live in the zone via dig, then cleaned up (test record deleted via nsupdate, temp test script removed). Also confirmed unauthenticated requests still get 401 from Apache before ever reaching the PHP, and apache2ctl configtest stayed clean throughout.

• Public static site (jayfield.org/www.jayfield.org) was 401'ing every visitor — done 2026-07-20 Found while auditing which services are reachable without credentials: a stray /var/www/html/.htaccess (AuthType Basic, same AuthName "Restricted Content"/AuthUserFile /etc/apache2/.htpasswd as the intentional dyndns.jayfield.org protection) was silently requiring login for the DocumentRoot both jayfield.org and www.jayfield.org share — /var/www has AllowOverride all, so it took effect with no vhost-level directive needed. File was dated 2025-02-18 (index.html/apache.html themselves are from 2022) — predates every other change in this doc by well over a year, almost certainly a forgotten leftover rather than intentional, and contradicts README.md's own description of both as a public static site. Fixed: moved to /root/removed-configs-backup/htaccess-var-www-html.<timestamp> (recoverable, not deleted — same convention as this file's other removed-config backups). No Apache reload needed (.htaccess is read per-request). Verified both jayfield.org and www.jayfield.org now return 200 with real page content; dyndns.jayfield.org's own (intentional, vhost-level) Basic Auth is untouched and still returns 401 without credentials.

• Lock down the dyndns TSIG key file permissions — done 2026-07-19 /etc/bind/dyndns.jayfield.org.key was 644 (world-readable), holding the shared secret that authenticates dynamic DNS updates for dyndns.jayfield.org with no IP-based allow-update backing it up. Fixed: chmod 640 + chown root:bind (owner was already correct), matching rndc.key's permissions.

• Suppress BIND version disclosure — done 2026-07-19 No version statement in named.conf.options, so dig CH TXT version.bind @ns1.jayfield.org leaked the exact BIND version (9.18.39) — a minor fingerprinting aid for attackers. Fixed: added version "unknown"; inside the options {} block, validated with named-checkconf, and applied via rndc reload. Verified dig CH TXT version.bind now returns "unknown".

• Add response rate limiting (RRL) — done 2026-07-19 No rate-limit {} block was configured. Since allow-query { any; } is required for a public authoritative zone, RRL is the standard mitigation against the server being used for spoofed-source DNS reflection/amplification against third parties. Fixed: added rate-limit { responses-per-second 10; window 5; } to named.conf.options, validated with named-checkconf, and applied via rndc reload. Started with conservative defaults (10/s, 5s window) — revisit if legitimate high-volume resolvers get throttled.

• Add HSTS headers to all vhosts — done 2026-07-19 No vhost sent a Strict-Transport-Security header, leaving a window for on-path SSL-stripping on a user's very first plaintext request to any domain. Fixed: added Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains" to all 6 domain :443 blocks (dyndns, jayfield.org, www, mail, web, cloud) — skipped default-ssl.conf since it's the self-signed catch-all with no real trusted hostname for a browser to pin. Validated with apache2ctl configtest, applied via systemctl reload apache2, verified live with curl -sI against all 6 domains.

• jayfield.org had no :80 vhost of its own — done 2026-07-19 Bare http://jayfield.org fell through to the dyndns default vhost and redirected to the wrong domain. Fixed: added a :80 block to jayfield.org.conf redirecting to https://jayfield.org/, matching the dyndns/mail pattern. Verified: curl -sI http://jayfield.org/ → 302 with Location: https://jayfield.org/.

• SSH: disable root login + password auth — done 2026-07-19 PermitRootLogin yes and PasswordAuthentication yes were both still enabled (the "known deviation" flagged in SETUP.md §1) — every credential-stuffing bot on the internet got a real login prompt to grind against. Fixed: PermitRootLogin no, PasswordAuthentication no, PubkeyAuthentication yes in /etc/ssh/sshd_config (backed up as sshd_config.bak.20260719225249). The Match Group sftp block's own PasswordAuthentication yes was left as-is — separate, already-chrooted, no-shell/no-forwarding accounts, not part of this risk. Validated with sshd -t, applied via systemctl reload sshd, verified live with sshd -T and a fresh key-only connection before considering it done.

• Portainer's :9443 UI is unreachable in HSTS-enforcing browsers — done 2026-07-20 https://alpha.jayfield.org:9443 threw MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT in Firefox with no click-through option ("keine Ausnahme kann hinzugefügt werden"). Root cause: the includeSubDomains HSTS policy sent by the real alpha.jayfield.org vhost (Strict-Transport-Security header, added in the item above) pins the hostname to HTTPS-with-a-trusted-cert on every port, but Portainer (README.md §Docker services) was still serving its stock self-signed cert directly on 9443. Not a MITM — self-inflicted by combining HSTS with an unproxied self-signed service. Fixed: added a portainer A record to db.jayfield.org (SOA serial 2022032216 → 2022032217); recreated the portainer container binding the HTTPS UI to 127.0.0.1:9443:9443 instead of publishing it (old container kept renamed until the new one was verified, then removed, matching the update procedure documented in README.md); added an Apache vhost for portainer.jayfield.org (ProxyPass/ProxyPassReverse to https://127.0.0.1:9443/, SSLProxyEngine on, SSLProxyVerify none + SSLProxyCheckPeerCN off/ SSLProxyCheckPeerName off for the loopback hop to Portainer's self-signed cert) — same pattern as web.jayfield.org/ cloud.jayfield.org (SETUP.md §4/§5); issued a real cert via certbot --apache -d portainer.jayfield.org --redirect; added the standard HSTS header. Left the 8000 edge-agent tunnel port published as-is — no evidence it's in use. Verified: openssl s_client/curl confirm the Let's Encrypt cert (not self-signed) and Strict-Transport-Security header on https://portainer.jayfield.org, http:// redirects 301 to https://, and ss -ltnp shows 9443 bound to 127.0.0.1 only (via docker-proxy), so it's no longer reachable on the public interface at all — access is now exclusively via https://portainer.jayfield.org. One transient wrinkle during rollout: the very first few requests through the new proxy vhost returned a stray 401 with dyndns's Basic Auth realm ("Restricted Content") — looked alarming, but LogLevel trace8 on the vhost showed Apache correctly SNI-matching and proxying to the new container with a clean backend 200; resolved itself within seconds (looked like a stale pooled proxy connection surviving the container recreate) and was stable across repeated checks before the debug logging was removed again.

• fail2ban: sshd jail was evadable via low-and-slow attempts — done 2026-07-19 Default findtime=30m/maxretry=3 never triggered for 47.76.192.176, which had been hitting root/jayfield logins roughly every 33 minutes for most of a day — just outside the 30-minute window, so failures kept aging out before reaching 3 strikes. Confirmed via fail2ban.log (Found/Ban counts) that no other jail showed the same per-IP timing pattern; a separate, distributed single-shot scan on apache-noscript was noted but left open (see README.md). Fixed: /etc/fail2ban/jail.d/sshd-findtime.local overrides sshd to findtime=1d, maxretry=4, bantime=1d, kept in its own drop-in rather than editing the Debian-packaged jail file. Applied via fail2ban-client reload sshd; 47.76.192.176 was banned on its very next attempt after the change.

• fail2ban: dovecot jail had the same low-and-slow gap as sshd — done 2026-07-25 found while auditing which IPs were hitting webmail hard enough to deserve a ban but weren't in any currently-banned list. Same default findtime=30m/maxretry=3 as sshd's pre-fix config. At least 4 IPs (47.95.196.60 167 hits, 82.157.132.42 144, 120.48.118.132 76, 188.164.195.116 57, 157.245.38.26 34) ran a coordinated IMAP password spray against generic mailbox names (admin@, test@, support@, sales@jayfield.org) starting almost simultaneously around 2026-07-21 22:40-23:38, each pacing itself at ~1 attempt/22-25min — just outside the 30-minute window, so none ever tripped 3 strikes. 120.48.118.132 was still active as late as 2026-07-25 13:32, so the campaign was ongoing, not historical. postfix-sasl shares the same default thresholds and mail stack but showed much lower per-IP volume in this pass — worth a follow-up check if it starts trending the same way. Follow-up 2026-07-26: checked postfix-sasl — still just scattered background noise, not a coordinated campaign. Top offender (172.94.9.203) logged 32 hits but all from late June/early July, none since; current activity is single/double attempts from many different IPs (no repeated per-IP low-and-slow pattern like the dovecot spray). findtime=30m/maxretry=3 defaults left as-is — no fix needed unless this changes. Fixed: /etc/fail2ban/jail.d/dovecot-findtime.local, same values as the sshd fix (findtime=1d, maxretry=4, bantime=1d). Applied via fail2ban-client reload dovecot; verified live with fail2ban-client get dovecot findtime/maxretry/bantime and confirmed the jail's existing ban state (47.88.56.201, 29 total bans) survived the reload.

• 24.04 upgrade left PHP/Docker on stale jammy-built binaries — done 2026-07-26 Found during a docs/scripts recheck: apt list --upgradable showed 30 pending packages, all libapache2-mod-php7.4/php8.0 (ondrej/php PPA, used by the dyndns app) and Docker CE, still installed as +ubuntu22.04.1+deb.sury.org / jammy-tagged builds even though the OS itself moved to noble on 2026-07-19/26. Root cause: do-release- upgrade correctly repoints third-party PPA sources at the new codename (apt-cache policy already showed noble candidates available), but doesn't itself pull down matching rebuilt packages for them — that needs a follow-up apt full-upgrade after the release upgrade completes, which hadn't been re-run. Fixed: sudo apt full-upgrade -y — clean run, "No services need to be restarted", "No containers need to be restarted". Verified: apt list --upgradable now empty; sanity-check.sh full pass (48/48, 0 warn/fail) immediately after; all 6 Docker containers confirmed running (they did restart automatically per their own --restart=always, ~22s uptime at check time) and every public vhost including a genuine Gitea API check returned its expected status code; dyndns.jayfield.org still 401s as expected (PHP module reload didn't disturb Basic Auth). SETUP.md updated to a 24.04 baseline and to document the two previously-undocumented third-party repos (ondrej/php, rspamd.com) a fresh build actually needs, plus this exact "release upgrade ≠ full upgrade" gotcha for next time.