Parses fail2ban.log for the past 7 days into two append-only CSVs
(per-IP and per-jail rollups) for longitudinal analysis, plus a
human-readable summary emailed weekly. Two detection heuristics run on
top of the raw data: distributed single-shot-per-IP scans across a /24
(shape of the apache-noscript campaign), and regular-timing evasion on
repeat hits (shape of the sshd/dovecot low-and-slow campaigns) - both
modeled on real campaigns found earlier this engagement. A live test
run against real logs correctly re-identified the same dovecot
campaign IPs found manually earlier, with matching interval timing.
Runs weekly via cron, Sunday 06:00, clear of logrotate's daily timer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codifies every manual verification done throughout today's 22.04->24.04
upgrade work into a single script: package consistency, SSH hardening,
DNS/DNSSEC, full mail stack (including a live rspamd scan test - would
have caught today's phantom-active rspamd immediately), MariaDB/vmail,
fail2ban jails and custom hardening, all public vhosts, and Docker
containers. Runs automatically once per boot via a systemd oneshot
unit, since every significant change so far has ended in a reboot
anyway; also safe to run manually. Emails a pass/warn/fail summary to
jens@jayfield.org via the local mail stack. Tested live: manual run
and an actual reboot both produced identical clean results (44 pass,
1 warn - pending routine package updates, 0 fail).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran do-release-upgrade detached and non-interactive after the specific
failure mode that killed the 2026-07-18 attempt (an unattended
interactive prompt), validated with a full test reboot first. Caught
and fixed 12 more leftover packages from the original incomplete
upgrade that an earlier cleanup pass had missed. Completed successfully
into Ubuntu 24.04.4 / kernel 6.8.0-136.
Two real post-upgrade problems found and fixed same day: fail2ban
failed to start due to a genuine gap in the noble package (a
[DEFAULT] value referenced but never defined), fixed with an owned
jail.d drop-in; rspamd was fully removed (not just held back like
PHP), its service masked as a safety measure, its apt source migrated
to the new deb822 format and disabled - reinstalled from a re-enabled
noble-pointed repo and verified it's genuinely scanning again, not
just a phantom "active" service.
Full verification pass confirms mail, DNS/DNSSEC, dyndns, Docker,
and SSH hardening all intact. MariaDB auto-upgraded again to noble's
native 10.11.14 as part of this run; vmail data verified unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Backed up all databases (mysqldump) and /etc/mysql before starting;
dry-run confirmed the exact expected package transition. Ran cleanly -
postinst auto-detected and upgraded the old datadir's system tables.
Verified end-to-end: vmail schema intact, doveadm lists both real
mailboxes correctly, and Postfix's MySQL lookups resolve against the
upgraded database. No Galera clustering was actually active, so the
galera-3->4 swap was a no-op in practice.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found the underlying cause of the flaky third-party repos: a
do-release-upgrade from 2026-07-18 stalled mid-way at an interactive
dpkg prompt and was never resumed, leaving mariadb-server-10.3,
docker, rspamd, and php7.4/8.0 all on stale focal-era state. Fixed the
three third-party apt sources (docker, rspamd, ondrej PHP PPA) to
point at jammy - only actually worked once the DNSSEC bug was fixed,
since that was silently blocking repo resolution too. Reviewed
rspamd's 3.11->4.1.3 changelog against the live config before
upgrading (no breaking changes apply here), then ran apt upgrade
(dry-run confirmed zero removals, no mariadb involvement) covering
docker-ce, PHP, rspamd, and 74 other packages. Restarted all
needrestart-flagged services carefully, verifying each (ssh first and
deliberately given it's the only remote access, then DNS/mail/security
services, then containers). Everything confirmed healthy.
MariaDB 10.3->10.6 migration and the actual 22.04->24.04 upgrade are
still pending as deliberately separate next steps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found while investigating flaky third-party apt repo resolution
(download.docker.com, ppa.launchpad.net): BIND's own recursive
resolver was returning SERVFAIL for essentially any DNSSEC-signed
external domain, root-caused via dnssec.log to a stale/inconsistent
root trust-anchor state, likely dating to when full local recursion
was first enabled on 2026-07-19. Reset managed-keys.bind to force a
clean RFC 5011 re-bootstrap; verified previously-failing domains now
resolve correctly and the authoritative jayfield.org zone is
unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Account has no hostname or key left to act on since the auth-binding
fix, but its password predated the whole migration - rotated it via
htpasswd -b. Verified new password works, old one is rejected, and the
account is still fully inert for DNS updates either way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Second real WAN IP change on vpn produced another clean authenticated
update with no test collision this time. Both uschi and vpn have now
each been confirmed live across a real IP change on their dedicated
per-host login - nothing further to track on this item.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README.md's credential-audit table and command-injection writeup still
described the per-user auth binding gap as open; it's been closed since
2026-07-26. SETUP.md's TSIG section still taught the old single-shared-
key pattern for a fresh install - replaced with the per-host key +
update-policy pattern actually running in production now, plus a
pointer to the phased-rollout approach documented in TODO.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real FritzBox update authenticated as uschi landed after a WAN IP
change, confirmed via dig. Documented the earlier test-value collision
where a verification write clobbered the device's first post-migration
update, resolved by its next natural IP change - no lasting impact,
but noted as a lesson for future verification steps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
uschi's device should be using its own dedicated login now, and with
jens/kack/test's own hostnames already removed, nothing needed the
fallback anymore. Removed the update-policy fallback grant, retired
the original shared TSIG key (in production since March 2022), and
deleted the now-dead legacy code path from nsupdate.php and its ipv4/
ipv6 variants. Per-host auth binding for dyndns is now unconditional
with no exceptions. Verified jens can no longer update any host, while
uschi/vpn continue to update themselves normally.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same treatment as kack/test: no device had actually updated jens's own
record since Feb 2024. Removed the DNS record and its dedicated
update-policy grant/key, but deliberately kept the jens login and the
legacy shared-key fallback grant in place, since uschi's device still
depends on it. Re-verified the jens->uschi fallback path still works
after the change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Neither had shown real recent activity (kack not since Feb 2024, test
with no traffic in the per-vhost log's short history), so rather than
give them dedicated per-host logins like vpn/uschi, removed their DNS
records, TSIG keys, update-policy grants, and .htpasswd accounts
outright. Backed up before removal; jens/uschi/vpn unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
FritzBox reconfigured with its dedicated per-host login; a real WAN IP
change after recycling the connection produced a genuine authenticated
update, confirmed in the access log and via dig. First device fully
migrated off the legacy shared-key fallback.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the gap where any dyndns credential could update any of the
zone's 5 hostnames. Confirmed live that a FritzBox was already exposed
to this (authenticating as the shared 'jens' account while updating
'uschi'), so implemented a phased rollout: new per-host TSIG keys and
.htpasswd accounts enforce the binding at the BIND layer itself, with
a logged legacy fallback for the pre-existing shared account only,
so unmigrated devices keep working until reconfigured. Also fixed a
bug where nic/update.php reported "good" even on a rejected update.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were left as "revisit if it escalates" notes from earlier audits.
Checked live logs on alpha via SSH (port 10022): postfix-sasl remains
scattered low-volume noise, not a coordinated low-and-slow campaign like
the dovecot spray; the original three /24s behind the apache-noscript
distributed scan are no longer active, current activity is normal
per-IP jail behavior. No fixes needed for either; also reconfirmed the
dyndns per-user auth binding gap is still present and unfixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Manual dig confirms 89.58.8.149 still delisted as of 2026-07-26, matching
the automated 1h propagation check. Not conclusive yet - Barracuda's
temporary 48h reputation bump is still in effect; the scheduled 48h
routine on 2026-07-27 is the real test of whether removal stuck.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mail IP 89.58.8.149 got flagged on Barracuda's BRBL; local checks found
no sign of actual abuse (empty queue, minimal legit outbound, no
compromised SASL accounts), so removal was requested. Tracking the
confirmation number and scheduled follow-up checks here until resolved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same evasion pattern already fixed for sshd recurred against dovecot: a
coordinated IMAP password spray paced just outside the 30-minute findtime
window ran unbanned for days. Widened dovecot's findtime the same way,
and updated SETUP.md so future provisioning applies this to every
credential jail up front instead of waiting for it to reappear.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real root cause turned out to differ from the original TODO: there was
no broken global CustomLog, mod_vhost_combined was already logging
every vhost without its own CustomLog into other_vhosts_access.log.
Added a dedicated CustomLog per real site anyway for easier incident
review, verified live with curl against all 7 domains after reload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README's public-access table and "previously resolved" summary now
reflect the dyndns injection fix and the TSIG-permission regression it
uncovered. Adds two open TODO items found during that audit: the dyndns
app's lack of real per-user auth binding, and the fact that access
logging for every real vhost on this host is currently a no-op.
nsupdate.php/nsupdate_ipv4.php/nsupdate_ipv6.php rewritten to build the
nsupdate command as a string written to proc_open()'s stdin, using the
array form of the command so no shell is ever invoked -- no value can be
interpreted as shell syntax regardless of content. Added hostname/IP
validation as defense in depth. Originals backed up.
Also fixes a regression discovered while testing: www-data never got
read access to the TSIG key when it was tightened to 640 root:bind
(2026-07-19), so legitimate dyndns updates were silently failing
(REFUSED) independent of the vulnerability. Fixed with a POSIX ACL
scoped to just that key file rather than group membership, since the
bind group also owns rndc.key (full remote BIND control).
Verified end-to-end via the real PHP functions: injection payloads
rejected with no side effects, invalid IP rejected, legitimate update
succeeds and is visible in the zone, unauthenticated requests still 401.
nsupdate.php/nsupdate_ipv4.php/nsupdate_ipv6.php pass unsanitized $_GET
input straight into exec()'d shell commands. Gated by Apache Basic Auth
on the dyndns vhost, so not anonymously exploitable, but a real OS
command injection reachable by anyone with valid (or leaked) dyndns
credentials. Documents the finding and the fix approach; not yet applied.
A stray /var/www/html/.htaccess (dated 2025-02-18, predating everything
else in TODO.md) was silently requiring HTTP Basic Auth for jayfield.org
and www.jayfield.org's shared DocumentRoot, 401ing every real visitor to
the intentionally-public static site. Removed (backed up, not deleted).
Also adds a README table auditing which public hostnames do/don't
require credentials, prompted by checking this.
Moved Portainer's HTTPS UI behind an Apache reverse proxy on its own
subdomain (portainer.jayfield.org) instead of publishing it directly on
:9443 with a self-signed cert, which HSTS's includeSubDomains policy made
unreachable in Firefox with no click-through exception. Same pattern as
the existing web.jayfield.org/cloud.jayfield.org proxied subdomains: DNS
A record, Apache vhost proxying to 127.0.0.1:9443 over HTTPS, real
Let's Encrypt cert via certbot, standard HSTS header. Portainer container
recreated to bind 9443 to loopback only; verified end-to-end (cert chain,
HSTS header, redirect, and that :9443 is no longer reachable externally).
Firefox refuses https://alpha.jayfield.org:9443 with
MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT and no click-through: the
includeSubDomains HSTS policy on the real alpha.jayfield.org vhost pins
the hostname to trusted-cert-only HTTPS on every port, but Portainer
still serves its stock self-signed cert directly on 9443. Records the
root cause and the planned reverse-proxy fix; no server changes made yet.
Records the host's service inventory, setup runbook, mail account list,
sync plan, package diff vs. clean install, and hardening TODOs, including
today's SSH hardening (key-only, no root login) and the fail2ban sshd
findtime fix for a low-and-slow brute-force evasion pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdTyvEJkfNLDVAWt8WQ6X9
jensandClaude Sonnet 5