docs-alpha.jayfield.org/README.md

alpha.jayfield.org — Web Services Overview

Host: alpha.jayfield.org (89.58.8.149), Ubuntu 24.04 LTS (upgraded from 22.04 on 2026-07-26 — see TODO.md for the full rollout). SSH: ssh -p 10022 jens@alpha.jayfield.org

All administration in this repo's notes is performed remotely against this host over SSH (port 10022) — there is no local checkout of any service config; changes are made directly on the server and recorded here afterward.

This host runs two layers of web services: native services managed by system Apache, and containerized services managed by Docker (visible in Portainer). Apache reverse-proxies into the Docker layer where needed.

Native services (systemd / apt)

Service	Role
apache2 (2.4.58)	Terminates HTTP/HTTPS for all vhosts, reverse-proxies to backends (Docker containers, rspamd, Portainer)
bind9	Authoritative DNS for the jayfield.org zone (ns1/ns2.jayfield.org are both this host); also the box's own recursive resolver — see below
docker	Runs the containerized web services (see below)
certbot (apt, 2.9.0)	Let's Encrypt cert issuance/renewal, via certbot.timer (systemd)
Postfix (master)	SMTP, ports 25/587, on the public IP + localhost
Dovecot	IMAP/IMAPS (143/993) + ManageSieve (4190)
rspamd (4.1.3, rspamd.com PPA — not the OS's own older native package)	Spam filtering for the mail stack, localhost-only (11332–11334), proxied at mail.jayfield.org/rspamd
redis-server	Localhost-only (6379); backs rspamd's Bayes classifier, history, and reputation caching
mariadb (10.11.14, noble-native)	Holds vmail (Postfix/Dovecot virtual accounts) — see TODO.md for the 10.3→10.6→10.11 migration history

Python: python3.12.3 (noble's default). Previously python3.10.12 on 22.04, itself fixed from a stalled python3.8 leftover of the original incomplete 20.04→22.04 upgrade (breaking apt's certbot via a pytz/zoneinfo incompatibility) — that whole incomplete-upgrade saga is now fully resolved, see TODO.md.

Post-boot sanity check

/usr/local/sbin/sanity-check.sh (added 2026-07-26, after the 24.04 upgrade turned up two real post-upgrade breakages — see TODO.md) checks that the whole stack is actually healthy, not just "systemd says active": package/dpkg consistency, disk space, SSH hardening (PermitRootLogin/PasswordAuthentication), BIND + DNSSEC validation + the dyndns zone, the full mail stack including a live rspamc symbols scan (catches a phantom-active rspamd, which is exactly what happened during the 24.04 upgrade) and whether anything is actually listening on Postfix's configured milter port, MariaDB/vmail data + Postfix's SQL lookups, fail2ban's jail count and the custom findtime hardening, all 7 public vhosts by expected HTTP status code (plus a genuine Gitea API check, not just a static-page response), and all 6 Docker containers.

Runs automatically once per boot via sanity-check.service (a oneshot unit, systemctl enabled, After= all the relevant services plus a 10s settle delay before running) — covers OS upgrades, kernel updates, and any other change that ends in a reboot, which in practice has been all of them so far. Also safe to run manually after a change that doesn't reboot (a plain apt upgrade, a config edit): sudo /usr/local/sbin/sanity-check.sh. Each run logs to /var/log/sanity-check/<timestamp>.log and emails a pass/warn/fail summary to jens@jayfield.org via the local mail stack (no external dependency). Exits non-zero if anything actually failed (warnings, like routine pending package upgrades, don't fail the run).

Weekly attacker summary

/usr/local/sbin/attacker-check.py (added 2026-07-26) parses fail2ban.log (current + recent rotations, gzipped or not — reads enough history to cover a full week regardless of exactly when logrotate's daily midnight run lands relative to the cron job) for the past 7 days and turns it into two append-only CSVs for longitudinal analysis, plus a human-readable weekly summary emailed to root@jayfield.org and saved as a dated report file:

• /var/log/attacker-check/per_ip_weekly.csv — one row per (jail, IP) per week: event count, first/last seen, /24 subnet, and interval statistics (mean, stdev) between repeat hits.
• /var/log/attacker-check/per_jail_weekly.csv — weekly rollup per jail: total events, distinct IPs, distinct /24s, new bans.
• /var/log/attacker-check/reports/<week-ending-date>.txt — the full human-readable report for that run, one file per date.

root@jayfield.org didn't resolve to anything before this (not in /etc/aliases, not in the vmail aliases table) — added it as a new vmail.aliases row → jens@jayfield.org, matching the existing postmaster/hostmaster/webmaster/wlan pattern.

Two detection heuristics run on top of the raw data, both modeled directly on real campaigns found earlier this engagement (see TODO.md):

• Distributed single-shot scans: flags any /24 subnet with ≥5 distinct IPs averaging ≤3 events each against the same jail — the shape of the apache-noscript campaign found 2026-07-19.
• Regular-timing evasion: flags any IP with ≥3 repeat hits on the same jail whose inter-hit timing has a coefficient of variation under 15% — the shape of the sshd/dovecot low-and-slow campaigns found and fixed 2026-07-19/2026-07-25 (attackers spacing requests just past a jail's findtime window). A first live run against this week's real logs correctly re-identified the same dovecot campaign IPs from that earlier manual investigation, with matching ~22–28 minute intervals.

Runs weekly via /etc/cron.d/attacker-check, Sunday 06:00 (clear of logrotate's daily midnight timer). Safe to also run manually at any time: sudo /usr/local/sbin/attacker-check.py.

System users

User	uid	Home	Purpose
jens	1001	/home/jens	Primary sudo admin account — all administration in this repo is done as this user over SSH
alex	1005	/home/alex	Secondary, non-sudo personal account (added 2026-07-19)

Both also exist as separate Nextcloud accounts (see "Password vault sync" below) and as chrooted, SFTP-only accounts ftp_jens/ftp_alex under /var/sftp/ — the latter currently password-locked, not deleted (see below).

DNS resolver architecture

Fixed 2026-07-19: the box's own outbound DNS resolution (used by Postfix's postscreen for DNSBL lookups, and by every other service) previously forwarded through the hosting provider's shared resolvers (46.38.225.230/46.38.252.230). Confirmed via a live test: GMX's own mail servers (212.227.15.15/.19, verified via reverse DNS as mout.gmx.net) were being dropped by postscreen on a false positive — Spamhaus's DNSBL was returning 127.255.255.254, its documented rate-limit sentinel ("this querying resolver has exceeded our free-tier volume"), not a real listing. The shared resolver was almost certainly pooling query volume from other tenants of the same hosting provider, pushing it over Spamhaus's threshold.

Fix:

• Removed the forwarders {} block from named.conf.options — bind9 now does full iterative/recursive resolution itself (root hints → TLD → authoritative) for trusted clients (localhost + localnets) instead of relaying to the shared upstream.
• Repointed /etc/systemd/resolved.conf's DNS= from the shared provider IPs to 127.0.0.1 (i.e. bind9), then systemctl restart systemd-resolved. The 127.0.0.53 stub listener and /etc/resolv.conf symlink were left untouched, so Docker's DNS-detection logic (which special-cases the stub) is unaffected.
• (An unbound install was tried first as a separate local resolver, but bind9 already listens on 127.0.0.1:53/[::1]:53 — listen-on-v6 { any; } plus BIND's default of listening on all IPv4 interfaces — so they couldn't coexist on the same port. Reusing bind9 for both roles was simpler than fighting that conflict.)

Verified: both GMX IPs now correctly resolve as not listed; postscreen now PASSes them. A real end-to-end test email ("Claude email test #01") delivered successfully to both jens@ and otto2022@jayfield.org after the fix.

Known caveat: direct recursive resolution occasionally hits a single slow/unresponsive server somewhere in a delegation chain, where a well-provisioned forwarding resolver's warm cache usually wouldn't. Seen once during testing — a transient PTR lookup timeout caused one retry to soft-fail (450, not 550) before a second retry succeeded seconds later. Not expected to be a recurring problem, but worth knowing if a one-off 450 4.7.25 Client host rejected shows up in mail.log.

Apache virtual hosts

ServerName	:80	:443	Notes
dyndns.jayfield.org	→ redirect to :443	Basic Auth (Restricted Content), DocumentRoot /var/www/dyndns (PHP)	Also the effective default vhost for unmatched hostnames (e.g. bare alpha.jayfield.org falls through here)
jayfield.org	→ redirect to :443	Static site (/var/www/html)	Previously had no :80 vhost of its own — fell through to the dyndns default and redirected to the wrong domain; fixed
www.jayfield.org	→ redirect to :443	Static site (/var/www/html)	Renewal fixed to use the apache certbot plugin (was standalone, which required stopping Apache)
mail.jayfield.org	→ redirect to :443	Webmail-adjacent, proxies /rspamd → http://localhost:11334/
web.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker web (nginx) container on 127.0.0.1:8081	New — see below
cloud.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker nextcloud container on 127.0.0.1:8082, sets X-Forwarded-Proto: https	Nextcloud — see below
portainer.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker portainer container on 127.0.0.1:9443 over HTTPS (SSLProxyEngine/SSLProxyVerify none for the loopback hop to Portainer's self-signed cert)	Added 2026-07-20 — see "Portainer" below
git.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker gitea container on 127.0.0.1:8083, sets X-Forwarded-Proto: https	Added 2026-07-26 — migrated from vlda-01, see "Gitea" below
default (_default_:443, 000-default on :80)	Apache's stock placeholder page	—

All Let's Encrypt certs (9 total, one per hostname/group) auto-renew via certbot.timer; verified with certbot renew --dry-run.

All 8 domain :443 blocks send Strict-Transport-Security: max-age=63072000; includeSubDomains (mod_headers). Deliberately skipped on the default-ssl.conf catch-all — it's the self-signed fallback for unmatched SNI/Host headers, not a real trusted hostname a browser could pin.

Docker services (visible in Portainer)

Container	Image	Ports	Purpose
portainer	portainer/portainer-ce:latest	8000 (edge-agent tunnel, published), 9443 (HTTPS UI, 127.0.0.1-only), 9000 (UI, internal only)	Docker management UI — https://portainer.jayfield.org, reverse-proxied by Apache to 127.0.0.1:9443 (fixed 2026-07-20, see below)
web	nginx:latest	127.0.0.1:8081 (localhost-only, not published externally)	Backing service for web.jayfield.org; only reachable through the Apache reverse proxy
nextcloud	nextcloud:apache	127.0.0.1:8082 (localhost-only)	Nextcloud app (PHP+Apache bundled in the image); backing service for cloud.jayfield.org
nextcloud-db	mariadb:lts	internal only (nextcloud_net)	Dedicated DB for Nextcloud — kept separate from the host's own MySQL (which only holds vmail)
nextcloud-redis	redis:alpine	internal only (nextcloud_net)	File locking + distributed cache + PHP session storage for Nextcloud. Dedicated container rather than reusing the host's native redis-server, since that instance is bound to 127.0.0.1/::1 only with no auth — reusing it would have meant loosening a shared service's network exposure just to save one small container
gitea	gitea/gitea:1.27.0	127.0.0.1:8083 (localhost-only)	Git hosting, migrated from vlda-01 2026-07-26 — see "Gitea" below

Data persists in the portainer_data named volume, independent of the container itself — updating Portainer means recreating the container against a freshly pulled image while keeping this volume and the /var/run/docker.sock bind mount; the old container is kept renamed (not deleted) until the new one is verified healthy.

Two early-test containers (my-apache-app, a stopped httpd, and clever_jepsen, a stale hello-world container from initial Docker setup) have been removed as cleanup.

web.jayfield.org request flow

Internet → Apache :80/:443 (web.jayfield.org vhost)
             → Let's Encrypt TLS termination
             → ProxyPass http://127.0.0.1:8081/
                 → Docker container "web" (nginx) on the docker0 bridge

The container is intentionally bound to 127.0.0.1 only, so it cannot be reached directly on port 8081 from the internet — Apache is the only path in. This is the pattern to follow for any future container-backed site: publish the container to 127.0.0.1:<port>, add an Apache vhost for the subdomain that proxies to it, then run certbot --apache -d <subdomain> to get HTTPS.

Portainer (portainer.jayfield.org)

Internet → Apache :80/:443 (portainer.jayfield.org vhost)
             → Let's Encrypt TLS termination
             → ProxyPass https://127.0.0.1:9443/ (SSLProxyVerify none —
                 backend keeps Portainer's own self-signed cert, but that
                 hop never leaves the host)
                 → Docker container "portainer" HTTPS UI, published to
                   127.0.0.1 only

Previously Portainer published 9443 directly to the public interface with its own self-signed cert; fixed 2026-07-20 after that became unreachable in HSTS-enforcing browsers (see TODO.md). The container was recreated with -p 127.0.0.1:9443:9443 (old container kept renamed until the new one was verified healthy, then removed — same caution as the update procedure below). The 8000 edge-agent tunnel port is still published directly; nothing currently depends on it, but it wasn't in scope for this fix.

Nextcloud (cloud.jayfield.org)

Minimal, data-hosting-focused deployment — core Files app only. No Collabora/OnlyOffice, Talk, full-text search, external storage backends, or LDAP/SSO; those can be layered on later without re-architecting.

Internet → Apache :80/:443 (cloud.jayfield.org vhost)
             → Let's Encrypt TLS termination, X-Forwarded-Proto: https
             → ProxyPass http://127.0.0.1:8082/
                 → container "nextcloud" (nextcloud:apache)
                     ↕ private network "nextcloud_net"
                 → container "nextcloud-db" (mariadb:lts)
                 → container "nextcloud-redis" (redis:alpine)

• DNS: cloud A record added to /etc/bind/db.jayfield.org (serial bumped 2022032213 → 2022032214).
• Data volume: bind-mounted at /srv/nextcloud/data on the host (owned by uid/gid 33, matching www-data both on the host and inside the container) rather than an opaque named Docker volume — so it's directly inspectable/backupable with normal tools (du, rsync, borg).
• App volume: named Docker volume nextcloud_html, holds Nextcloud core + config.php; survives image upgrades.
• DB volume: named Docker volume nextcloud_db.
• Auto-install: done via the image's env-var-driven installer (MYSQL_*, NEXTCLOUD_ADMIN_USER/NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_TRUSTED_DOMAINS, TRUSTED_PROXIES, OVERWRITEPROTOCOL=https, REDIS_HOST) rather than leaving an unconfigured setup wizard reachable on the public internet.
• Credentials: DB root/user passwords and the initial admin password are in /root/.nextcloud-credentials on the server (600 root:root) — not committed anywhere, not in this repo.
• Cron: Nextcloud's background-job mode is set to cron (occ background:cron); a root crontab entry runs docker exec -u www-data nextcloud php -f /var/www/html/cron.php every 5 minutes. No sidecar cron container.
• PHP limits: image defaults are already generous for file hosting (512M memory_limit/upload_max_filesize/post_max_size, unlimited max_execution_time) — no tuning needed for this pass.

Gitea (git.jayfield.org)

Migrated 2026-07-26 from vlda-01 (home intranet server) per SYNC-PLAN.md's build order — first item on that plan, "no open questions." Full migration, not a sync: vlda-01's Gitea container was stopped, its entire /data volume (SQLite DB, all repos, LFS objects, host SSH keys) copied straight across via a piped ssh ... tar | ssh ... tar (no intermediate disk, ~4GB), and brought up on alpha as a permanent container — same web/cloud/portainer pattern (publish to 127.0.0.1:<port>, Apache vhost, certbot --apache).

Internet → Apache :80/:443 (git.jayfield.org vhost)
             → Let's Encrypt TLS termination, X-Forwarded-Proto: https
             → ProxyPass http://127.0.0.1:8083/
                 → container "gitea" (gitea/gitea:1.27.0)
                     → SQLite DB + repos, bind-mounted at /srv/gitea/data

• Version pinned, not :latest: gitea/gitea:1.27.0, matching what was actually running on vlda-01 — a version-compatible move first, upgrading is a separate later decision.
• Data: bind-mounted at /srv/gitea/data on the host (matching the /srv/nextcloud/data convention — directly inspectable/backupable), not an opaque named volume.
• Ownership: the source data was owned by uid/gid 1000 on vlda-01, but 1000 is already vmail on alpha (Dovecot's mailbox user) — remapped everything to a dedicated 1010:1010 (chown -R, container run with -e USER_UID=1010 -e USER_GID=1010) to avoid any collision with or confusion around the real vmail account.
• SQLite, not the host's MariaDB: vlda-01's Gitea was already self-contained SQLite (DB_TYPE = sqlite3), so no DB export/import or extra container was needed — the DB file just came along inside /data.
• app.ini updated for the new home: DOMAIN/SSH_DOMAIN/ROOT_URL repointed from the old LAN IP (192.168.22.90) to git.jayfield.org; DISABLE_SSH set to true — the existing repo remotes already used HTTP(S), not git@.../ssh://, so git-over-SSH was dropped rather than opening yet another public port to preserve a mode nothing was actually using. HTTPS clone/push covers the same ground through the same Apache TLS termination every other service uses.
• Reverse-proxy scheme awareness: without X-Forwarded-Proto: https (added to the vhost, same as cloud.jayfield.org), Gitea's API correctly used ROOT_URL for some fields but generated http:// (not https://) clone_url/html_url for repos — since the Apache→container hop itself is plain HTTP and Gitea otherwise has no way to know the original request arrived over TLS. Fixed and verified via the API.
• Verified end-to-end: all previously-existing repos present via the API (including this docs-alpha.jayfield.org repo), admin user (jayfield) intact; a real git clone over HTTPS reproduced identical commit history to the working copy; this repo's own origin remote repointed to https://git.jayfield.org/... and a real git push succeeded against it.
• vlda-01's copy: left stopped (container not removed, data not deleted) as a cold backup — reversible if ever needed; not decommissioned outright.

Password vault sync (KeePass, via Nextcloud)

Each user's KeePass database now syncs through Nextcloud's own WebDAV endpoint instead of a bespoke Apache-WebDAV + SFTP-upload pipeline:

User	Nextcloud account	Vault file	WebDAV URL
jens	jens	sicher_jens.kdbx	https://cloud.jayfield.org/remote.php/dav/files/jens/sicher_jens.kdbx
alex	alex	sicher_alex.kdbx	https://cloud.jayfield.org/remote.php/dav/files/alex/sicher_alex.kdbx

• Each .kdbx lives at the root of that user's Nextcloud Files storage (/srv/nextcloud/data/<user>/files/, 640 www-data:www-data), picked up via occ files:scan <user>.
• Verified 2026-07-19 with a raw WebDAV PROPFIND/GET/PUT/GET round trip against the alex account — correct Content-Length, matching SHA-256 before and after a write. (KeePassXC has no native WebDAV client and there's no display on this host to drive a real KeePass session, so this HTTP-level check is the practical equivalent — it exercises the same verbs a WebDAV-aware sync client or plugin would use.)
• A 600-permission backup copy of each vault also still lives in the respective user's home dir (/home/jens/sicher_jens.kdbx, /home/alex/sicher_alex.kdbx) on this host.
• Superseded setup: an Apache mod_dav_fs container (docker-apache2-php / my-httpd image, HTTP Basic Auth via .htpasswd) running on the admin workstation used to serve the vault directly, with files uploaded to this host via the chrooted ftp_jens/ftp_alex SFTP accounts under /var/sftp/. That local container/image has been removed (it ran with --rm and had no cron/systemd autostart, so nothing else depended on it). The ftp_jens/ftp_alex accounts remain on this host (chroot and files untouched) but are password-locked (passwd -l) since nothing uses them anymore — reversible with passwd -u if SFTP is ever needed again.

Public access / authentication overview

Audited 2026-07-20 — which public hostnames require credentials:

Host	Requires credentials?	Notes
jayfield.org / www.jayfield.org	No (intentionally public static site)	Was accidentally 401'ing every visitor via a stray .htaccess — fixed, see TODO.md
dyndns.jayfield.org	Yes — HTTP Basic Auth, one dedicated account per hostname	Vhost-level <Directory> block, intentional. Had an OS command injection in its update scripts (fixed 2026-07-20) and no real per-user auth binding (fixed 2026-07-26 with per-host TSIG keys — see TODO.md)
mail.jayfield.org	N/A — root is 403 (no content served there)
mail.jayfield.org/rspamd	Yes — rspamd's own login page
cloud.jayfield.org (Nextcloud)	Yes — redirects to /login
portainer.jayfield.org	Yes — admin account already initialized (/api/users/admin/check → 204, no open setup wizard)
web.jayfield.org	No	Currently just the stock nginx welcome page (no real content deployed yet), so nothing sensitive is exposed — but worth revisiting before real content goes there if it's meant to stay public

Open items / known quirks

• Intranet migration from vlda-01 (files, Gitea, Jellyfin, Dogecoin node/wallet) — planning stage only, not started. See SYNC-PLAN.md for the full per-service design, open questions, and build order.

• PTR/EHLO mismatch (workaround applied, real fix still open): the PTR record for this host's IP (89.58.8.149) resolves to alpha.jayfield.org, but Postfix's myhostname was mail.jayfield.org (matching the MX record) — a mismatch some strict receivers penalize. 2026-07-19: myhostname changed to alpha.jayfield.org to match the existing PTR (verified live: EHLO → 250-alpha.jayfield.org); see SETUP.md's mail-stack section. The real fix is repointing the PTR to mail.jayfield.org and then switching myhostname back — not done yet, since PTR isn't controlled from this host's own DNS zone. Checked 2026-07-19: the ISP's control panel has no self-service PTR/rDNS option at all, so this needs a support ticket to the ISP asking them to set the PTR for 89.58.8.149 to mail.jayfield.org (worth noting in the ticket that the forward record already resolves correctly — mail.jayfield.org → 89.58.8.149 — since some providers require that before they'll set rDNS). If the ISP can't or won't do it, the current myhostname = alpha.jayfield.org workaround is a fine permanent setup as-is, not just a stopgap — PTR and EHLO already agree, which is what actually matters for deliverability.

• Distributed low-and-slow scan on apache-noscript: found 2026-07-19 while auditing fail2ban jails for the sshd timing-evasion pattern below — a cluster of ~20 IPs across three /24s (91.230.225.0/24, 185.192.71.0/24, 185.92.25.0/24) each probed exactly once, never repeating. No per-IP findtime/maxretry tuning can catch this by design — it needs subnet-level (CIDR) banning instead, which risks blocking legitimate traffic sharing that block (e.g. carrier NAT). Left as routine scan noise for now; revisit if it escalates. Follow-up 2026-07-26: rechecked — the original three /24s are gone from current logs entirely (that specific campaign appears to have moved on). apache-noscript shows normal jail activity (42 total bans, 3 currently banned) from ordinary repeat-offender IPs already caught by per-IP thresholds, not a new distributed single-shot pattern. No escalation; still fine to leave as-is.

• No real nameserver redundancy: 2026-07-19, ns1.jayfield.org and ns2.jayfield.org were registered as this domain's NS records at the registrar. Delegation checked and confirmed consistent — the .org registry (checked against two separate registry servers), this host's own authoritative zone, and both ns1/ns2 themselves (queried directly, aa flag set, matching SOA serial 2022032214 at the time of that check — serial has since moved to 2022032216 from the SPF/DMARC/DKIM fix below, unrelated to NS/glue) all agree. But both names glue to the same IP (89.58.8.149, i.e. this same host) — RFC 2182 §3 recommends at least two nameservers not on the same network segment, and this setup has zero real redundancy: if alpha goes down, both "nameservers" disappear together. Tried unregistering ns2 at the registrar since it wasn't adding real redundancy anyway — confirmed the ISP/registrar requires a minimum of two NS records at the delegation layer and rejected the change, so both stay registered as-is. Redundancy risk remains open; the only real fix is a second nameserver on genuinely separate infrastructure.

(Previously resolved: legacy /portainer proxy paths on the default vhost and jayfield.org, stale DNS records test, kacke, pupsi, mausi, hase, super, and a disabled nextcloud.conf vhost plus its a2dissite marker file — a leftover from a Nextcloud install that was disabled at some point but never actually deployed (no /var/www/nextcloud, no package, no DB, no cert) — all cleaned up. Also: mnt-vlda-01.mount's recurring boot failure — root-caused to a dormant WireGuard tunnel (alpha_at_moon, routing the same 192.168.22.0/24 home-LAN range the CIFS target lived on) — resolved 2026-07-19 by removing all three together rather than reviving the tunnel: wireguard/wireguard-tools purged, nginx + libnginx-mod-rtmp purged (an unrelated, previously-uncatalogued RTMP streaming service on :1935, found via the same diff pass, also unused and removed), and samba/cifs-utils purged along with the fstab entry and stored CIFS credentials. Configs backed up to /root/removed-configs-backup/ on the server before deletion, in case any of the three are wanted again. Zero failed systemd units after. Also: KeePass vault sync migrated 2026-07-19 from local Apache-WebDAV + SFTP-upload to Nextcloud's built-in WebDAV — see "Password vault sync" above. Also: SPF, DMARC, and both DKIM (2021/2022._domainkey) TXT records in db.jayfield.org were silently broken — root cause was unquoted TXT rdata in the zone file: BIND's zone parser treats an unquoted ; as a start-of-comment (so v=DMARC1; p=reject; was being served as just "v=DMARC1", and both DKIM keys as just "v=DKIM1" with the k=rsa; p=<key> portion silently dropped — meaning DKIM signature verification had been failing for all outbound mail), and an unquoted multi-token SPF line was being split into multiple concatenated character-strings instead of one, merging in an unrelated google-site-verification= token and invalidating the SPF syntax. Fixed 2026-07-19 by quoting every TXT value (splitting the long DKIM keys across two <=255-byte quoted strings each, since a single character-string is capped at 255 bytes — see named-checkzone's "text string too long" if this is missed) and separating google-site-verification into its own TXT record. SOA serial bumped 2022032214 → 2022032216. Verified directly against the authoritative server (dig ... @89.58.8.149) post-reload; public resolvers catch up as the old records' TTLs (up to 3600s) expire. Also 2026-07-19: audited for phpMyAdmin/Adminer/webmin/cPanel/Plesk/ ISPConfig/phpPgAdmin residue (dpkg logs, rc-state packages, filesystem-wide filename search, Apache configs, MySQL users/DBs, crontabs, Docker images/volumes) — clean; the only name hits were stock fail2ban/augeas package files, not evidence of a prior install. Two stray public files (ShowKeyPlus_x64.zip, MediaCreationTool1909.exe — unrelated pirated Windows tooling, plus an empty test file) were found sitting in /var/www/html/pub, publicly served off the default vhost; moved to /home/jens/pub and the now-empty pub/ directory removed from the web root. Also: SSH hardened (PermitRootLogin no, PasswordAuthentication no, key-only) and the sshd fail2ban jail's findtime widened from the 30-minute default to 24h (maxretry 3→4, bantime 4h→24h) after 47.76.192.176 was found grinding root/jayfield SSH logins roughly every 33 minutes for most of a day — just outside the old 30-minute window, so it never accumulated enough strikes to get banned; see TODO.md and SETUP.md §1/§3 for the fix and the same evasion check applied to every other jail (only one other pattern turned up — see "Open items" above). Also: Portainer's :9443 UI, previously published directly to the public interface on its own self-signed cert, became unreachable in HSTS-enforcing browsers (MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT, no click-through) once the includeSubDomains HSTS policy above started pinning alpha.jayfield.org to trusted-cert-only HTTPS on every port — not a MITM, a self-inflicted interaction between two otherwise-correct pieces of config. Fixed 2026-07-20 by moving it behind an Apache reverse proxy on its own subdomain, portainer.jayfield.org, same pattern as web.jayfield.org/cloud.jayfield.org; see "Portainer" above and TODO.md for the full details. Also: an audit of which public hostnames require credentials (prompted by the Portainer/HSTS work above) turned up an OS command injection in the dyndns update scripts (/var/www/dyndns/nsupdate*.php built a shell command by directly interpolating unsanitized $_GET input into exec()) — gated by the vhost's Basic Auth, so not anonymously exploitable, but a real privilege escalation for anyone with valid dyndns credentials. Fixed 2026-07-20 by rewriting the update path to use proc_open() with array-form arguments (no shell involved at all) plus strict hostname/IP validation. That same audit also surfaced (a) a regression from the 2026-07-19 TSIG key hardening that had silently broken legitimate dyndns updates — fixed with a POSIX ACL scoped to just that key file, not group membership, since the bind group also owns rndc.key — and (b) that access logging for every real vhost on this host is currently a no-op, and (c) that the dyndns app has no real per-user auth binding. (b) fixed 2026-07-25 — see TODO.md for the root cause and the per-vhost CustomLog fix. (c) fixed 2026-07-26: every dyndns hostname now has its own dedicated .htpasswd account and TSIG key, with BIND's update-policy (not just an app-level check) refusing any update where the authenticated user doesn't match the requested hostname — see TODO.md for the full rollout, including the now-retired shared key and legacy fallback.