docs-alpha.jayfield.org/TODO.md

alpha.jayfield.org — TODO

Outstanding hardening items (see README.md for the full service overview). Nothing here is urgent; all are low-risk, no-downtime changes.

• Mail server IP (89.58.8.149) listed on Barracuda's BRBL — found 2026-07-25 (user report: "jayfield.org is on the Barracuda MX blacklist"). Confirmed via dig +short 149.8.58.89.b.barracudacentral.org A → 127.0.0.2 (listed); clean on Spamhaus ZEN, SpamCop, and SORBS — Barracuda-specific. Checked the server for actual abuse before requesting removal: mail queue empty, outbound volume ~11 msgs over the prior week, only known SASL logins (jens@/otto2022@jayfield.org from the usual home IP), no backscatter, correct PTR (alpha.jayfield.org), SPF/DKIM/DMARC (p=reject) all already live — no sign of a compromised account or open relay, so most likely a stale/inherited reputation hit on the IP rather than active abuse from this host. Submitted a removal request via https://www.barracudacentral.org/rbl/removal-request on 2026-07-25. Confirmation number: BBR21785009436-19117-9182. Per Barracuda's response, the IP gets a temporary 48h reputation bump to "normal" while they investigate (up to ~1h to propagate), after which it either stays clear or reverts to listed. Status 2026-07-26: still clear (dig +short 149.8.58.89.b.barracudacentral.org A → NXDOMAIN, no 127.0.0.2) — matches the 1h check (routine trig_01F1RMhEtNHtAbuHb9ffe1oX, fired 2026-07-25 21:02 UTC, also clear). This is expected during the 48h "normal" bump window, not yet proof of permanent delisting. The real test is the 48h check (routine trig_019GayXSo5TW5eSxiJ99Ghrp, scheduled 2026-07-27 20:03 UTC) — if it reverts to listed, next step is a manual follow-up to Barracuda citing the confirmation number above.

• dyndns app has no real per-user auth binding — found 2026-07-20, fixed 2026-07-26. index.php's pass == '_NO_PASS_' check is a hardcoded magic string, not a real per-user credential — nothing tied the single shared Apache Basic Auth identity (jens, the only .htpasswd account) to which hostname label got updated. Confirmed live in the access log: a FritzBox ("Fritz!Box DDNS/1.0.3") was authenticating as jens while updating the uschi record — i.e. every device was already relying on this exact gap, self-policing which host it claimed via an unauthenticated hostname=/user= param. Fixed with real per-host TSIG binding, not just an app-level check: - Added a dedicated .htpasswd account for each of kack, test, uschi, vpn (jens's existing account/password untouched). - Generated 5 per-host TSIG keys (hmac-sha512, tsig-keygen), one per hostname, each in its own /etc/bind/<host>.dyndns.jayfield.org.key (root:bind 640 + setfacl grant for www-data, same pattern as the original key). - Replaced the zone's flat allow-update { key dyndns.jayfield.org; }; in named.conf.default-zones with an update-policy granting each new key permission to update only its own name (grant "<host>" name <host>. A AAAA TXT;), plus one explicit legacy fallback grant for the original shared key (subdomain rule, zone-wide) — kept temporarily so unmigrated devices don't break (see below). - nsupdate.php/nsupdate_ipv4.php/nsupdate_ipv6.php now sign with the key belonging to $_SERVER['PHP_AUTH_USER'] (the authenticated identity), not whatever hostname was requested. If they match, the new per-host key is used — BIND itself then refuses the update if that key isn't granted that name, independent of any PHP-side check. If the authenticated user is specifically the legacy jens account and doesn't match, it falls back to the old shared key (logged via syslog(LOG_WARNING, ...) for migration tracking) — every other account gets rejected outright (-1) on a mismatch, so newly-created per-host logins can never touch another host, even via the fallback. - Also fixed a pre-existing bug found in nic/update.php while testing this: it printed good <ip> unconditionally, without ever checking nsupdate's return code — so a rejected update silently looked successful to the calling device. Now prints the real dyndns2-protocol response (good <ip> / badauth). All files backed up to /root/removed-configs-backup/dyndns-per-host-auth-20260726094500/. Validated with named-checkconf/apache2ctl configtest, applied via rndc reload/systemctl reload apache2. Tested both by driving nsupdate() directly as www-data and end-to-end over HTTPS with the new vpn credentials: own-host updates succeed, cross-host attempts from new accounts are rejected (-1/badauth) with the target record provably unchanged, and the legacy jens→uschi pattern still succeeds unchanged. Test IPs used during verification were restored to the real production values (test, uschi, vpn) immediately after; live dig confirmed the final state matches the pre-change records exactly. Follow-up: the jens legacy fallback (and its .htpasswd account) should stay only until every real device — including whichever ones are still authenticating as jens to update kack/ test/uschi — has been reconfigured with its own dedicated login. Once dyndns legacy shared-key update stops appearing in syslog, remove the fallback grant and the jens account can be scoped down to just its own host too. vpn migrated 2026-07-26: home FritzBox reconfigured with the new vpn login and confirmed live — after a WAN IP change (real connection recycle, not a test), the access log shows a genuine Fritz!Box DDNS/1.0.3 request authenticated as vpn updating vpn.dyndns.jayfield.org to the new IP (217.229.60.187), and dig confirms the zone picked it up. First host fully off the legacy fallback. kack/test/uschi still need migrating.

• dyndns/default-vhost access logging is a no-op — done 2026-07-25 found 2026-07-20 while investigating the command injection below. dyndns.jayfield.org, jayfield.org, www.jayfield.org, mail.jayfield.org, web.jayfield.org, and cloud.jayfield.org had no CustomLog directive of their own, and access.log (only written by default-ssl.conf/000-default.conf) had 0-8 lines — looking like no real vhost was being logged at all. Root cause (2026-07-25 re-check): the fallback assumption was wrong. There's no global CustomLog access.log in apache2.conf; instead conf-enabled/other-vhosts-access-log.conf enables mod_vhost_combined, which was already writing every vhost-without-its-own-CustomLog's traffic to /var/log/apache2/other_vhosts_access.log (5,432 lines, confirmed real traffic incl. bot scans against cloud.jayfield.org). So requests were never actually un-logged — just commingled in one shared, harder-to-grep file instead of access.log. Fixed anyway, since a dedicated per-site log remains the better end state for incident review: added an explicit CustomLog ${APACHE_LOG_DIR}/<site>_access.log combined to every <VirtualHost> block for dyndns, jayfield.org, www, mail, web, cloud, and portainer (the :80 and :443/le-ssl blocks of each site share one log file). Configs backed up to /root/removed-configs-backup/vhost-customlog-20260725203959/. /etc/logrotate.d/apache2 already globs *.log, so no logrotate change needed. Validated with apache2ctl configtest, applied via systemctl reload apache2, verified live with curl against all 7 domains and confirmed each new <site>_access.log recorded the hit.

• OS command injection in the dyndns update scripts — done 2026-07-20 /var/www/dyndns/nsupdate.php, nsupdate_ipv4.php, and nsupdate_ipv6.php (used by both index.php and nic/update.php, the router/ddclient-facing endpoint) built a shell command string by directly interpolating unsanitized $_GET input (hostname/myip/ ip, which become $host/$ip/$ipv4/$ipv6) and passed it to PHP's exec(), which runs it via /bin/sh -c — no escapeshellarg()/escapeshellcmd() anywhere. A request like hostname=$(id>/tmp/pwned).dyndns.jayfield.org would have executed arbitrary shell commands as www-data (the domain-suffix check still passed after the .-split). nic/update.php had no application-level auth of its own at all. Mitigating factor: the whole dyndns.jayfield.org vhost sits behind Apache Basic Auth, so this needed valid (or leaked/brute-forced) dyndns credentials to reach — not exploitable by an anonymous visitor. Compounding factors found during the audit: PHP is 7.4.33, EOL since Nov 2022, disable_functions was empty (exec fully available); index.php's pass == '_NO_PASS_' check is a hardcoded magic string, not real per-user auth, so any valid dyndns credential can update any of the zone's 5 hostnames (jens, kack, test, uschi, vpn), not just its own — left as-is, out of scope for this fix; and the vhost's access logging turned out to be a no-op (shared access.log has 0 lines), so past exploitation couldn't be fully ruled out from logs alone — though file mtimes, www-data processes/crontab, and zone contents all looked unremarkable, consistent with unexploited. Fixed: all three files backed up to /root/removed-configs-backup/dyndns-preinjectionfix-<timestamp>/, then rewritten to build the nsupdate command list as a plain string written to proc_open()'s stdin pipe using the array form of the command (array("/usr/bin/nsupdate", "-k", ...)) — this never invokes a shell at all, so no value can be interpreted as shell syntax regardless of content. Added strict preg_match validation requiring $host/$domain to be shaped like a valid hostname (letters/digits/hyphens, dot-separated labels) and filter_var(..., FILTER_VALIDATE_IP, FILTER_FLAG_IPV4/IPV6) on the IP before use, as defense in depth on top of removing the shell. Regression found and fixed along the way: functional testing showed legitimate updates were already silently failing (update failed: REFUSED, permission denied reading the TSIG key) — turned out www-data was never granted read access to /etc/bind/dyndns.jayfield.org.key when its permissions were tightened to 640 root:bind in the item below, so the dyndns web update feature had been broken for real users since 2026-07-19, independent of this vulnerability. Fixed with a narrowly-scoped POSIX ACL (setfacl -m u:www-data:r /etc/bind/dyndns.jayfield.org.key, acl package installed) rather than adding www-data to the bind group — the group also owns rndc.key (full remote BIND control), which would have been a much bigger privilege grant than dyndns needs. Verified via sudo -u www-data php driving the real functions directly (same code path as the web UI, without needing the Basic Auth password): two injection payloads ($(...) and ;-chained) both rejected (-1, no /tmp side-effect file created), an invalid IP rejected (-1), and a legitimate update succeeded (0), confirmed live in the zone via dig, then cleaned up (test record deleted via nsupdate, temp test script removed). Also confirmed unauthenticated requests still get 401 from Apache before ever reaching the PHP, and apache2ctl configtest stayed clean throughout.

• Public static site (jayfield.org/www.jayfield.org) was 401'ing every visitor — done 2026-07-20 Found while auditing which services are reachable without credentials: a stray /var/www/html/.htaccess (AuthType Basic, same AuthName "Restricted Content"/AuthUserFile /etc/apache2/.htpasswd as the intentional dyndns.jayfield.org protection) was silently requiring login for the DocumentRoot both jayfield.org and www.jayfield.org share — /var/www has AllowOverride all, so it took effect with no vhost-level directive needed. File was dated 2025-02-18 (index.html/apache.html themselves are from 2022) — predates every other change in this doc by well over a year, almost certainly a forgotten leftover rather than intentional, and contradicts README.md's own description of both as a public static site. Fixed: moved to /root/removed-configs-backup/htaccess-var-www-html.<timestamp> (recoverable, not deleted — same convention as this file's other removed-config backups). No Apache reload needed (.htaccess is read per-request). Verified both jayfield.org and www.jayfield.org now return 200 with real page content; dyndns.jayfield.org's own (intentional, vhost-level) Basic Auth is untouched and still returns 401 without credentials.

• Lock down the dyndns TSIG key file permissions — done 2026-07-19 /etc/bind/dyndns.jayfield.org.key was 644 (world-readable), holding the shared secret that authenticates dynamic DNS updates for dyndns.jayfield.org with no IP-based allow-update backing it up. Fixed: chmod 640 + chown root:bind (owner was already correct), matching rndc.key's permissions.

• Suppress BIND version disclosure — done 2026-07-19 No version statement in named.conf.options, so dig CH TXT version.bind @ns1.jayfield.org leaked the exact BIND version (9.18.39) — a minor fingerprinting aid for attackers. Fixed: added version "unknown"; inside the options {} block, validated with named-checkconf, and applied via rndc reload. Verified dig CH TXT version.bind now returns "unknown".

• Add response rate limiting (RRL) — done 2026-07-19 No rate-limit {} block was configured. Since allow-query { any; } is required for a public authoritative zone, RRL is the standard mitigation against the server being used for spoofed-source DNS reflection/amplification against third parties. Fixed: added rate-limit { responses-per-second 10; window 5; } to named.conf.options, validated with named-checkconf, and applied via rndc reload. Started with conservative defaults (10/s, 5s window) — revisit if legitimate high-volume resolvers get throttled.

• Add HSTS headers to all vhosts — done 2026-07-19 No vhost sent a Strict-Transport-Security header, leaving a window for on-path SSL-stripping on a user's very first plaintext request to any domain. Fixed: added Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains" to all 6 domain :443 blocks (dyndns, jayfield.org, www, mail, web, cloud) — skipped default-ssl.conf since it's the self-signed catch-all with no real trusted hostname for a browser to pin. Validated with apache2ctl configtest, applied via systemctl reload apache2, verified live with curl -sI against all 6 domains.

• jayfield.org had no :80 vhost of its own — done 2026-07-19 Bare http://jayfield.org fell through to the dyndns default vhost and redirected to the wrong domain. Fixed: added a :80 block to jayfield.org.conf redirecting to https://jayfield.org/, matching the dyndns/mail pattern. Verified: curl -sI http://jayfield.org/ → 302 with Location: https://jayfield.org/.

• SSH: disable root login + password auth — done 2026-07-19 PermitRootLogin yes and PasswordAuthentication yes were both still enabled (the "known deviation" flagged in SETUP.md §1) — every credential-stuffing bot on the internet got a real login prompt to grind against. Fixed: PermitRootLogin no, PasswordAuthentication no, PubkeyAuthentication yes in /etc/ssh/sshd_config (backed up as sshd_config.bak.20260719225249). The Match Group sftp block's own PasswordAuthentication yes was left as-is — separate, already-chrooted, no-shell/no-forwarding accounts, not part of this risk. Validated with sshd -t, applied via systemctl reload sshd, verified live with sshd -T and a fresh key-only connection before considering it done.

• Portainer's :9443 UI is unreachable in HSTS-enforcing browsers — done 2026-07-20 https://alpha.jayfield.org:9443 threw MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT in Firefox with no click-through option ("keine Ausnahme kann hinzugefügt werden"). Root cause: the includeSubDomains HSTS policy sent by the real alpha.jayfield.org vhost (Strict-Transport-Security header, added in the item above) pins the hostname to HTTPS-with-a-trusted-cert on every port, but Portainer (README.md §Docker services) was still serving its stock self-signed cert directly on 9443. Not a MITM — self-inflicted by combining HSTS with an unproxied self-signed service. Fixed: added a portainer A record to db.jayfield.org (SOA serial 2022032216 → 2022032217); recreated the portainer container binding the HTTPS UI to 127.0.0.1:9443:9443 instead of publishing it (old container kept renamed until the new one was verified, then removed, matching the update procedure documented in README.md); added an Apache vhost for portainer.jayfield.org (ProxyPass/ProxyPassReverse to https://127.0.0.1:9443/, SSLProxyEngine on, SSLProxyVerify none + SSLProxyCheckPeerCN off/ SSLProxyCheckPeerName off for the loopback hop to Portainer's self-signed cert) — same pattern as web.jayfield.org/ cloud.jayfield.org (SETUP.md §4/§5); issued a real cert via certbot --apache -d portainer.jayfield.org --redirect; added the standard HSTS header. Left the 8000 edge-agent tunnel port published as-is — no evidence it's in use. Verified: openssl s_client/curl confirm the Let's Encrypt cert (not self-signed) and Strict-Transport-Security header on https://portainer.jayfield.org, http:// redirects 301 to https://, and ss -ltnp shows 9443 bound to 127.0.0.1 only (via docker-proxy), so it's no longer reachable on the public interface at all — access is now exclusively via https://portainer.jayfield.org. One transient wrinkle during rollout: the very first few requests through the new proxy vhost returned a stray 401 with dyndns's Basic Auth realm ("Restricted Content") — looked alarming, but LogLevel trace8 on the vhost showed Apache correctly SNI-matching and proxying to the new container with a clean backend 200; resolved itself within seconds (looked like a stale pooled proxy connection surviving the container recreate) and was stable across repeated checks before the debug logging was removed again.

• fail2ban: sshd jail was evadable via low-and-slow attempts — done 2026-07-19 Default findtime=30m/maxretry=3 never triggered for 47.76.192.176, which had been hitting root/jayfield logins roughly every 33 minutes for most of a day — just outside the 30-minute window, so failures kept aging out before reaching 3 strikes. Confirmed via fail2ban.log (Found/Ban counts) that no other jail showed the same per-IP timing pattern; a separate, distributed single-shot scan on apache-noscript was noted but left open (see README.md). Fixed: /etc/fail2ban/jail.d/sshd-findtime.local overrides sshd to findtime=1d, maxretry=4, bantime=1d, kept in its own drop-in rather than editing the Debian-packaged jail file. Applied via fail2ban-client reload sshd; 47.76.192.176 was banned on its very next attempt after the change.

• fail2ban: dovecot jail had the same low-and-slow gap as sshd — done 2026-07-25 found while auditing which IPs were hitting webmail hard enough to deserve a ban but weren't in any currently-banned list. Same default findtime=30m/maxretry=3 as sshd's pre-fix config. At least 4 IPs (47.95.196.60 167 hits, 82.157.132.42 144, 120.48.118.132 76, 188.164.195.116 57, 157.245.38.26 34) ran a coordinated IMAP password spray against generic mailbox names (admin@, test@, support@, sales@jayfield.org) starting almost simultaneously around 2026-07-21 22:40-23:38, each pacing itself at ~1 attempt/22-25min — just outside the 30-minute window, so none ever tripped 3 strikes. 120.48.118.132 was still active as late as 2026-07-25 13:32, so the campaign was ongoing, not historical. postfix-sasl shares the same default thresholds and mail stack but showed much lower per-IP volume in this pass — worth a follow-up check if it starts trending the same way. Follow-up 2026-07-26: checked postfix-sasl — still just scattered background noise, not a coordinated campaign. Top offender (172.94.9.203) logged 32 hits but all from late June/early July, none since; current activity is single/double attempts from many different IPs (no repeated per-IP low-and-slow pattern like the dovecot spray). findtime=30m/maxretry=3 defaults left as-is — no fix needed unless this changes. Fixed: /etc/fail2ban/jail.d/dovecot-findtime.local, same values as the sshd fix (findtime=1d, maxretry=4, bantime=1d). Applied via fail2ban-client reload dovecot; verified live with fail2ban-client get dovecot findtime/maxretry/bantime and confirmed the jail's existing ban state (47.88.56.201, 29 total bans) survived the reload.