jens/docs-alpha.jayfield.org
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e9b51dbffe2fac183cc645b8552a21f665d3fe68
docs-alpha.jayfield.org/TODO.md
T
jensandClaude Sonnet 5 628f75303e Route attacker-check reports to root@jayfield.org, add dated archive 
root@jayfield.org didn't resolve to anything (missing from both
/etc/aliases and vmail.aliases) - added it as a new alias to
jens@jayfield.org, matching the existing postmaster/hostmaster/
webmaster/wlan pattern, so the weekly report doesn't just bounce.
Also added per-run dated report files under
/var/log/attacker-check/reports/ alongside the existing CSVs, for a
plain-text historical archive browsable by date.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-26 14:45:08 +02:00

628 lines
44 KiB
Markdown
Raw Blame History

# alpha.jayfield.org — TODO
Outstanding hardening items (see `README.md` for the full service overview).
Nothing here is urgent; all are low-risk, no-downtime changes.
- [x] **Weekly attacker-summary report routing to `root@jayfield.org`** — done 2026-07-26.
`attacker-check.py`'s report email was pointed at `root@jayfield.org`, but that address
didn't resolve to anything (not in `/etc/aliases`, not in `vmail.aliases`) — would have
bounced every week. Added it as a new `vmail.aliases` row → `jens@jayfield.org`, matching
the existing `postmaster`/`hostmaster`/`webmaster`/`wlan` pattern. Also added per-run
dated report files (`/var/log/attacker-check/reports/<date>.txt`), alongside the two
existing append-only CSVs. Verified: `postmap -q root@jayfield.org
mysql:/etc/postfix/sql/aliases.cf` resolves correctly, a live re-run delivered to
`jens@jayfield.org` (`orig_to=<root@jayfield.org>` in the Postfix log) and wrote
`reports/2026-07-26.txt`.
- [x] **BIND's own DNSSEC validation was broadly broken for signed domains** — found and fixed 2026-07-26
while investigating why third-party apt repos (`download.docker.com`, `ppa.launchpad.net`) kept
failing to resolve ("Temporary failure resolving") during prep for an OS release upgrade.
`dig`-direct queries and `apt`/`curl` (via the real NSS/`systemd-resolved` path,
`127.0.0.53` → forwards to local BIND on `127.0.0.1`) gave inconsistent results for the
same hostname, which pointed at BIND's own recursion rather than a network flake.
Confirmed with a quick sample: `github.com`, `launchpad.net`, `ppa.launchpad.net`,
`deb.debian.org`, and `download.docker.com` **all** returned `SERVFAIL` when queried
directly against `127.0.0.1`, every time (4/4 reproductions), while unsigned zones
(e.g. `archive.ubuntu.com`) resolved fine — i.e. broken specifically for DNSSEC-signed
domains, not domain-specific and not transient. Ruled out fragmentation/router
interference: forcing `+tcp`, `+bufsize=512`, and `+noedns` against BIND all still
SERVFAILed, while the same lookups against `8.8.8.8` over the same network path
succeeded fine (`NOERROR`) — so it wasn't a UDP/EDNS size or FritzBox path issue.
`dnssec.log` (one of the per-category log files already split out in
`named.conf.logging`) gave the exact cause: `validating com/DS: no valid signature
found`. `rndc managed-keys status` showed two root trust-anchor keys — key id `20326`
"trusted since" 2022-03-17 and key id `38696` "trusted since" 2025-02-10 (consistent
with the real 2025 ICANN root KSK rollover) — both looked superficially fine (neither
marked for removal), but validation against the live root zone was failing regardless.
`bind9`/`bind9-libs`/`dns-root-data` were already at the latest available jammy
versions, ruling out a simple stale-package fix. Root cause is almost certainly that
this BIND instance only started doing full local recursive resolution on 2026-07-19 (see
the forwarders-removed entry below) — before that it was forwarding to the ISP's
resolver, so it was never actively tracking/validating the root DNSSEC chain itself,
and its RFC 5011 managed-keys state was stale/inconsistent by the time real validation
started happening.
Fixed with the standard recovery procedure for this class of problem: stopped `named`,
backed up `/var/cache/bind/managed-keys.bind`(`.jnl`) to
`/root/removed-configs-backup/bind-managed-keys-reset-20260726131408/`, deleted the
live copies (forcing BIND to rebuild its trust anchor from scratch via RFC 5011 on next
start, exactly as a fresh install would), restarted `named`. Verified: all 5 previously
SERVFAILing domains above now return `NOERROR`; `jayfield.org`'s own authoritative zone
still resolves correctly (unaffected by the restart); `rndc managed-keys status` now
shows both root keys freshly re-trusted as of the restart timestamp, confirming a real
re-validation happened rather than a fluke.
**Why this mattered beyond the immediate apt problem**: this box uses BIND as its
general OS resolver (via `systemd-resolved` → `127.0.0.1`), so this was silently
breaking DNSSEC validation for *any* signed external domain something on this host
might need to reach — not just apt/PPA lookups.
- [x] **Cleaned up an incomplete 20.04→22.04 release upgrade** — found and fixed 2026-07-26,
while investigating third-party apt repos ahead of a planned 22.04→24.04 upgrade.
`/var/log/dist-upgrade/` showed a `do-release-upgrade` run from 2026-07-18 that got most
of the way through (core OS, Apache, BIND, SSH — matches everything else fixed this
week) but stalled at an interactive dpkg conffile prompt for `/etc/systemd/resolved.conf`
and was never resumed. Left behind: `mariadb-server-10.3`/`mariadb-client-10.3` (and
`galera-3`, `fuse`, a couple of `libtss2`/`libsemanage` libs) still installed and marked
"pending removal" but never actually replaced by jammy's `mariadb-server-10.6`; and three
third-party apt sources (`docker.list`, `rspamd.list`, `ondrej-ubuntu-php-focal.list`)
still pointed at `focal`, with the PHP PPA disabled entirely (the release-upgrade tool
correctly remapped its `deb` line to `jammy` but left it commented out — `.distUpgrade`
backups of the originals were left alongside, clearly meant to be manually re-enabled
after upgrade completion, which never happened). System itself was stable throughout
(`dpkg --audit`/`apt-get check` both clean) — not an emergency, just deferred work.
Fixed: pointed all three sources at `jammy` (backed up originals to
`/root/removed-configs-backup/apt-repo-fixes-20260726132055/` and
`dyndns-per-host-auth-...`-adjacent dirs), confirmed with `apt update` that all three now
fetch cleanly (only worked once the DNSSEC fix above landed — this Docker repo failure
had looked like "just" a stale codename, but was actually blocked by the SERVFAIL bug
too). Reviewed rspamd's actual 3.11.0→4.1.3 changelog against this host's live
`local.d`/`override.d` config before upgrading — none of the documented breaking changes
(ed25519 DKIM, `mx_check` symbol renames, neural autolearn, worker `ssl=true`, custom Lua
`task:get_urls()`/`orig` address handling) touch anything actually configured here
(stock RSA DKIM/ARC, Redis-backed Bayes, milter proxy, no custom Lua). Ran
`apt upgrade --dry-run` first and confirmed **zero removals** and no `mariadb`/`galera`
involvement (a plain `apt upgrade` correctly refuses the `mariadb-server` metapackage
transition since it requires package replacement, not just a version bump) before
applying: 77 packages upgraded, 13 newly-installed dependency libraries, `docker-ce`
5:28.1.1→5:29.6.2, PHP 7.4.33/8.0.30 rebuilt for jammy (still the same, EOL, PHP
versions — just properly matching this OS release instead of running focal binaries),
rspamd 3.11.0→4.1.3.
`needrestart` flagged ~25 services after the upgrade, including `ssh.service`,
`named.service`, `mariadb.service`, `postfix@-.service`, `dovecot.service`, and
`fail2ban.service`. Restarted `ssh` first and deliberately (`sshd -t` config check, then
restart, then verified with a genuinely fresh SSH connection rather than assuming the
existing one survived) given it's the only remote access to this box; then the other
mail/DNS/security services one at a time with functional checks after each (SMTP/IMAP
banners, `fail2ban-client status` confirming all 19 jails and the custom
`findtime=1d` sshd/dovecot hardening survived, live `dig` against BIND). Confirmed
`docker-ce`'s own postinst had already restarted the daemon automatically (not in
needrestart's list) and all 5 containers (`portainer`, `nextcloud`, `nextcloud-redis`,
`nextcloud-db`, `web`) came back up without disruption; `rspamd 4.1.3` config syntax
checks clean (`rspamadm configtest`) with only a pre-existing, non-blocking
task-timeout-tuning advisory, web UI reachable. Left `user@1001.service` (a per-login
systemd session, not a real service) unrestarted since restarting it would disrupt the
active login for no real benefit.
**`mariadb-server-10.3`→`10.6` migration completed 2026-07-26**: took a full logical
backup first (`mysqldump --all-databases --routines --triggers --events
--single-transaction`, ~2MB, verified complete — all 4 databases including `vmail`
present, proper "Dump completed" footer) plus a tarball of `/etc/mysql`, to
`/root/db-backups/`. Confirmed via `apt install --dry-run mariadb-server` that the
transition was exactly as expected (remove `mariadb-server/client(-core)-10.3` +
`galera-3`, install the `-10.6`/`galera-4` equivalents, nothing else touched) before
running it for real. Went smoothly: package postinst detected the old 10.3 datadir and
ran the system-table upgrade automatically (confirmed via `mariadb-upgrade`: "already
upgraded to 10.6.23-MariaDB... no need to run mysql_upgrade again"). Verified end-to-end
afterward — `vmail` schema intact (1 domain, 2 accounts, 4 aliases, matching pre-upgrade
state), `doveadm user '*'` correctly lists both real mailboxes
(`jens@`/`otto2022@jayfield.org`), and Postfix's own MySQL lookups
(`postmap -q ... mysql:/etc/postfix/sql/{domains,accounts}.cf`) resolve correctly against
the upgraded database. No galera clustering was actually in use (just an unused compiled-in
dependency), so the `galera-3`→`4` swap needed no further action.
- [x] **22.04 → 24.04 (Noble) release upgrade** — completed 2026-07-26, once the box was in a
clean, consistent 22.04 state (all the items above). Confirmed beforehand that the
dyndns app's PHP 7.4 dependency (already EOL) still has a `noble` build in Ondřej's PPA
(initial check of the plain `Packages` file gave a false "not available" — that file
404s uncompressed, only `Packages.gz` is actually published; the correct check confirmed
7.4 through 8.5 all build for `noble`), so no PHP migration was needed first.
**How it was run, given exactly what killed the *previous* upgrade attempt** (the
2026-07-18 one documented above, stuck forever at an unanswered dpkg prompt): launched
`do-release-upgrade -m server -f DistUpgradeViewNonInteractive` fully detached
(`setsid`/`nohup`, stdin from `/dev/null`, logged to a file), so it can't hang waiting on
an interactive answer and survives regardless of what happens to the SSH session. Did a
full test reboot first (not just service restarts) to confirm the box comes back up
cleanly on its own — it did, DNS/DNSSEC/mail/Docker all fine — before relying on the same
mechanism for the upgrade's own final reboot.
First attempt refused to start ("Please install all available updates for your release
before upgrading"): 12 more packages turned out to be left over from the *original*
2026-07-18 incomplete upgrade that the earlier cleanup pass had missed (`fuse`→`fuse3`,
`fwupd`, `open-vm-tools`, `passwd`, `mailutils`, etc. — all needing package replacement,
same pattern as the `mariadb-server` transition). `apt full-upgrade --dry-run` confirmed
a clean transition (11 upgraded, 20 newly installed, only 5 old-focal-era libraries
removed, e.g. `libmailutils6`→`8`), applied it, then relaunched.
Second run completed the package transition successfully (confirmed via
`/etc/os-release` → `Ubuntu 24.04.4 LTS`), but not before an alarming few minutes where
SSH was completely unreachable (`Connection refused`, not a timeout) — turned out to be
`openssh-server` itself being replaced partway through installing/configuring hundreds
of packages, not an actual reboot or network failure (`ssh.service`'s own "active since"
timestamp landed right in the middle of that window). The non-interactive frontend did
hit a real conffile prompt this time (`/etc/fail2ban/jail.conf`) and — unlike the
2026-07-18 attempt — auto-answered it instead of hanging, proving the fix works. Also
correctly recognized `php7.4`/`php8.0`/`rspamd` as "foreign" (third-party-sourced,
repos disabled during the upgrade) and left them alone rather than deleting them as
obsolete — except it did delete `rspamd` anyway (see below). Rebooted into the new
kernel (`6.8.0-136-generic`) the same way as the earlier test reboot; came back up
cleanly again.
**Two real problems found post-reboot, both fixed same day:**
- `fail2ban` failed to start: `Bad value substitution ... 'ignorecommands_dir' which is
not a valid option name` in the `apache-fakegooglebot` jail. Root cause: the
non-interactive frontend kept our existing `jail.conf` at the earlier conffile prompt
(we've never touched it directly — all our real customization lives in
`jail.d/*.local` drop-ins), but the *upgraded* `fail2ban` package expects a
`[DEFAULT]` value for `ignorecommands_dir` that isn't defined anywhere in either the
old or the freshly-`--force-confnew`-reinstalled `jail.conf` — a genuine gap in the
`noble` package (the referenced script directory,
`/etc/fail2ban/filter.d/ignorecommands/`, does exist on disk from the package's own
file list, just nothing points `ignorecommands_dir` at it). Fixed with a small owned
drop-in, `jail.d/ignorecommands-dir.local`, defining it — same convention as our other
fail2ban customizations, no package-owned file touched. Verified: all 19 jails present,
custom `findtime=1d` sshd/dovecot hardening intact.
- `rspamd` was **fully removed** (`dpkg -l` showed `rc`, not just held back like
`php7.4`/`php8.0` were) — its `systemd` unit was also masked (symlinked to `/dev/null`)
by the upgrade tool as a safety measure for a package it could no longer account for.
Its `.list` source was gone entirely too (unlike Docker/PHP, whose disabled entries
were left commented-out) — the upgrade tool had migrated it to the newer deb822
`.sources` format instead (`rspamd.sources`, `Enabled: no`, already correctly
re-pointed at `noble`). Docker's source, meanwhile, was gone with *no* `.sources`
replacement at all (had to recreate `docker.list` from scratch, keyring was still
intact). Fixed: enabled the `rspamd.sources` and `ondrej-ubuntu-php-*.sources` files,
recreated `docker.list` pointed at `noble`, confirmed all three fetch cleanly, then
`apt install rspamd` (candidate correctly resolved to the same `4.1.3` build, not
Ubuntu's own older native `3.8.1-1ubuntu3` noble package). Verified for real this
time, not just "service active" (which had been a false positive from a legacy SysV
init shim silently succeeding with no binary behind it): real daemon process,
listening on `11332`/`11333`/`11334`, `rspamadm configtest` clean, and
`rspamc symbols` against a test message showing genuine scanning (DMARC checks, ARC
signing with the real `jayfield.org`/`2022` key, full symbol scoring) — not just an
empty process. Note for later: `milter_default_action = accept` in Postfix meant mail
kept flowing the whole time rspamd was down, just completely unfiltered and unsigned
— not a full outage, but worth knowing that gap exists.
Full verification pass after all fixes: SSH hardening intact
(`PermitRootLogin no`/`PasswordAuthentication no`), DNSSEC still validating
(`github.com` → `NOERROR`), `jayfield.org` zone resolving correctly, all 5 Docker
containers up without disruption, all 7 Apache vhosts responding as expected, the dyndns
app verified end-to-end (real per-host-credentialed update via `vpn`'s login → correct
TSIG key → BIND → `dig` confirms). `mariadb-server` was upgraded again automatically as
part of this same run, `10.6.23`→`10.11.14` (noble's native version) — verified `vmail`
schema intact (1 domain, 2 accounts, 4 aliases, unchanged), `doveadm`/Postfix MySQL
lookups both still correct.
- [ ] **Mail server IP (`89.58.8.149`) listed on Barracuda's BRBL** — found
2026-07-25 (user report: "jayfield.org is on the Barracuda MX
blacklist"). Confirmed via `dig +short 149.8.58.89.b.barracudacentral.org A`
→ `127.0.0.2` (listed); clean on Spamhaus ZEN, SpamCop, and SORBS —
Barracuda-specific. Checked the server for actual abuse before
requesting removal: mail queue empty, outbound volume ~11 msgs over
the prior week, only known SASL logins (`jens@`/`otto2022@jayfield.org`
from the usual home IP), no backscatter, correct PTR
(`alpha.jayfield.org`), SPF/DKIM/DMARC (`p=reject`) all already live —
no sign of a compromised account or open relay, so most likely a
stale/inherited reputation hit on the IP rather than active abuse
from this host.
Submitted a removal request via
`https://www.barracudacentral.org/rbl/removal-request` on 2026-07-25.
**Confirmation number: `BBR21785009436-19117-9182`.** Per Barracuda's
response, the IP gets a temporary 48h reputation bump to "normal"
while they investigate (up to ~1h to propagate), after which it
either stays clear or reverts to listed.
**Status 2026-07-26: still clear** (`dig +short
149.8.58.89.b.barracudacentral.org A` → NXDOMAIN, no `127.0.0.2`) —
matches the 1h check (routine `trig_01F1RMhEtNHtAbuHb9ffe1oX`, fired
2026-07-25 21:02 UTC, also clear). This is expected during the 48h
"normal" bump window, not yet proof of permanent delisting. The real
test is the 48h check (routine `trig_019GayXSo5TW5eSxiJ99Ghrp`,
scheduled 2026-07-27 20:03 UTC) — if it reverts to listed, next step
is a manual follow-up to Barracuda citing the confirmation number
above.
- [x] **dyndns app has no real per-user auth binding** — found 2026-07-20,
fixed 2026-07-26. `index.php`'s `pass == '_NO_PASS_'` check is a
hardcoded magic string, not a real per-user credential — nothing tied
the single shared Apache Basic Auth identity (`jens`, the only
`.htpasswd` account) to *which* hostname label got updated. Confirmed
live in the access log: a FritzBox ("Fritz!Box DDNS/1.0.3") was
authenticating as `jens` while updating the `uschi` record — i.e. every
device was already relying on this exact gap, self-policing which
host it claimed via an unauthenticated `hostname=`/`user=` param.
Fixed with real per-host TSIG binding, not just an app-level check:
- Added a dedicated `.htpasswd` account for each of `kack`, `test`,
`uschi`, `vpn` (`jens`'s existing account/password untouched).
- Generated 5 per-host TSIG keys (`hmac-sha512`, `tsig-keygen`), one
per hostname, each in its own `/etc/bind/<host>.dyndns.jayfield.org.key`
(`root:bind 640` + `setfacl` grant for `www-data`, same pattern as
the original key).
- Replaced the zone's flat `allow-update { key dyndns.jayfield.org; };`
in `named.conf.default-zones` with an `update-policy` granting each
new key permission to update *only* its own name (`grant "<host>"
name <host>. A AAAA TXT;`), plus one explicit legacy fallback grant
for the original shared key (`subdomain` rule, zone-wide) — kept
temporarily so unmigrated devices don't break (see below).
- `nsupdate.php`/`nsupdate_ipv4.php`/`nsupdate_ipv6.php` now sign with
the key belonging to `$_SERVER['PHP_AUTH_USER']` (the *authenticated*
identity), not whatever hostname was requested. If they match, the
new per-host key is used — BIND itself then refuses the update if
that key isn't granted that name, independent of any PHP-side
check. If the authenticated user is specifically the legacy `jens`
account and doesn't match, it falls back to the old shared key
(logged via `syslog(LOG_WARNING, ...)` for migration tracking) —
every other account gets rejected outright (`-1`) on a mismatch,
so newly-created per-host logins can never touch another host, even
via the fallback.
- Also fixed a pre-existing bug found in `nic/update.php` while
testing this: it printed `good <ip>` unconditionally, without ever
checking `nsupdate`'s return code — so a rejected update silently
looked successful to the calling device. Now prints the real
dyndns2-protocol response (`good <ip>` / `badauth`).
All files backed up to
`/root/removed-configs-backup/dyndns-per-host-auth-20260726094500/`.
Validated with `named-checkconf`/`apache2ctl configtest`, applied via
`rndc reload`/`systemctl reload apache2`. Tested both by driving
`nsupdate()` directly as `www-data` and end-to-end over HTTPS with
the new `vpn` credentials: own-host updates succeed, cross-host
attempts from new accounts are rejected (`-1`/`badauth`) with the
target record provably unchanged, and the legacy `jens`→`uschi`
pattern still succeeds unchanged. Test IPs used during verification
were restored to the real production values (`test`, `uschi`, `vpn`)
immediately after; live `dig` confirmed the final state matches the
pre-change records exactly.
**Follow-up**: the `jens` legacy fallback (and its `.htpasswd`
account) should stay only until every real device — including
whichever ones are still authenticating as `jens` to update `uschi`
— has been reconfigured with its own dedicated login. Once `dyndns
legacy shared-key update` stops appearing in `syslog`, remove the
fallback `grant` and the `jens` account can be scoped down to just
its own host too.
**`vpn` migrated 2026-07-26**: home FritzBox reconfigured with the
new `vpn` login and confirmed live — after a WAN IP change (real
connection recycle, not a test), the access log shows a genuine
`Fritz!Box DDNS/1.0.3` request authenticated as `vpn` updating
`vpn.dyndns.jayfield.org` to the new IP (`217.229.60.187`), and `dig`
confirms the zone picked it up. First host fully off the legacy
fallback. `uschi`'s device was reconfigured too but hasn't sent a
real authenticated update yet (FritzBox only re-registers on WAN IP
change or manual renew, neither has happened since) — not yet
confirmed live.
**`kack`/`test` removed 2026-07-26** instead of migrated: `kack`
hadn't updated since Feb 2024 and `test` showed no real traffic in
the (short) per-vhost access log history either — both looked like
stale/unused entries rather than active devices, so removed outright
rather than given dedicated logins. Deleted their `A`/`AAAA`/`TXT`
records via `nsupdate` (using the legacy shared key), dropped their
`update-policy` grants and `include` lines from
`named.conf.default-zones`, moved their per-host TSIG key files and
`.htpasswd` entries (`htpasswd -D`) out. Backed up to
`/root/removed-configs-backup/dyndns-remove-kack-test-20260726101439/`.
Validated with `named-checkconf`/`apache2ctl configtest`, applied via
`rndc reload`/`systemctl reload apache2`, confirmed both names now
resolve to nothing (`dig` empty) and `jens`/`uschi`/`vpn` untouched;
`rndc sync` forced the on-disk zone file to reflect it immediately
rather than waiting for BIND's next periodic dump.
**`jens.dyndns.jayfield.org` hostname removed 2026-07-26** — same
story as `kack`: stale since Feb 2024, no device actually updating
it (the only `jens`-authenticated traffic ever seen was the legacy
fallback updating *other* hosts, never its own). Deleted the record,
dropped its dedicated `update-policy` grant and key `include` line,
moved `jens.dyndns.jayfield.org.key` to
`/root/removed-configs-backup/dyndns-remove-jenshost-20260726101844/`.
Important distinction from the `kack`/`test` removal: this only
removed the *hostname*. The `jens` **login** and the legacy shared-key
fallback grant (`grant "dyndns.jayfield.org" subdomain
dyndns.jayfield.org. ANY;`) were deliberately left in place, since
`uschi`'s device still authenticates as `jens` through that fallback
— removing the login would have broken it immediately. Re-verified
after this change (direct `nsupdate()` call as `www-data`, writing
`uschi`'s own current real IP back so nothing actually changed) that
the `jens`→`uschi` legacy path still returns `0`. `named-checkconf`/
`rndc reload` clean throughout.
**Legacy fallback removed entirely 2026-07-26** — confirmed the
`jens` fallback wasn't needed for `uschi` anymore (its own dedicated
login/key from the original fix is what should be used going
forward), and with `jens`'s and `kack`/`test`'s own hostnames already
gone, nothing was left depending on it. Removed the
`grant "dyndns.jayfield.org" subdomain dyndns.jayfield.org. ANY;`
fallback grant and its key `include` from `named.conf.default-zones`;
retired the original shared TSIG key itself
(`/etc/bind/dyndns.jayfield.org.key`, in production since March 2022)
to
`/root/removed-configs-backup/dyndns-remove-legacy-fallback-20260726102228/`;
removed the now-dead legacy-path branch from
`nsupdate.php`/`nsupdate_ipv4.php`/`nsupdate_ipv6.php` — `nsupdate()`
now unconditionally rejects (`-1`) any request where the
authenticated user doesn't match the requested host, no exceptions.
Verified: `jens` attempting to update `uschi` now fails immediately
(`ret=-1`, no BIND round-trip at all — rejected in PHP before ever
building an `nsupdate` command), while `uschi` and `vpn` updating
themselves still succeed (`ret=0`, confirmed via `dig` against their
real IPs). Per-user auth binding for dyndns is now unconditional,
closing this item out completely — no more grandfathered exceptions
of any kind. The `jens` `.htpasswd` account itself was left in place
(not asked to remove it, and it's now inert for DNS updates anyway
since it has no hostname or grant left to use).
**`uschi` migration confirmed live 2026-07-26**: after the legacy
fallback was removed, a real WAN IP change (connection recycle, not
a test) produced a genuine `Fritz!Box DDNS/1.0.3` request
authenticated as `uschi` updating `uschi.dyndns.jayfield.org` to
`92.252.63.190`, confirmed via `dig`. (One wrinkle along the way: an
earlier verification test of mine — writing `uschi`'s old IP back
to confirm the fix — landed moments after that device's *first*
post-migration update and clobbered it back to the stale value; the
device's next natural IP change overwrote it correctly, so no
lasting effect, but a reminder to use non-colliding test values when
a real device might update concurrently.) Both `uschi` and `vpn` are
now fully migrated to dedicated per-host logins with no legacy
exceptions remaining anywhere in the setup.
**`vpn` re-confirmed 2026-07-26**: a second WAN IP change produced
another clean authenticated update (`87.180.9.214`), this time with
no test collision — TXT timestamp matches exactly. Both `uschi` and
`vpn` have now each been confirmed live across a real IP change with
their own dedicated login; this item is fully done, nothing further
to track.
**`jens` password rotated 2026-07-26**: since the account is now
unused (no hostname or key left to act on) and its old password
predates this whole migration, rotated it for hygiene via
`htpasswd -b`. Old `.htpasswd` backed up to
`/root/removed-configs-backup/dyndns-jens-rotate-20260726124040/`.
Verified: new password authenticates fine (`200`), old password now
gets `401`, and — as expected since the fix — even a successful
`jens` login still can't perform any DNS update (`badauth`). Account
kept around (not asked to remove it), but fully inert either way.
- [x] **dyndns/default-vhost access logging is a no-op** — done 2026-07-25
found 2026-07-20 while investigating the command injection below.
`dyndns.jayfield.org`, `jayfield.org`, `www.jayfield.org`,
`mail.jayfield.org`, `web.jayfield.org`, and `cloud.jayfield.org`
had no `CustomLog` directive of their own, and `access.log` (only
written by `default-ssl.conf`/`000-default.conf`) had 0-8 lines —
looking like no real vhost was being logged at all.
Root cause (2026-07-25 re-check): the fallback assumption was wrong.
There's no global `CustomLog access.log` in `apache2.conf`; instead
`conf-enabled/other-vhosts-access-log.conf` enables `mod_vhost_combined`,
which was already writing every vhost-without-its-own-`CustomLog`'s
traffic to `/var/log/apache2/other_vhosts_access.log` (5,432 lines,
confirmed real traffic incl. bot scans against `cloud.jayfield.org`).
So requests were never actually un-logged — just commingled in one
shared, harder-to-grep file instead of `access.log`.
Fixed anyway, since a dedicated per-site log remains the better
end state for incident review: added an explicit
`CustomLog ${APACHE_LOG_DIR}/<site>_access.log combined` to every
`<VirtualHost>` block for `dyndns`, `jayfield.org`, `www`, `mail`,
`web`, `cloud`, and `portainer` (the `:80` and `:443`/`le-ssl` blocks
of each site share one log file). Configs backed up to
`/root/removed-configs-backup/vhost-customlog-20260725203959/`.
`/etc/logrotate.d/apache2` already globs `*.log`, so no logrotate
change needed. Validated with `apache2ctl configtest`, applied via
`systemctl reload apache2`, verified live with `curl` against all 7
domains and confirmed each new `<site>_access.log` recorded the hit.
- [x] **OS command injection in the dyndns update scripts** — done 2026-07-20
`/var/www/dyndns/nsupdate.php`, `nsupdate_ipv4.php`, and
`nsupdate_ipv6.php` (used by both `index.php` and `nic/update.php`,
the router/ddclient-facing endpoint) built a shell command string by
directly interpolating unsanitized `$_GET` input (`hostname`/`myip`/
`ip`, which become `$host`/`$ip`/`$ipv4`/`$ipv6`) and passed it to
PHP's `exec()`, which runs it via `/bin/sh -c` — no
`escapeshellarg()`/`escapeshellcmd()` anywhere. A request like
`hostname=$(id>/tmp/pwned).dyndns.jayfield.org` would have executed
arbitrary shell commands as `www-data` (the domain-suffix check
still passed after the `.`-split). `nic/update.php` had no
application-level auth of its own at all.
Mitigating factor: the whole `dyndns.jayfield.org` vhost sits behind
Apache Basic Auth, so this needed valid (or leaked/brute-forced)
dyndns credentials to reach — not exploitable by an anonymous
visitor. Compounding factors found during the audit: PHP is
`7.4.33`, EOL since Nov 2022, `disable_functions` was empty (`exec`
fully available); `index.php`'s `pass == '_NO_PASS_'` check is a
hardcoded magic string, not real per-user auth, so any valid dyndns
credential can update *any* of the zone's 5 hostnames (`jens`,
`kack`, `test`, `uschi`, `vpn`), not just its own — left as-is, out
of scope for this fix; and the vhost's access logging turned out to
be a no-op (shared `access.log` has 0 lines), so past exploitation
couldn't be fully ruled out from logs alone — though file mtimes,
`www-data` processes/crontab, and zone contents all looked
unremarkable, consistent with unexploited.
Fixed: all three files backed up to
`/root/removed-configs-backup/dyndns-preinjectionfix-<timestamp>/`,
then rewritten to build the nsupdate command list as a plain string
written to `proc_open()`'s stdin pipe using the **array form** of
the command (`array("/usr/bin/nsupdate", "-k", ...)`) — this never
invokes a shell at all, so no value can be interpreted as shell
syntax regardless of content. Added strict `preg_match` validation
requiring `$host`/`$domain` to be shaped like a valid hostname
(letters/digits/hyphens, dot-separated labels) and
`filter_var(..., FILTER_VALIDATE_IP, FILTER_FLAG_IPV4/IPV6)` on the
IP before use, as defense in depth on top of removing the shell.
**Regression found and fixed along the way**: functional testing
showed legitimate updates were *already silently failing*
(`update failed: REFUSED`, `permission denied` reading the TSIG
key) — turned out `www-data` was never granted read access to
`/etc/bind/dyndns.jayfield.org.key` when its permissions were
tightened to `640 root:bind` in the item below, so the dyndns web
update feature had been broken for real users since 2026-07-19,
independent of this vulnerability. Fixed with a narrowly-scoped
POSIX ACL (`setfacl -m u:www-data:r
/etc/bind/dyndns.jayfield.org.key`, `acl` package installed) rather
than adding `www-data` to the `bind` group — the group also owns
`rndc.key` (full remote BIND control), which would have been a much
bigger privilege grant than dyndns needs.
Verified via `sudo -u www-data php` driving the real functions
directly (same code path as the web UI, without needing the Basic
Auth password): two injection payloads (`$(...)` and `;`-chained)
both rejected (`-1`, no `/tmp` side-effect file created), an
invalid IP rejected (`-1`), and a legitimate update succeeded
(`0`), confirmed live in the zone via `dig`, then cleaned up (test
record deleted via `nsupdate`, temp test script removed). Also
confirmed unauthenticated requests still get `401` from Apache
before ever reaching the PHP, and `apache2ctl configtest` stayed
clean throughout.
- [x] **Public static site (`jayfield.org`/`www.jayfield.org`) was 401'ing every visitor** — done 2026-07-20
Found while auditing which services are reachable without credentials:
a stray `/var/www/html/.htaccess` (`AuthType Basic`, same
`AuthName "Restricted Content"`/`AuthUserFile /etc/apache2/.htpasswd`
as the intentional `dyndns.jayfield.org` protection) was silently
requiring login for the DocumentRoot both `jayfield.org` and
`www.jayfield.org` share — `/var/www` has `AllowOverride all`, so it
took effect with no vhost-level directive needed. File was dated
2025-02-18 (`index.html`/`apache.html` themselves are from 2022) —
predates every other change in this doc by well over a year, almost
certainly a forgotten leftover rather than intentional, and
contradicts `README.md`'s own description of both as a public static
site.
Fixed: moved to `/root/removed-configs-backup/htaccess-var-www-html.<timestamp>`
(recoverable, not deleted — same convention as this file's other
removed-config backups). No Apache reload needed (`.htaccess` is
read per-request). Verified both `jayfield.org` and
`www.jayfield.org` now return `200` with real page content;
`dyndns.jayfield.org`'s own (intentional, vhost-level) Basic Auth is
untouched and still returns `401` without credentials.
- [x] **Lock down the dyndns TSIG key file permissions** — done 2026-07-19
`/etc/bind/dyndns.jayfield.org.key` was `644` (world-readable), holding
the shared secret that authenticates dynamic DNS updates for
`dyndns.jayfield.org` with no IP-based `allow-update` backing it up.
Fixed: `chmod 640` + `chown root:bind` (owner was already correct),
matching `rndc.key`'s permissions.
- [x] **Suppress BIND version disclosure** — done 2026-07-19
No `version` statement in `named.conf.options`, so
`dig CH TXT version.bind @ns1.jayfield.org` leaked the exact BIND
version (`9.18.39`) — a minor fingerprinting aid for attackers.
Fixed: added `version "unknown";` inside the `options {}` block,
validated with `named-checkconf`, and applied via `rndc reload`.
Verified `dig CH TXT version.bind` now returns `"unknown"`.
- [x] **Add response rate limiting (RRL)** — done 2026-07-19
No `rate-limit {}` block was configured. Since `allow-query { any; }`
is required for a public authoritative zone, RRL is the standard
mitigation against the server being used for spoofed-source DNS
reflection/amplification against third parties.
Fixed: added `rate-limit { responses-per-second 10; window 5; }` to
`named.conf.options`, validated with `named-checkconf`, and applied
via `rndc reload`. Started with conservative defaults (10/s, 5s
window) — revisit if legitimate high-volume resolvers get throttled.
- [x] **Add HSTS headers to all vhosts** — done 2026-07-19
No vhost sent a `Strict-Transport-Security` header, leaving a window
for on-path SSL-stripping on a user's very first plaintext request to
any domain.
Fixed: added `Header always set Strict-Transport-Security
"max-age=63072000; includeSubDomains"` to all 6 domain `:443` blocks
(`dyndns`, `jayfield.org`, `www`, `mail`, `web`, `cloud`) — skipped
`default-ssl.conf` since it's the self-signed catch-all with no real
trusted hostname for a browser to pin. Validated with
`apache2ctl configtest`, applied via `systemctl reload apache2`,
verified live with `curl -sI` against all 6 domains.
- [x] **`jayfield.org` had no `:80` vhost of its own** — done 2026-07-19
Bare `http://jayfield.org` fell through to the `dyndns` default vhost
and redirected to the wrong domain.
Fixed: added a `:80` block to `jayfield.org.conf` redirecting to
`https://jayfield.org/`, matching the `dyndns`/`mail` pattern.
Verified: `curl -sI http://jayfield.org/` → `302` with
`Location: https://jayfield.org/`.
- [x] **SSH: disable root login + password auth** — done 2026-07-19
`PermitRootLogin yes` and `PasswordAuthentication yes` were both still
enabled (the "known deviation" flagged in `SETUP.md` §1) — every
credential-stuffing bot on the internet got a real login prompt to
grind against.
Fixed: `PermitRootLogin no`, `PasswordAuthentication no`,
`PubkeyAuthentication yes` in `/etc/ssh/sshd_config` (backed up as
`sshd_config.bak.20260719225249`). The `Match Group sftp` block's own
`PasswordAuthentication yes` was left as-is — separate, already-chrooted,
no-shell/no-forwarding accounts, not part of this risk. Validated with
`sshd -t`, applied via `systemctl reload sshd`, verified live with
`sshd -T` and a fresh key-only connection before considering it done.
- [x] **Portainer's `:9443` UI is unreachable in HSTS-enforcing browsers** — done 2026-07-20
`https://alpha.jayfield.org:9443` threw
`MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT` in Firefox with no click-through
option ("keine Ausnahme kann hinzugefügt werden"). Root cause: the
`includeSubDomains` HSTS policy sent by the real `alpha.jayfield.org`
vhost (`Strict-Transport-Security` header, added in the item above)
pins the *hostname* to HTTPS-with-a-trusted-cert on every port, but
Portainer (`README.md` §Docker services) was still serving its stock
self-signed cert directly on `9443`. Not a MITM — self-inflicted by
combining HSTS with an unproxied self-signed service.
Fixed: added a `portainer` A record to `db.jayfield.org` (SOA serial
`2022032216` → `2022032217`); recreated the `portainer` container
binding the HTTPS UI to `127.0.0.1:9443:9443` instead of publishing it
(old container kept renamed until the new one was verified, then
removed, matching the update procedure documented in `README.md`);
added an Apache vhost for `portainer.jayfield.org`
(`ProxyPass`/`ProxyPassReverse` to `https://127.0.0.1:9443/`,
`SSLProxyEngine on`, `SSLProxyVerify none` + `SSLProxyCheckPeerCN off`/
`SSLProxyCheckPeerName off` for the loopback hop to Portainer's
self-signed cert) — same pattern as `web.jayfield.org`/
`cloud.jayfield.org` (`SETUP.md` §4/§5); issued a real cert via
`certbot --apache -d portainer.jayfield.org --redirect`; added the
standard HSTS header. Left the `8000` edge-agent tunnel port
published as-is — no evidence it's in use.
Verified: `openssl s_client`/`curl` confirm the Let's Encrypt cert
(not self-signed) and `Strict-Transport-Security` header on
`https://portainer.jayfield.org`, `http://` redirects 301 to
`https://`, and `ss -ltnp` shows `9443` bound to `127.0.0.1` only
(via `docker-proxy`), so it's no longer reachable on the public
interface at all — access is now exclusively via
`https://portainer.jayfield.org`.
One transient wrinkle during rollout: the very first few requests
through the new proxy vhost returned a stray `401` with dyndns's
Basic Auth realm ("Restricted Content") — looked alarming, but
`LogLevel trace8` on the vhost showed Apache correctly SNI-matching
and proxying to the new container with a clean backend `200`;
resolved itself within seconds (looked like a stale pooled proxy
connection surviving the container recreate) and was stable across
repeated checks before the debug logging was removed again.
- [x] **fail2ban: `sshd` jail was evadable via low-and-slow attempts** — done 2026-07-19
Default `findtime=30m`/`maxretry=3` never triggered for `47.76.192.176`,
which had been hitting `root`/`jayfield` logins roughly every 33 minutes
for most of a day — just outside the 30-minute window, so failures kept
aging out before reaching 3 strikes. Confirmed via `fail2ban.log`
(`Found`/`Ban` counts) that no other jail showed the same per-IP timing
pattern; a separate, *distributed* single-shot scan on `apache-noscript`
was noted but left open (see `README.md`).
Fixed: `/etc/fail2ban/jail.d/sshd-findtime.local` overrides `sshd` to
`findtime=1d`, `maxretry=4`, `bantime=1d`, kept in its own drop-in
rather than editing the Debian-packaged jail file. Applied via
`fail2ban-client reload sshd`; `47.76.192.176` was banned on its very
next attempt after the change.
- [x] **fail2ban: `dovecot` jail had the same low-and-slow gap as `sshd`** — done 2026-07-25
found while auditing which IPs were hitting webmail hard enough to
deserve a ban but weren't in any currently-banned list. Same default
`findtime=30m`/`maxretry=3` as `sshd`'s pre-fix config. At least 4 IPs
(`47.95.196.60` 167 hits, `82.157.132.42` 144, `120.48.118.132` 76,
`188.164.195.116` 57, `157.245.38.26` 34) ran a coordinated IMAP
password spray against generic mailbox names (`admin@`, `test@`,
`support@`, `sales@jayfield.org`) starting almost simultaneously around
2026-07-21 22:40-23:38, each pacing itself at ~1 attempt/22-25min —
just outside the 30-minute window, so none ever tripped 3 strikes.
`120.48.118.132` was still active as late as 2026-07-25 13:32, so the
campaign was ongoing, not historical. `postfix-sasl` shares the same
default thresholds and mail stack but showed much lower per-IP volume
in this pass — worth a follow-up check if it starts trending the same
way.
**Follow-up 2026-07-26**: checked `postfix-sasl` — still just scattered
background noise, not a coordinated campaign. Top offender
(`172.94.9.203`) logged 32 hits but all from late June/early July, none
since; current activity is single/double attempts from many different
IPs (no repeated per-IP low-and-slow pattern like the dovecot spray).
`findtime=30m`/`maxretry=3` defaults left as-is — no fix needed unless
this changes.
Fixed: `/etc/fail2ban/jail.d/dovecot-findtime.local`, same values as
the `sshd` fix (`findtime=1d`, `maxretry=4`, `bantime=1d`). Applied via
`fail2ban-client reload dovecot`; verified live with
`fail2ban-client get dovecot findtime/maxretry/bantime` and confirmed
the jail's existing ban state (`47.88.56.201`, 29 total bans) survived
the reload.
Reference in New Issue View Git Blame Copy Permalink