docs-alpha.jayfield.org/TODO.md

# alpha.jayfield.org — TODO

Outstanding hardening items (see `README.md` for the full service overview).
Nothing here is urgent; all are low-risk, no-downtime changes.

- [ ] **dyndns app has no real per-user auth binding** — found 2026-07-20,
      while fixing the command injection below.
      `index.php`'s `pass == '_NO_PASS_'` check is a hardcoded magic
      string, not a real per-user credential — there's nothing tying the
      single shared Apache Basic Auth identity to *which* hostname label
      gets updated. Any caller with valid dyndns credentials can update
      any of the zone's 5 hostnames (`jens`, `kack`, `test`, `uschi`,
      `vpn`), not just their own. Low severity today since apparently only
      one person (`jens`) has credentials, but worth fixing before adding
      any second dyndns user. Fix: either move to per-host TSIG-style
      tokens, or add a static host-allowlist per `.htpasswd` user in the
      PHP.

- [x] **dyndns/default-vhost access logging is a no-op** — done 2026-07-25
      found 2026-07-20 while investigating the command injection below.
      `dyndns.jayfield.org`, `jayfield.org`, `www.jayfield.org`,
      `mail.jayfield.org`, `web.jayfield.org`, and `cloud.jayfield.org`
      had no `CustomLog` directive of their own, and `access.log` (only
      written by `default-ssl.conf`/`000-default.conf`) had 0-8 lines —
      looking like no real vhost was being logged at all.
      Root cause (2026-07-25 re-check): the fallback assumption was wrong.
      There's no global `CustomLog access.log` in `apache2.conf`; instead
      `conf-enabled/other-vhosts-access-log.conf` enables `mod_vhost_combined`,
      which was already writing every vhost-without-its-own-`CustomLog`'s
      traffic to `/var/log/apache2/other_vhosts_access.log` (5,432 lines,
      confirmed real traffic incl. bot scans against `cloud.jayfield.org`).
      So requests were never actually un-logged — just commingled in one
      shared, harder-to-grep file instead of `access.log`.
      Fixed anyway, since a dedicated per-site log remains the better
      end state for incident review: added an explicit
      `CustomLog ${APACHE_LOG_DIR}/<site>_access.log combined` to every
      `<VirtualHost>` block for `dyndns`, `jayfield.org`, `www`, `mail`,
      `web`, `cloud`, and `portainer` (the `:80` and `:443`/`le-ssl` blocks
      of each site share one log file). Configs backed up to
      `/root/removed-configs-backup/vhost-customlog-20260725203959/`.
      `/etc/logrotate.d/apache2` already globs `*.log`, so no logrotate
      change needed. Validated with `apache2ctl configtest`, applied via
      `systemctl reload apache2`, verified live with `curl` against all 7
      domains and confirmed each new `<site>_access.log` recorded the hit.

- [x] **OS command injection in the dyndns update scripts** — done 2026-07-20
      `/var/www/dyndns/nsupdate.php`, `nsupdate_ipv4.php`, and
      `nsupdate_ipv6.php` (used by both `index.php` and `nic/update.php`,
      the router/ddclient-facing endpoint) built a shell command string by
      directly interpolating unsanitized `$_GET` input (`hostname`/`myip`/
      `ip`, which become `$host`/`$ip`/`$ipv4`/`$ipv6`) and passed it to
      PHP's `exec()`, which runs it via `/bin/sh -c` — no
      `escapeshellarg()`/`escapeshellcmd()` anywhere. A request like
      `hostname=$(id>/tmp/pwned).dyndns.jayfield.org` would have executed
      arbitrary shell commands as `www-data` (the domain-suffix check
      still passed after the `.`-split). `nic/update.php` had no
      application-level auth of its own at all.
      Mitigating factor: the whole `dyndns.jayfield.org` vhost sits behind
      Apache Basic Auth, so this needed valid (or leaked/brute-forced)
      dyndns credentials to reach — not exploitable by an anonymous
      visitor. Compounding factors found during the audit: PHP is
      `7.4.33`, EOL since Nov 2022, `disable_functions` was empty (`exec`
      fully available); `index.php`'s `pass == '_NO_PASS_'` check is a
      hardcoded magic string, not real per-user auth, so any valid dyndns
      credential can update *any* of the zone's 5 hostnames (`jens`,
      `kack`, `test`, `uschi`, `vpn`), not just its own — left as-is, out
      of scope for this fix; and the vhost's access logging turned out to
      be a no-op (shared `access.log` has 0 lines), so past exploitation
      couldn't be fully ruled out from logs alone — though file mtimes,
      `www-data` processes/crontab, and zone contents all looked
      unremarkable, consistent with unexploited.
      Fixed: all three files backed up to
      `/root/removed-configs-backup/dyndns-preinjectionfix-<timestamp>/`,
      then rewritten to build the nsupdate command list as a plain string
      written to `proc_open()`'s stdin pipe using the **array form** of
      the command (`array("/usr/bin/nsupdate", "-k", ...)`) — this never
      invokes a shell at all, so no value can be interpreted as shell
      syntax regardless of content. Added strict `preg_match` validation
      requiring `$host`/`$domain` to be shaped like a valid hostname
      (letters/digits/hyphens, dot-separated labels) and
      `filter_var(..., FILTER_VALIDATE_IP, FILTER_FLAG_IPV4/IPV6)` on the
      IP before use, as defense in depth on top of removing the shell.
      **Regression found and fixed along the way**: functional testing
      showed legitimate updates were *already silently failing*
      (`update failed: REFUSED`, `permission denied` reading the TSIG
      key) — turned out `www-data` was never granted read access to
      `/etc/bind/dyndns.jayfield.org.key` when its permissions were
      tightened to `640 root:bind` in the item below, so the dyndns web
      update feature had been broken for real users since 2026-07-19,
      independent of this vulnerability. Fixed with a narrowly-scoped
      POSIX ACL (`setfacl -m u:www-data:r
      /etc/bind/dyndns.jayfield.org.key`, `acl` package installed) rather
      than adding `www-data` to the `bind` group — the group also owns
      `rndc.key` (full remote BIND control), which would have been a much
      bigger privilege grant than dyndns needs.
      Verified via `sudo -u www-data php` driving the real functions
      directly (same code path as the web UI, without needing the Basic
      Auth password): two injection payloads (`$(...)` and `;`-chained)
      both rejected (`-1`, no `/tmp` side-effect file created), an
      invalid IP rejected (`-1`), and a legitimate update succeeded
      (`0`), confirmed live in the zone via `dig`, then cleaned up (test
      record deleted via `nsupdate`, temp test script removed). Also
      confirmed unauthenticated requests still get `401` from Apache
      before ever reaching the PHP, and `apache2ctl configtest` stayed
      clean throughout.

- [x] **Public static site (`jayfield.org`/`www.jayfield.org`) was 401'ing every visitor** — done 2026-07-20
      Found while auditing which services are reachable without credentials:
      a stray `/var/www/html/.htaccess` (`AuthType Basic`, same
      `AuthName "Restricted Content"`/`AuthUserFile /etc/apache2/.htpasswd`
      as the intentional `dyndns.jayfield.org` protection) was silently
      requiring login for the DocumentRoot both `jayfield.org` and
      `www.jayfield.org` share — `/var/www` has `AllowOverride all`, so it
      took effect with no vhost-level directive needed. File was dated
      2025-02-18 (`index.html`/`apache.html` themselves are from 2022) —
      predates every other change in this doc by well over a year, almost
      certainly a forgotten leftover rather than intentional, and
      contradicts `README.md`'s own description of both as a public static
      site.
      Fixed: moved to `/root/removed-configs-backup/htaccess-var-www-html.<timestamp>`
      (recoverable, not deleted — same convention as this file's other
      removed-config backups). No Apache reload needed (`.htaccess` is
      read per-request). Verified both `jayfield.org` and
      `www.jayfield.org` now return `200` with real page content;
      `dyndns.jayfield.org`'s own (intentional, vhost-level) Basic Auth is
      untouched and still returns `401` without credentials.

- [x] **Lock down the dyndns TSIG key file permissions** — done 2026-07-19
      `/etc/bind/dyndns.jayfield.org.key` was `644` (world-readable), holding
      the shared secret that authenticates dynamic DNS updates for
      `dyndns.jayfield.org` with no IP-based `allow-update` backing it up.
      Fixed: `chmod 640` + `chown root:bind` (owner was already correct),
      matching `rndc.key`'s permissions.

- [x] **Suppress BIND version disclosure** — done 2026-07-19
      No `version` statement in `named.conf.options`, so
      `dig CH TXT version.bind @ns1.jayfield.org` leaked the exact BIND
      version (`9.18.39`) — a minor fingerprinting aid for attackers.
      Fixed: added `version "unknown";` inside the `options {}` block,
      validated with `named-checkconf`, and applied via `rndc reload`.
      Verified `dig CH TXT version.bind` now returns `"unknown"`.

- [x] **Add response rate limiting (RRL)** — done 2026-07-19
      No `rate-limit {}` block was configured. Since `allow-query { any; }`
      is required for a public authoritative zone, RRL is the standard
      mitigation against the server being used for spoofed-source DNS
      reflection/amplification against third parties.
      Fixed: added `rate-limit { responses-per-second 10; window 5; }` to
      `named.conf.options`, validated with `named-checkconf`, and applied
      via `rndc reload`. Started with conservative defaults (10/s, 5s
      window) — revisit if legitimate high-volume resolvers get throttled.

- [x] **Add HSTS headers to all vhosts** — done 2026-07-19
      No vhost sent a `Strict-Transport-Security` header, leaving a window
      for on-path SSL-stripping on a user's very first plaintext request to
      any domain.
      Fixed: added `Header always set Strict-Transport-Security
      "max-age=63072000; includeSubDomains"` to all 6 domain `:443` blocks
      (`dyndns`, `jayfield.org`, `www`, `mail`, `web`, `cloud`) — skipped
      `default-ssl.conf` since it's the self-signed catch-all with no real
      trusted hostname for a browser to pin. Validated with
      `apache2ctl configtest`, applied via `systemctl reload apache2`,
      verified live with `curl -sI` against all 6 domains.

- [x] **`jayfield.org` had no `:80` vhost of its own** — done 2026-07-19
      Bare `http://jayfield.org` fell through to the `dyndns` default vhost
      and redirected to the wrong domain.
      Fixed: added a `:80` block to `jayfield.org.conf` redirecting to
      `https://jayfield.org/`, matching the `dyndns`/`mail` pattern.
      Verified: `curl -sI http://jayfield.org/` → `302` with
      `Location: https://jayfield.org/`.

- [x] **SSH: disable root login + password auth** — done 2026-07-19
      `PermitRootLogin yes` and `PasswordAuthentication yes` were both still
      enabled (the "known deviation" flagged in `SETUP.md` §1) — every
      credential-stuffing bot on the internet got a real login prompt to
      grind against.
      Fixed: `PermitRootLogin no`, `PasswordAuthentication no`,
      `PubkeyAuthentication yes` in `/etc/ssh/sshd_config` (backed up as
      `sshd_config.bak.20260719225249`). The `Match Group sftp` block's own
      `PasswordAuthentication yes` was left as-is — separate, already-chrooted,
      no-shell/no-forwarding accounts, not part of this risk. Validated with
      `sshd -t`, applied via `systemctl reload sshd`, verified live with
      `sshd -T` and a fresh key-only connection before considering it done.

- [x] **Portainer's `:9443` UI is unreachable in HSTS-enforcing browsers** — done 2026-07-20
      `https://alpha.jayfield.org:9443` threw
      `MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT` in Firefox with no click-through
      option ("keine Ausnahme kann hinzugefügt werden"). Root cause: the
      `includeSubDomains` HSTS policy sent by the real `alpha.jayfield.org`
      vhost (`Strict-Transport-Security` header, added in the item above)
      pins the *hostname* to HTTPS-with-a-trusted-cert on every port, but
      Portainer (`README.md` §Docker services) was still serving its stock
      self-signed cert directly on `9443`. Not a MITM — self-inflicted by
      combining HSTS with an unproxied self-signed service.
      Fixed: added a `portainer` A record to `db.jayfield.org` (SOA serial
      `2022032216` → `2022032217`); recreated the `portainer` container
      binding the HTTPS UI to `127.0.0.1:9443:9443` instead of publishing it
      (old container kept renamed until the new one was verified, then
      removed, matching the update procedure documented in `README.md`);
      added an Apache vhost for `portainer.jayfield.org`
      (`ProxyPass`/`ProxyPassReverse` to `https://127.0.0.1:9443/`,
      `SSLProxyEngine on`, `SSLProxyVerify none` + `SSLProxyCheckPeerCN off`/
      `SSLProxyCheckPeerName off` for the loopback hop to Portainer's
      self-signed cert) — same pattern as `web.jayfield.org`/
      `cloud.jayfield.org` (`SETUP.md` §4/§5); issued a real cert via
      `certbot --apache -d portainer.jayfield.org --redirect`; added the
      standard HSTS header. Left the `8000` edge-agent tunnel port
      published as-is — no evidence it's in use.
      Verified: `openssl s_client`/`curl` confirm the Let's Encrypt cert
      (not self-signed) and `Strict-Transport-Security` header on
      `https://portainer.jayfield.org`, `http://` redirects 301 to
      `https://`, and `ss -ltnp` shows `9443` bound to `127.0.0.1` only
      (via `docker-proxy`), so it's no longer reachable on the public
      interface at all — access is now exclusively via
      `https://portainer.jayfield.org`.
      One transient wrinkle during rollout: the very first few requests
      through the new proxy vhost returned a stray `401` with dyndns's
      Basic Auth realm ("Restricted Content") — looked alarming, but
      `LogLevel trace8` on the vhost showed Apache correctly SNI-matching
      and proxying to the new container with a clean backend `200`;
      resolved itself within seconds (looked like a stale pooled proxy
      connection surviving the container recreate) and was stable across
      repeated checks before the debug logging was removed again.

- [x] **fail2ban: `sshd` jail was evadable via low-and-slow attempts** — done 2026-07-19
      Default `findtime=30m`/`maxretry=3` never triggered for `47.76.192.176`,
      which had been hitting `root`/`jayfield` logins roughly every 33 minutes
      for most of a day — just outside the 30-minute window, so failures kept
      aging out before reaching 3 strikes. Confirmed via `fail2ban.log`
      (`Found`/`Ban` counts) that no other jail showed the same per-IP timing
      pattern; a separate, *distributed* single-shot scan on `apache-noscript`
      was noted but left open (see `README.md`).
      Fixed: `/etc/fail2ban/jail.d/sshd-findtime.local` overrides `sshd` to
      `findtime=1d`, `maxretry=4`, `bantime=1d`, kept in its own drop-in
      rather than editing the Debian-packaged jail file. Applied via
      `fail2ban-client reload sshd`; `47.76.192.176` was banned on its very
      next attempt after the change.

- [x] **fail2ban: `dovecot` jail had the same low-and-slow gap as `sshd`** — done 2026-07-25
      found while auditing which IPs were hitting webmail hard enough to
      deserve a ban but weren't in any currently-banned list. Same default
      `findtime=30m`/`maxretry=3` as `sshd`'s pre-fix config. At least 4 IPs
      (`47.95.196.60` 167 hits, `82.157.132.42` 144, `120.48.118.132` 76,
      `188.164.195.116` 57, `157.245.38.26` 34) ran a coordinated IMAP
      password spray against generic mailbox names (`admin@`, `test@`,
      `support@`, `sales@jayfield.org`) starting almost simultaneously around
      2026-07-21 22:40-23:38, each pacing itself at ~1 attempt/22-25min —
      just outside the 30-minute window, so none ever tripped 3 strikes.
      `120.48.118.132` was still active as late as 2026-07-25 13:32, so the
      campaign was ongoing, not historical. `postfix-sasl` shares the same
      default thresholds and mail stack but showed much lower per-IP volume
      in this pass — worth a follow-up check if it starts trending the same
      way.
      Fixed: `/etc/fail2ban/jail.d/dovecot-findtime.local`, same values as
      the `sshd` fix (`findtime=1d`, `maxretry=4`, `bantime=1d`). Applied via
      `fail2ban-client reload dovecot`; verified live with
      `fail2ban-client get dovecot findtime/maxretry/bantime` and confirmed
      the jail's existing ban state (`47.88.56.201`, 29 total bans) survived
      the reload.