Fix Portainer :9443 HSTS/self-signed-cert conflict

Moved Portainer's HTTPS UI behind an Apache reverse proxy on its own
subdomain (portainer.jayfield.org) instead of publishing it directly on
:9443 with a self-signed cert, which HSTS's includeSubDomains policy made
unreachable in Firefox with no click-through exception. Same pattern as
the existing web.jayfield.org/cloud.jayfield.org proxied subdomains: DNS
A record, Apache vhost proxying to 127.0.0.1:9443 over HTTPS, real
Let's Encrypt cert via certbot, standard HSTS header. Portainer container
recreated to bind 9443 to loopback only; verified end-to-end (cert chain,
HSTS header, redirect, and that :9443 is no longer reachable externally).

README.md

@@ -94,12 +94,13 @@ one-off `450 4.7.25 Client host rejected` shows up in `mail.log`.
 | `mail.jayfield.org` | → redirect to :443 | Webmail-adjacent, proxies `/rspamd` → `http://localhost:11334/` | |
 | `web.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `web` (nginx) container** on `127.0.0.1:8081` | New — see below |
 | `cloud.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `nextcloud` container** on `127.0.0.1:8082`, sets `X-Forwarded-Proto: https` | Nextcloud — see below |
+| `portainer.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `portainer` container** on `127.0.0.1:9443` over HTTPS (`SSLProxyEngine`/`SSLProxyVerify none` for the loopback hop to Portainer's self-signed cert) | Added 2026-07-20 — see "Portainer" below |
 | default (`_default_:443`, `000-default` on :80) | Apache's stock placeholder page | — | |
 
-All Let's Encrypt certs (7 total, one per hostname/group) auto-renew via
+All Let's Encrypt certs (8 total, one per hostname/group) auto-renew via
 `certbot.timer`; verified with `certbot renew --dry-run`.
 
-All 6 domain `:443` blocks send `Strict-Transport-Security: max-age=63072000;
+All 7 domain `:443` blocks send `Strict-Transport-Security: max-age=63072000;
 includeSubDomains` (`mod_headers`). Deliberately skipped on the
 `default-ssl.conf` catch-all — it's the self-signed fallback for unmatched
 SNI/Host headers, not a real trusted hostname a browser could pin.
@@ -108,7 +109,7 @@ SNI/Host headers, not a real trusted hostname a browser could pin.
 
 | Container | Image | Ports | Purpose |
 |---|---|---|---|
-| `portainer` | `portainer/portainer-ce:2.39.5` | `8000` (edge-agent tunnel, published), `9443` (HTTPS UI, published), `9000` (UI, internal only) | Docker management UI — `https://alpha.jayfield.org:9443` directly on its stock self-signed cert (**broken in HSTS-enforcing browsers**, see "Open items" below; planned fix is a `portainer.jayfield.org` reverse-proxy vhost, not yet applied) |
+| `portainer` | `portainer/portainer-ce:latest` | `8000` (edge-agent tunnel, published), `9443` (HTTPS UI, `127.0.0.1`-only), `9000` (UI, internal only) | Docker management UI — `https://portainer.jayfield.org`, reverse-proxied by Apache to `127.0.0.1:9443` (fixed 2026-07-20, see below) |
 | `web` | `nginx:latest` | `127.0.0.1:8081` (localhost-only, not published externally) | Backing service for `web.jayfield.org`; only reachable through the Apache reverse proxy |
 | `nextcloud` | `nextcloud:apache` | `127.0.0.1:8082` (localhost-only) | Nextcloud app (PHP+Apache bundled in the image); backing service for `cloud.jayfield.org` |
 | `nextcloud-db` | `mariadb:lts` | internal only (`nextcloud_net`) | Dedicated DB for Nextcloud — kept separate from the host's own MySQL (which only holds `vmail`) |
@@ -139,6 +140,26 @@ This is the pattern to follow for any future container-backed site: publish
 the container to `127.0.0.1:<port>`, add an Apache vhost for the subdomain
 that proxies to it, then run `certbot --apache -d <subdomain>` to get HTTPS.
 
+### Portainer (`portainer.jayfield.org`)
+
+```
+Internet → Apache :80/:443 (portainer.jayfield.org vhost)
+             → Let's Encrypt TLS termination
+             → ProxyPass https://127.0.0.1:9443/ (SSLProxyVerify none —
+                 backend keeps Portainer's own self-signed cert, but that
+                 hop never leaves the host)
+                 → Docker container "portainer" HTTPS UI, published to
+                   127.0.0.1 only
+```
+
+Previously Portainer published `9443` directly to the public interface with
+its own self-signed cert; fixed 2026-07-20 after that became unreachable in
+HSTS-enforcing browsers (see `TODO.md`). The container was recreated with
+`-p 127.0.0.1:9443:9443` (old container kept renamed until the new one was
+verified healthy, then removed — same caution as the update procedure
+below). The `8000` edge-agent tunnel port is still published directly;
+nothing currently depends on it, but it wasn't in scope for this fix.
+
 ### Nextcloud (`cloud.jayfield.org`)
 
 Minimal, data-hosting-focused deployment — core Files app only. No
@@ -217,18 +238,6 @@ endpoint instead of a bespoke Apache-WebDAV + SFTP-upload pipeline:
 
 ## Open items / known quirks
 
-- **Portainer `:9443` UI unreachable in HSTS-enforcing browsers** — found
-  2026-07-20 when Firefox refused `https://alpha.jayfield.org:9443` with
-  `MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT` and no click-through exception.
-  Cause: the `includeSubDomains` HSTS policy sent by the real
-  `alpha.jayfield.org` vhost pins the hostname to HTTPS-with-a-trusted-cert
-  on *every* port, but Portainer still answers `9443` directly with its own
-  self-signed cert — not a MITM, a self-inflicted interaction between two
-  otherwise-correct pieces of config. Planned fix: move Portainer behind an
-  Apache reverse proxy on its own subdomain (`portainer.jayfield.org`),
-  same pattern as `web.jayfield.org`/`cloud.jayfield.org`. See `TODO.md`
-  for the concrete steps; not yet applied.
-
 - **Intranet migration from `vlda-01` (files, Gitea, Jellyfin, Dogecoin
   node/wallet)** — planning stage only, not started. See `SYNC-PLAN.md`
   for the full per-service design, open questions, and build order.
@@ -331,4 +340,14 @@ key-only) and the `sshd` fail2ban jail's `findtime` widened from the
 33 minutes for most of a day — just outside the old 30-minute window, so it
 never accumulated enough strikes to get banned; see `TODO.md` and
 `SETUP.md` §1/§3 for the fix and the same evasion check applied to every
-other jail (only one other pattern turned up — see "Open items" above).)
+other jail (only one other pattern turned up — see "Open items" above).
+Also: Portainer's `:9443` UI, previously published directly to the public
+interface on its own self-signed cert, became unreachable in
+HSTS-enforcing browsers (`MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT`, no
+click-through) once the `includeSubDomains` HSTS policy above started
+pinning `alpha.jayfield.org` to trusted-cert-only HTTPS on every port —
+not a MITM, a self-inflicted interaction between two otherwise-correct
+pieces of config. Fixed 2026-07-20 by moving it behind an Apache reverse
+proxy on its own subdomain, `portainer.jayfield.org`, same pattern as
+`web.jayfield.org`/`cloud.jayfield.org`; see "Portainer" above and
+`TODO.md` for the full details.)

TODO.md

@@ -61,26 +61,45 @@ Nothing here is urgent; all are low-risk, no-downtime changes.
       `sshd -t`, applied via `systemctl reload sshd`, verified live with
       `sshd -T` and a fresh key-only connection before considering it done.
 
-- [ ] **Portainer's `:9443` UI is unreachable in HSTS-enforcing browsers** — found 2026-07-20
+- [x] **Portainer's `:9443` UI is unreachable in HSTS-enforcing browsers** — done 2026-07-20
-      `https://alpha.jayfield.org:9443` throws
+      `https://alpha.jayfield.org:9443` threw
       `MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT` in Firefox with no click-through
       option ("keine Ausnahme kann hinzugefügt werden"). Root cause: the
       `includeSubDomains` HSTS policy sent by the real `alpha.jayfield.org`
       vhost (`Strict-Transport-Security` header, added in the item above)
       pins the *hostname* to HTTPS-with-a-trusted-cert on every port, but
-      Portainer (`README.md` §Docker services) still serves its stock
+      Portainer (`README.md` §Docker services) was still serving its stock
       self-signed cert directly on `9443`. Not a MITM — self-inflicted by
       combining HSTS with an unproxied self-signed service.
-      Planned fix (not yet applied): rebind Portainer's published port to
+      Fixed: added a `portainer` A record to `db.jayfield.org` (SOA serial
-      `127.0.0.1:9443:9443`, add a `portainer.jayfield.org` Apache vhost
+      `2022032216` → `2022032217`); recreated the `portainer` container
+      binding the HTTPS UI to `127.0.0.1:9443:9443` instead of publishing it
+      (old container kept renamed until the new one was verified, then
+      removed, matching the update procedure documented in `README.md`);
+      added an Apache vhost for `portainer.jayfield.org`
       (`ProxyPass`/`ProxyPassReverse` to `https://127.0.0.1:9443/`,
-      `SSLProxyEngine on`, `SSLProxyVerify none` for the loopback hop),
+      `SSLProxyEngine on`, `SSLProxyVerify none` + `SSLProxyCheckPeerCN off`/
-      issue a real cert via `certbot --apache -d portainer.jayfield.org`,
+      `SSLProxyCheckPeerName off` for the loopback hop to Portainer's
-      and add the standard HSTS header — same pattern already used for
+      self-signed cert) — same pattern as `web.jayfield.org`/
-      `web.jayfield.org`/`cloud.jayfield.org` (`SETUP.md` §4/§5). DNS needs
+      `cloud.jayfield.org` (`SETUP.md` §4/§5); issued a real cert via
-      a new `portainer` A record in `db.jayfield.org` first (SOA serial
+      `certbot --apache -d portainer.jayfield.org --redirect`; added the
-      bump). Leave the `8000` edge-agent tunnel port as-is — no evidence
+      standard HSTS header. Left the `8000` edge-agent tunnel port
-      it's in use.
+      published as-is — no evidence it's in use.
+      Verified: `openssl s_client`/`curl` confirm the Let's Encrypt cert
+      (not self-signed) and `Strict-Transport-Security` header on
+      `https://portainer.jayfield.org`, `http://` redirects 301 to
+      `https://`, and `ss -ltnp` shows `9443` bound to `127.0.0.1` only
+      (via `docker-proxy`), so it's no longer reachable on the public
+      interface at all — access is now exclusively via
+      `https://portainer.jayfield.org`.
+      One transient wrinkle during rollout: the very first few requests
+      through the new proxy vhost returned a stray `401` with dyndns's
+      Basic Auth realm ("Restricted Content") — looked alarming, but
+      `LogLevel trace8` on the vhost showed Apache correctly SNI-matching
+      and proxying to the new container with a clean backend `200`;
+      resolved itself within seconds (looked like a stale pooled proxy
+      connection surviving the container recreate) and was stable across
+      repeated checks before the debug logging was removed again.
 
 - [x] **fail2ban: `sshd` jail was evadable via low-and-slow attempts** — done 2026-07-19
       Default `findtime=30m`/`maxretry=3` never triggered for `47.76.192.176`,