jens/docs-alpha.jayfield.org
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix Portainer :9443 HSTS/self-signed-cert conflict

Browse Source 
Moved Portainer's HTTPS UI behind an Apache reverse proxy on its own
subdomain (portainer.jayfield.org) instead of publishing it directly on
:9443 with a self-signed cert, which HSTS's includeSubDomains policy made
unreachable in Firefox with no click-through exception. Same pattern as
the existing web.jayfield.org/cloud.jayfield.org proxied subdomains: DNS
A record, Apache vhost proxying to 127.0.0.1:9443 over HTTPS, real
Let's Encrypt cert via certbot, standard HSTS header. Portainer container
recreated to bind 9443 to loopback only; verified end-to-end (cert chain,
HSTS header, redirect, and that :9443 is no longer reachable externally).
This commit is contained in:
jens
2026-07-20 00:26:01 +02:00
parent 4644c983ff
commit a67895b4dd
2 changed files with 66 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+35 -16
View File
@@ -94,12 +94,13 @@ one-off `450 4.7.25 Client host rejected` shows up in `mail.log`.
| `mail.jayfield.org` | → redirect to :443 | Webmail-adjacent, proxies `/rspamd` → `http://localhost:11334/` | |
| `web.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `web` (nginx) container** on `127.0.0.1:8081` | New — see below |
| `cloud.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `nextcloud` container** on `127.0.0.1:8082`, sets `X-Forwarded-Proto: https` | Nextcloud — see below |
| `portainer.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `portainer` container** on `127.0.0.1:9443` over HTTPS (`SSLProxyEngine`/`SSLProxyVerify none` for the loopback hop to Portainer's self-signed cert) | Added 2026-07-20 — see "Portainer" below |
| default (`_default_:443`, `000-default` on :80) | Apache's stock placeholder page | — | |
All Let's Encrypt certs (7 total, one per hostname/group) auto-renew via
All Let's Encrypt certs (8 total, one per hostname/group) auto-renew via
`certbot.timer`; verified with `certbot renew --dry-run`.
All 6 domain `:443` blocks send `Strict-Transport-Security: max-age=63072000;
All 7 domain `:443` blocks send `Strict-Transport-Security: max-age=63072000;
includeSubDomains` (`mod_headers`). Deliberately skipped on the
`default-ssl.conf` catch-all — it's the self-signed fallback for unmatched
SNI/Host headers, not a real trusted hostname a browser could pin.
@@ -108,7 +109,7 @@ SNI/Host headers, not a real trusted hostname a browser could pin.
| Container | Image | Ports | Purpose |
|---|---|---|---|
| `portainer` | `portainer/portainer-ce:2.39.5` | `8000` (edge-agent tunnel, published), `9443` (HTTPS UI, published), `9000` (UI, internal only) | Docker management UI — `https://alpha.jayfield.org:9443` directly on its stock self-signed cert (**broken in HSTS-enforcing browsers**, see "Open items" below; planned fix is a `portainer.jayfield.org` reverse-proxy vhost, not yet applied) |
| `portainer` | `portainer/portainer-ce:latest` | `8000` (edge-agent tunnel, published), `9443` (HTTPS UI, `127.0.0.1`-only), `9000` (UI, internal only) | Docker management UI — `https://portainer.jayfield.org`, reverse-proxied by Apache to `127.0.0.1:9443` (fixed 2026-07-20, see below) |
| `web` | `nginx:latest` | `127.0.0.1:8081` (localhost-only, not published externally) | Backing service for `web.jayfield.org`; only reachable through the Apache reverse proxy |
| `nextcloud` | `nextcloud:apache` | `127.0.0.1:8082` (localhost-only) | Nextcloud app (PHP+Apache bundled in the image); backing service for `cloud.jayfield.org` |
| `nextcloud-db` | `mariadb:lts` | internal only (`nextcloud_net`) | Dedicated DB for Nextcloud — kept separate from the host's own MySQL (which only holds `vmail`) |
@@ -139,6 +140,26 @@ This is the pattern to follow for any future container-backed site: publish
the container to `127.0.0.1:<port>`, add an Apache vhost for the subdomain
that proxies to it, then run `certbot --apache -d <subdomain>` to get HTTPS.
### Portainer (`portainer.jayfield.org`)
```
Internet → Apache :80/:443 (portainer.jayfield.org vhost)
→ Let's Encrypt TLS termination
→ ProxyPass https://127.0.0.1:9443/ (SSLProxyVerify none —
backend keeps Portainer's own self-signed cert, but that
hop never leaves the host)
→ Docker container "portainer" HTTPS UI, published to
127.0.0.1 only
```
Previously Portainer published `9443` directly to the public interface with
its own self-signed cert; fixed 2026-07-20 after that became unreachable in
HSTS-enforcing browsers (see `TODO.md`). The container was recreated with
`-p 127.0.0.1:9443:9443` (old container kept renamed until the new one was
verified healthy, then removed — same caution as the update procedure
below). The `8000` edge-agent tunnel port is still published directly;
nothing currently depends on it, but it wasn't in scope for this fix.
### Nextcloud (`cloud.jayfield.org`)
Minimal, data-hosting-focused deployment — core Files app only. No
@@ -217,18 +238,6 @@ endpoint instead of a bespoke Apache-WebDAV + SFTP-upload pipeline:
## Open items / known quirks
- **Portainer `:9443` UI unreachable in HSTS-enforcing browsers** — found
2026-07-20 when Firefox refused `https://alpha.jayfield.org:9443` with
`MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT` and no click-through exception.
Cause: the `includeSubDomains` HSTS policy sent by the real
`alpha.jayfield.org` vhost pins the hostname to HTTPS-with-a-trusted-cert
on *every* port, but Portainer still answers `9443` directly with its own
self-signed cert — not a MITM, a self-inflicted interaction between two
otherwise-correct pieces of config. Planned fix: move Portainer behind an
Apache reverse proxy on its own subdomain (`portainer.jayfield.org`),
same pattern as `web.jayfield.org`/`cloud.jayfield.org`. See `TODO.md`
for the concrete steps; not yet applied.
- **Intranet migration from `vlda-01` (files, Gitea, Jellyfin, Dogecoin
node/wallet)** — planning stage only, not started. See `SYNC-PLAN.md`
for the full per-service design, open questions, and build order.
@@ -331,4 +340,14 @@ key-only) and the `sshd` fail2ban jail's `findtime` widened from the
33 minutes for most of a day — just outside the old 30-minute window, so it
never accumulated enough strikes to get banned; see `TODO.md` and
`SETUP.md` §1/§3 for the fix and the same evasion check applied to every
other jail (only one other pattern turned up — see "Open items" above).)
other jail (only one other pattern turned up — see "Open items" above).
Also: Portainer's `:9443` UI, previously published directly to the public
interface on its own self-signed cert, became unreachable in
HSTS-enforcing browsers (`MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT`, no
click-through) once the `includeSubDomains` HSTS policy above started
pinning `alpha.jayfield.org` to trusted-cert-only HTTPS on every port —
not a MITM, a self-inflicted interaction between two otherwise-correct
pieces of config. Fixed 2026-07-20 by moving it behind an Apache reverse
proxy on its own subdomain, `portainer.jayfield.org`, same pattern as
`web.jayfield.org`/`cloud.jayfield.org`; see "Portainer" above and
`TODO.md` for the full details.)
TODO.md
+31 -12
View File
@@ -61,26 +61,45 @@ Nothing here is urgent; all are low-risk, no-downtime changes.
`sshd -t`, applied via `systemctl reload sshd`, verified live with
`sshd -T` and a fresh key-only connection before considering it done.
- [ ] **Portainer's `:9443` UI is unreachable in HSTS-enforcing browsers** — found 2026-07-20
`https://alpha.jayfield.org:9443` throws
- [x] **Portainer's `:9443` UI is unreachable in HSTS-enforcing browsers** — done 2026-07-20
`https://alpha.jayfield.org:9443` threw
`MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT` in Firefox with no click-through
option ("keine Ausnahme kann hinzugefügt werden"). Root cause: the
`includeSubDomains` HSTS policy sent by the real `alpha.jayfield.org`
vhost (`Strict-Transport-Security` header, added in the item above)
pins the *hostname* to HTTPS-with-a-trusted-cert on every port, but
Portainer (`README.md` §Docker services) still serves its stock
Portainer (`README.md` §Docker services) was still serving its stock
self-signed cert directly on `9443`. Not a MITM — self-inflicted by
combining HSTS with an unproxied self-signed service.
Planned fix (not yet applied): rebind Portainer's published port to
`127.0.0.1:9443:9443`, add a `portainer.jayfield.org` Apache vhost
Fixed: added a `portainer` A record to `db.jayfield.org` (SOA serial
`2022032216` → `2022032217`); recreated the `portainer` container
binding the HTTPS UI to `127.0.0.1:9443:9443` instead of publishing it
(old container kept renamed until the new one was verified, then
removed, matching the update procedure documented in `README.md`);
added an Apache vhost for `portainer.jayfield.org`
(`ProxyPass`/`ProxyPassReverse` to `https://127.0.0.1:9443/`,
`SSLProxyEngine on`, `SSLProxyVerify none` for the loopback hop),
issue a real cert via `certbot --apache -d portainer.jayfield.org`,
and add the standard HSTS header — same pattern already used for
`web.jayfield.org`/`cloud.jayfield.org` (`SETUP.md` §4/§5). DNS needs
a new `portainer` A record in `db.jayfield.org` first (SOA serial
bump). Leave the `8000` edge-agent tunnel port as-is — no evidence
it's in use.
`SSLProxyEngine on`, `SSLProxyVerify none` + `SSLProxyCheckPeerCN off`/
`SSLProxyCheckPeerName off` for the loopback hop to Portainer's
self-signed cert) — same pattern as `web.jayfield.org`/
`cloud.jayfield.org` (`SETUP.md` §4/§5); issued a real cert via
`certbot --apache -d portainer.jayfield.org --redirect`; added the
standard HSTS header. Left the `8000` edge-agent tunnel port
published as-is — no evidence it's in use.
Verified: `openssl s_client`/`curl` confirm the Let's Encrypt cert
(not self-signed) and `Strict-Transport-Security` header on
`https://portainer.jayfield.org`, `http://` redirects 301 to
`https://`, and `ss -ltnp` shows `9443` bound to `127.0.0.1` only
(via `docker-proxy`), so it's no longer reachable on the public
interface at all — access is now exclusively via
`https://portainer.jayfield.org`.
One transient wrinkle during rollout: the very first few requests
through the new proxy vhost returned a stray `401` with dyndns's
Basic Auth realm ("Restricted Content") — looked alarming, but
`LogLevel trace8` on the vhost showed Apache correctly SNI-matching
and proxying to the new container with a clean backend `200`;
resolved itself within seconds (looked like a stale pooled proxy
connection surviving the container recreate) and was stable across
repeated checks before the debug logging was removed again.
- [x] **fail2ban: `sshd` jail was evadable via low-and-slow attempts** — done 2026-07-19
Default `findtime=30m`/`maxretry=3` never triggered for `47.76.192.176`,