docs-alpha.jayfield.org/README.md

# alpha.jayfield.org — Web Services Overview

Host: `alpha.jayfield.org` (`89.58.8.149`), Ubuntu 22.04 LTS.
SSH: `ssh -p 10022 jens@alpha.jayfield.org`

All administration in this repo's notes is performed remotely against
this host over SSH (port `10022`) — there is no local checkout of any
service config; changes are made directly on the server and recorded
here afterward.

This host runs two layers of web services: native services managed by
system Apache, and containerized services managed by Docker (visible in
Portainer). Apache reverse-proxies into the Docker layer where needed.

## Native services (systemd / apt)

| Service | Role |
|---|---|
| `apache2` (2.4.52) | Terminates HTTP/HTTPS for all vhosts, reverse-proxies to backends (Docker containers, rspamd, Portainer) |
| `bind9` | Authoritative DNS for the `jayfield.org` zone (`ns1`/`ns2.jayfield.org` are both this host); also the box's own recursive resolver — see below |
| `docker` | Runs the containerized web services (see below) |
| `certbot` (apt, 1.21.0) | Let's Encrypt cert issuance/renewal, via `certbot.timer` (systemd) |
| Postfix (`master`) | SMTP, ports 25/587, on the public IP + localhost |
| Dovecot | IMAP/IMAPS (143/993) + ManageSieve (4190) |
| rspamd | Spam filtering for the mail stack, localhost-only (11332–11334), proxied at `mail.jayfield.org/rspamd` |
| redis-server | Localhost-only (6379); backs rspamd's Bayes classifier, history, and reputation caching |

Python: `python3.10.12` (fixed from a stalled `python3.8` leftover of an
incomplete 20.04→22.04 upgrade — this was breaking apt's certbot via a
`pytz`/`zoneinfo` incompatibility).

## System users

| User | uid | Home | Purpose |
|---|---|---|---|
| `jens` | 1001 | `/home/jens` | Primary sudo admin account — all administration in this repo is done as this user over SSH |
| `alex` | 1005 | `/home/alex` | Secondary, non-sudo personal account (added 2026-07-19) |

Both also exist as separate Nextcloud accounts (see "Password vault sync"
below) and as chrooted, SFTP-only accounts `ftp_jens`/`ftp_alex` under
`/var/sftp/` — the latter currently password-locked, not deleted (see
below).

## DNS resolver architecture

Fixed 2026-07-19: the box's own outbound DNS resolution (used by Postfix's
`postscreen` for DNSBL lookups, and by every other service) previously
forwarded through the hosting provider's shared resolvers
(`46.38.225.230`/`46.38.252.230`). Confirmed via a live test: GMX's own
mail servers (`212.227.15.15`/`.19`, verified via reverse DNS as
`mout.gmx.net`) were being **dropped by `postscreen` on a false positive**
— Spamhaus's DNSBL was returning `127.255.255.254`, its documented
rate-limit sentinel ("this querying resolver has exceeded our free-tier
volume"), not a real listing. The shared resolver was almost certainly
pooling query volume from other tenants of the same hosting provider,
pushing it over Spamhaus's threshold.

Fix:
- Removed the `forwarders {}` block from `named.conf.options` — `bind9`
  now does full iterative/recursive resolution itself (root hints →
  TLD → authoritative) for `trusted` clients (`localhost` + `localnets`)
  instead of relaying to the shared upstream.
- Repointed `/etc/systemd/resolved.conf`'s `DNS=` from the shared
  provider IPs to `127.0.0.1` (i.e. `bind9`), then `systemctl restart
  systemd-resolved`. The `127.0.0.53` stub listener and `/etc/resolv.conf`
  symlink were left untouched, so Docker's DNS-detection logic (which
  special-cases the stub) is unaffected.
- (An `unbound` install was tried first as a separate local resolver, but
  `bind9` already listens on `127.0.0.1:53`/`[::1]:53` — `listen-on-v6 {
  any; }` plus BIND's default of listening on all IPv4 interfaces — so
  they couldn't coexist on the same port. Reusing `bind9` for both roles
  was simpler than fighting that conflict.)

Verified: both GMX IPs now correctly resolve as **not listed**;
`postscreen` now `PASS`es them. A real end-to-end test email
("Claude email test #01") delivered successfully to both `jens@` and
`otto2022@jayfield.org` after the fix.

**Known caveat**: direct recursive resolution occasionally hits a single
slow/unresponsive server somewhere in a delegation chain, where a
well-provisioned forwarding resolver's warm cache usually wouldn't. Seen
once during testing — a transient PTR lookup timeout caused one retry to
soft-fail (`450`, not `550`) before a second retry succeeded seconds
later. Not expected to be a recurring problem, but worth knowing if a
one-off `450 4.7.25 Client host rejected` shows up in `mail.log`.

## Apache virtual hosts

| ServerName | :80 | :443 | Notes |
|---|---|---|---|
| `dyndns.jayfield.org` | → redirect to :443 | Basic Auth (`Restricted Content`), `DocumentRoot /var/www/dyndns` (PHP) | Also the effective **default vhost** for unmatched hostnames (e.g. bare `alpha.jayfield.org` falls through here) |
| `jayfield.org` | → redirect to :443 | Static site (`/var/www/html`) | Previously had no `:80` vhost of its own — fell through to the `dyndns` default and redirected to the wrong domain; fixed |
| `www.jayfield.org` | → redirect to :443 | Static site (`/var/www/html`) | Renewal fixed to use the `apache` certbot plugin (was `standalone`, which required stopping Apache) |
| `mail.jayfield.org` | → redirect to :443 | Webmail-adjacent, proxies `/rspamd` → `http://localhost:11334/` | |
| `web.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `web` (nginx) container** on `127.0.0.1:8081` | New — see below |
| `cloud.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `nextcloud` container** on `127.0.0.1:8082`, sets `X-Forwarded-Proto: https` | Nextcloud — see below |
| default (`_default_:443`, `000-default` on :80) | Apache's stock placeholder page | — | |

All Let's Encrypt certs (7 total, one per hostname/group) auto-renew via
`certbot.timer`; verified with `certbot renew --dry-run`.

All 6 domain `:443` blocks send `Strict-Transport-Security: max-age=63072000;
includeSubDomains` (`mod_headers`). Deliberately skipped on the
`default-ssl.conf` catch-all — it's the self-signed fallback for unmatched
SNI/Host headers, not a real trusted hostname a browser could pin.

## Docker services (visible in Portainer)

| Container | Image | Ports | Purpose |
|---|---|---|---|
| `portainer` | `portainer/portainer-ce:2.39.5` | `8000` (edge-agent tunnel, published), `9443` (HTTPS UI, published), `9000` (UI, internal only) | Docker management UI — `https://alpha.jayfield.org:9443` |
| `web` | `nginx:latest` | `127.0.0.1:8081` (localhost-only, not published externally) | Backing service for `web.jayfield.org`; only reachable through the Apache reverse proxy |
| `nextcloud` | `nextcloud:apache` | `127.0.0.1:8082` (localhost-only) | Nextcloud app (PHP+Apache bundled in the image); backing service for `cloud.jayfield.org` |
| `nextcloud-db` | `mariadb:lts` | internal only (`nextcloud_net`) | Dedicated DB for Nextcloud — kept separate from the host's own MySQL (which only holds `vmail`) |
| `nextcloud-redis` | `redis:alpine` | internal only (`nextcloud_net`) | File locking + distributed cache + PHP session storage for Nextcloud. Dedicated container rather than reusing the host's native `redis-server`, since that instance is bound to `127.0.0.1`/`::1` only with no auth — reusing it would have meant loosening a shared service's network exposure just to save one small container |

Data persists in the `portainer_data` named volume, independent of the
container itself — updating Portainer means recreating the container
against a freshly pulled image while keeping this volume and the
`/var/run/docker.sock` bind mount; the old container is kept renamed
(not deleted) until the new one is verified healthy.

Two early-test containers (`my-apache-app`, a stopped `httpd`, and
`clever_jepsen`, a stale `hello-world` container from initial Docker
setup) have been removed as cleanup.

### `web.jayfield.org` request flow

```
Internet → Apache :80/:443 (web.jayfield.org vhost)
             → Let's Encrypt TLS termination
             → ProxyPass http://127.0.0.1:8081/
                 → Docker container "web" (nginx) on the docker0 bridge
```

The container is intentionally bound to `127.0.0.1` only, so it cannot be
reached directly on port 8081 from the internet — Apache is the only path in.
This is the pattern to follow for any future container-backed site: publish
the container to `127.0.0.1:<port>`, add an Apache vhost for the subdomain
that proxies to it, then run `certbot --apache -d <subdomain>` to get HTTPS.

### Nextcloud (`cloud.jayfield.org`)

Minimal, data-hosting-focused deployment — core Files app only. No
Collabora/OnlyOffice, Talk, full-text search, external storage backends,
or LDAP/SSO; those can be layered on later without re-architecting.

```
Internet → Apache :80/:443 (cloud.jayfield.org vhost)
             → Let's Encrypt TLS termination, X-Forwarded-Proto: https
             → ProxyPass http://127.0.0.1:8082/
                 → container "nextcloud" (nextcloud:apache)
                     ↕ private network "nextcloud_net"
                 → container "nextcloud-db" (mariadb:lts)
                 → container "nextcloud-redis" (redis:alpine)
```

- **DNS**: `cloud` A record added to `/etc/bind/db.jayfield.org`
  (serial bumped 2022032213 → 2022032214).
- **Data volume**: bind-mounted at `/srv/nextcloud/data` on the host
  (owned by uid/gid `33`, matching `www-data` both on the host and
  inside the container) rather than an opaque named Docker volume — so
  it's directly inspectable/backupable with normal tools (`du`,
  `rsync`, `borg`).
- **App volume**: named Docker volume `nextcloud_html`, holds Nextcloud
  core + `config.php`; survives image upgrades.
- **DB volume**: named Docker volume `nextcloud_db`.
- **Auto-install**: done via the image's env-var-driven installer
  (`MYSQL_*`, `NEXTCLOUD_ADMIN_USER`/`NEXTCLOUD_ADMIN_PASSWORD`,
  `NEXTCLOUD_TRUSTED_DOMAINS`, `TRUSTED_PROXIES`,
  `OVERWRITEPROTOCOL=https`, `REDIS_HOST`) rather than leaving an
  unconfigured setup wizard reachable on the public internet.
- **Credentials**: DB root/user passwords and the initial admin
  password are in `/root/.nextcloud-credentials` on the server
  (`600 root:root`) — not committed anywhere, not in this repo.
- **Cron**: Nextcloud's background-job mode is set to `cron` (`occ
  background:cron`); a root crontab entry runs
  `docker exec -u www-data nextcloud php -f /var/www/html/cron.php`
  every 5 minutes. No sidecar cron container.
- **PHP limits**: image defaults are already generous for file hosting
  (512M `memory_limit`/`upload_max_filesize`/`post_max_size`, unlimited
  `max_execution_time`) — no tuning needed for this pass.

### Password vault sync (KeePass, via Nextcloud)

Each user's KeePass database now syncs through Nextcloud's own WebDAV
endpoint instead of a bespoke Apache-WebDAV + SFTP-upload pipeline:

| User | Nextcloud account | Vault file | WebDAV URL |
|---|---|---|---|
| `jens` | `jens` | `sicher_jens.kdbx` | `https://cloud.jayfield.org/remote.php/dav/files/jens/sicher_jens.kdbx` |
| `alex` | `alex` | `sicher_alex.kdbx` | `https://cloud.jayfield.org/remote.php/dav/files/alex/sicher_alex.kdbx` |

- Each `.kdbx` lives at the root of that user's Nextcloud Files storage
  (`/srv/nextcloud/data/<user>/files/`, `640 www-data:www-data`), picked
  up via `occ files:scan <user>`.
- Verified 2026-07-19 with a raw WebDAV `PROPFIND`/`GET`/`PUT`/`GET`
  round trip against the `alex` account — correct `Content-Length`,
  matching SHA-256 before and after a write. (KeePassXC has no native
  WebDAV client and there's no display on this host to drive a real
  KeePass session, so this HTTP-level check is the practical
  equivalent — it exercises the same verbs a WebDAV-aware sync client
  or plugin would use.)
- A `600`-permission backup copy of each vault also still lives in the
  respective user's home dir (`/home/jens/sicher_jens.kdbx`,
  `/home/alex/sicher_alex.kdbx`) on this host.
- **Superseded setup**: an Apache `mod_dav_fs` container (`docker-apache2-php` /
  `my-httpd` image, HTTP Basic Auth via `.htpasswd`) running on the admin
  workstation used to serve the vault directly, with files uploaded to
  this host via the chrooted `ftp_jens`/`ftp_alex` SFTP accounts under
  `/var/sftp/`. That local container/image has been removed (it ran with
  `--rm` and had no cron/systemd autostart, so nothing else depended on
  it). The `ftp_jens`/`ftp_alex` accounts remain on this host (chroot and
  files untouched) but are password-locked (`passwd -l`) since nothing
  uses them anymore — reversible with `passwd -u` if SFTP is ever needed
  again.

## Open items / known quirks

- **Intranet migration from `vlda-01` (files, Gitea, Jellyfin, Dogecoin
  node/wallet)** — planning stage only, not started. See `SYNC-PLAN.md`
  for the full per-service design, open questions, and build order.

- **PTR/EHLO mismatch (workaround applied, real fix still open)**: the
  PTR record for this host's IP (`89.58.8.149`) resolves to
  `alpha.jayfield.org`, but Postfix's `myhostname` was `mail.jayfield.org`
  (matching the MX record) — a mismatch some strict receivers penalize.
  2026-07-19: `myhostname` changed to `alpha.jayfield.org` to match the
  existing PTR (verified live: `EHLO` → `250-alpha.jayfield.org`); see
  SETUP.md's mail-stack section. The real fix is repointing the PTR to
  `mail.jayfield.org` and then switching `myhostname` back — not done
  yet, since PTR isn't controlled from this host's own DNS zone. Checked
  2026-07-19: the ISP's control panel has no self-service PTR/rDNS
  option at all, so this needs a support ticket to the ISP asking them
  to set the PTR for `89.58.8.149` to `mail.jayfield.org` (worth noting
  in the ticket that the forward record already resolves correctly —
  `mail.jayfield.org` → `89.58.8.149` — since some providers require
  that before they'll set rDNS). If the ISP can't or won't do it, the
  current `myhostname = alpha.jayfield.org` workaround is a fine
  permanent setup as-is, not just a stopgap — PTR and EHLO already
  agree, which is what actually matters for deliverability.
- **Distributed low-and-slow scan on `apache-noscript`**: found 2026-07-19
  while auditing fail2ban jails for the `sshd` timing-evasion pattern below
  — a cluster of ~20 IPs across three `/24`s (`91.230.225.0/24`,
  `185.192.71.0/24`, `185.92.25.0/24`) each probed exactly once, never
  repeating. No per-IP `findtime`/`maxretry` tuning can catch this by
  design — it needs subnet-level (CIDR) banning instead, which risks
  blocking legitimate traffic sharing that block (e.g. carrier NAT). Left
  as routine scan noise for now; revisit if it escalates.

- **No real nameserver redundancy**: 2026-07-19, `ns1.jayfield.org` and
  `ns2.jayfield.org` were registered as this domain's NS records at the
  registrar. Delegation checked and confirmed consistent — the `.org`
  registry (checked against two separate registry servers), this host's
  own authoritative zone, and both `ns1`/`ns2` themselves (queried
  directly, `aa` flag set, matching SOA serial `2022032214` at the time
  of that check — serial has since moved to `2022032216` from the
  SPF/DMARC/DKIM fix below, unrelated to NS/glue) all agree.
  But both names glue to the *same* IP (`89.58.8.149`, i.e. this same
  host) — RFC 2182 §3 recommends at least two nameservers not on the
  same network segment, and this setup has zero real redundancy: if
  `alpha` goes down, both "nameservers" disappear together. Tried
  unregistering `ns2` at the registrar since it wasn't adding real
  redundancy anyway — confirmed the ISP/registrar requires a minimum of
  two NS records at the delegation layer and rejected the change, so
  both stay registered as-is. Redundancy risk remains open; the only
  real fix is a second nameserver on genuinely separate infrastructure.

(Previously resolved: legacy `/portainer` proxy paths on the
default vhost and `jayfield.org`, stale DNS records `test`, `kacke`,
`pupsi`, `mausi`, `hase`, `super`, and a disabled `nextcloud.conf` vhost
plus its `a2dissite` marker file — a leftover from a Nextcloud install
that was disabled at some point but never actually deployed (no
`/var/www/nextcloud`, no package, no DB, no cert) — all cleaned up.
Also: `mnt-vlda-01.mount`'s recurring boot failure — root-caused to a
dormant WireGuard tunnel (`alpha_at_moon`, routing the same
`192.168.22.0/24` home-LAN range the CIFS target lived on) — resolved
2026-07-19 by removing all three together rather than reviving the
tunnel: `wireguard`/`wireguard-tools` purged, `nginx` +
`libnginx-mod-rtmp` purged (an unrelated, previously-uncatalogued RTMP
streaming service on `:1935`, found via the same diff pass, also unused
and removed), and `samba`/`cifs-utils` purged along with the `fstab`
entry and stored CIFS credentials. Configs backed up to
`/root/removed-configs-backup/` on the server before deletion, in case
any of the three are wanted again. Zero failed systemd units after.
Also: KeePass vault sync migrated 2026-07-19 from local Apache-WebDAV +
SFTP-upload to Nextcloud's built-in WebDAV — see "Password vault sync"
above. Also: SPF, DMARC, and both DKIM (`2021`/`2022._domainkey`) TXT
records in `db.jayfield.org` were silently broken — root cause was
**unquoted TXT rdata in the zone file**: BIND's zone parser treats an
unquoted `;` as a start-of-comment (so `v=DMARC1; p=reject;` was being
served as just `"v=DMARC1"`, and both DKIM keys as just `"v=DKIM1"` with
the `k=rsa; p=<key>` portion silently dropped — meaning DKIM signature
verification had been failing for all outbound mail), and an unquoted
multi-token SPF line was being split into multiple concatenated
character-strings instead of one, merging in an unrelated
`google-site-verification=` token and invalidating the SPF syntax.
Fixed 2026-07-19 by quoting every TXT value (splitting the long DKIM
keys across two `<=255`-byte quoted strings each, since a single
character-string is capped at 255 bytes — see `named-checkzone`'s
"text string too long" if this is missed) and separating
`google-site-verification` into its own TXT record. SOA serial bumped
`2022032214` → `2022032216`. Verified directly against the authoritative
server (`dig ... @89.58.8.149`) post-reload; public resolvers catch up
as the old records' TTLs (up to 3600s) expire.
Also 2026-07-19: audited for phpMyAdmin/Adminer/webmin/cPanel/Plesk/
ISPConfig/phpPgAdmin residue (dpkg logs, `rc`-state packages, filesystem-wide
filename search, Apache configs, MySQL users/DBs, crontabs, Docker
images/volumes) — clean; the only name hits were stock `fail2ban`/`augeas`
package files, not evidence of a prior install. Two stray public files
(`ShowKeyPlus_x64.zip`, `MediaCreationTool1909.exe` — unrelated pirated
Windows tooling, plus an empty `test` file) were found sitting in
`/var/www/html/pub`, publicly served off the default vhost; moved to
`/home/jens/pub` and the now-empty `pub/` directory removed from the web
root. Also: SSH hardened (`PermitRootLogin no`, `PasswordAuthentication no`,
key-only) and the `sshd` fail2ban jail's `findtime` widened from the
30-minute default to 24h (`maxretry` 3→4, `bantime` 4h→24h) after
`47.76.192.176` was found grinding root/`jayfield` SSH logins roughly every
33 minutes for most of a day — just outside the old 30-minute window, so it
never accumulated enough strikes to get banned; see `TODO.md` and
`SETUP.md` §1/§3 for the fix and the same evasion check applied to every
other jail (only one other pattern turned up — see "Open items" above).)