docs-alpha.jayfield.org/README.md

# alpha.jayfield.org — Web Services Overview

Host: `alpha.jayfield.org` (`89.58.8.149`), Ubuntu 24.04 LTS (upgraded from
22.04 on 2026-07-26 — see `TODO.md` for the full rollout).
SSH: `ssh -p 10022 jens@alpha.jayfield.org`

All administration in this repo's notes is performed remotely against
this host over SSH (port `10022`) — there is no local checkout of any
service config; changes are made directly on the server and recorded
here afterward.

This host runs two layers of web services: native services managed by
system Apache, and containerized services managed by Docker (visible in
Portainer). Apache reverse-proxies into the Docker layer where needed.

## Domain namespace

`jayfield.org` covers two distinct things under one name:
- The root domain and its public subdomains (`alpha`, `www`, `mail`,
  `dyndns`, `cloud`, `web`, `portainer`, `git`, ...) all point at this
  host, `89.58.8.149` — everything this doc is about.
- `home.jayfield.org` is a separate, deliberately private namespace for
  the home intranet (`vlda-01` and other home-LAN devices, see
  `SYNC-PLAN.md`) — not part of this host's authoritative zone, not
  publicly resolvable, kept off public DNS by design.

## Native services (systemd / apt)

| Service | Role |
|---|---|
| `apache2` (2.4.58) | Terminates HTTP/HTTPS for all vhosts, reverse-proxies to backends (Docker containers, rspamd, Portainer) |
| `bind9` | Authoritative DNS for the `jayfield.org` zone (`ns1`/`ns2.jayfield.org` are both this host); also the box's own recursive resolver — see below |
| `docker` | Runs the containerized web services (see below) |
| `certbot` (apt, 2.9.0) | Let's Encrypt cert issuance/renewal, via `certbot.timer` (systemd) |
| Postfix (`master`) | SMTP, ports 25/587, on the public IP + localhost |
| Dovecot | IMAP/IMAPS (143/993) + ManageSieve (4190) |
| rspamd (4.1.3, `rspamd.com` PPA — not the OS's own older native package) | Spam filtering for the mail stack, localhost-only (11332–11334), proxied at `mail.jayfield.org/rspamd` |
| redis-server | Localhost-only (6379); backs rspamd's Bayes classifier, history, and reputation caching |
| `mariadb` (10.11.14, noble-native) | Holds `vmail` (Postfix/Dovecot virtual accounts) — see `TODO.md` for the `10.3`→`10.6`→`10.11` migration history |

Python: `python3.12.3` (noble's default). Previously `python3.10.12` on
22.04, itself fixed from a stalled `python3.8` leftover of the original
incomplete 20.04→22.04 upgrade (breaking apt's certbot via a
`pytz`/`zoneinfo` incompatibility) — that whole incomplete-upgrade saga is
now fully resolved, see `TODO.md`.

## Post-boot sanity check

Checked into this repo: [`scripts/alpha/sanity-check.sh`](scripts/alpha/sanity-check.sh) /
[`scripts/alpha/sanity-check.service`](scripts/alpha/sanity-check.service)
(Ubuntu 24.04 — see `scripts/alpha/README.md`).

`/usr/local/sbin/sanity-check.sh` (added 2026-07-26, after the 24.04
upgrade turned up two real post-upgrade breakages — see `TODO.md`) checks
that the whole stack is actually healthy, not just "systemd says active":
package/dpkg consistency, disk space, SSH hardening
(`PermitRootLogin`/`PasswordAuthentication`), BIND + DNSSEC validation +
the dyndns zone, the full mail stack including a live `rspamc symbols`
scan (catches a phantom-active rspamd, which is exactly what happened
during the 24.04 upgrade) and whether anything is actually listening on
Postfix's configured milter port, MariaDB/`vmail` data + Postfix's SQL
lookups, fail2ban's jail count and the custom `findtime` hardening, all 7
public vhosts by expected HTTP status code (plus a genuine Gitea API
check, not just a static-page response), and all 6 Docker containers.

Runs automatically once per boot via `sanity-check.service` (a oneshot
unit, `systemctl enable`d, `After=` all the relevant services plus a 10s
settle delay before running) — covers OS upgrades, kernel updates, and
any other change that ends in a reboot, which in practice has been all of
them so far. Also safe to run manually after a change that *doesn't*
reboot (a plain `apt upgrade`, a config edit): `sudo
/usr/local/sbin/sanity-check.sh`. Each run logs to
`/var/log/sanity-check/<timestamp>.log` and emails a pass/warn/fail
summary to `jens@jayfield.org` via the local mail stack (no external
dependency). Exits non-zero if anything actually failed (warnings, like
routine pending package upgrades, don't fail the run).

## Weekly attacker summary

Checked into this repo: [`scripts/alpha/attacker-check.py`](scripts/alpha/attacker-check.py) /
[`scripts/alpha/cron.d/attacker-check`](scripts/alpha/cron.d/attacker-check)
(Ubuntu 24.04 — see `scripts/alpha/README.md`).

`/usr/local/sbin/attacker-check.py` (added 2026-07-26) parses
`fail2ban.log` (current + recent rotations, gzipped or not — reads enough
history to cover a full week regardless of exactly when logrotate's daily
midnight run lands relative to the cron job) for the past 7 days and
turns it into two append-only CSVs for longitudinal analysis, plus a
human-readable weekly summary emailed to `root@jayfield.org` and saved
as a dated report file:

- `/var/log/attacker-check/per_ip_weekly.csv` — one row per (jail, IP)
  per week: event count, first/last seen, `/24` subnet, and interval
  statistics (mean, stdev) between repeat hits.
- `/var/log/attacker-check/per_jail_weekly.csv` — weekly rollup per jail:
  total events, distinct IPs, distinct `/24`s, new bans.
- `/var/log/attacker-check/reports/<week-ending-date>.txt` — the full
  human-readable report for that run, one file per date.

`root@jayfield.org` didn't resolve to anything before this (not in
`/etc/aliases`, not in the `vmail` aliases table) — added it as a new
`vmail.aliases` row → `jens@jayfield.org`, matching the existing
`postmaster`/`hostmaster`/`webmaster`/`wlan` pattern.

Two detection heuristics run on top of the raw data, both modeled
directly on real campaigns found earlier this engagement (see `TODO.md`):
- **Distributed single-shot scans**: flags any `/24` subnet with ≥5
  distinct IPs averaging ≤3 events each against the same jail — the
  shape of the `apache-noscript` campaign found 2026-07-19.
- **Regular-timing evasion**: flags any IP with ≥3 repeat hits on the
  same jail whose inter-hit timing has a coefficient of variation under
  15% — the shape of the `sshd`/`dovecot` low-and-slow campaigns found
  and fixed 2026-07-19/2026-07-25 (attackers spacing requests just past
  a jail's `findtime` window). A first live run against this week's real
  logs correctly re-identified the same dovecot campaign IPs from that
  earlier manual investigation, with matching ~22–28 minute intervals.

Runs weekly via `/etc/cron.d/attacker-check`, Sunday 06:00 (clear of
logrotate's daily midnight timer). Safe to also run manually at any
time: `sudo /usr/local/sbin/attacker-check.py`.

## System users

| User | uid | Home | Purpose |
|---|---|---|---|
| `jens` | 1001 | `/home/jens` | Primary sudo admin account — all administration in this repo is done as this user over SSH |
| `alex` | 1005 | `/home/alex` | Secondary, non-sudo personal account (added 2026-07-19) |

Both also exist as separate Nextcloud accounts (see "Password vault sync"
below) and as chrooted, SFTP-only accounts `ftp_jens`/`ftp_alex` under
`/var/sftp/` — the latter currently password-locked, not deleted (see
below).

## DNS resolver architecture

Fixed 2026-07-19: the box's own outbound DNS resolution (used by Postfix's
`postscreen` for DNSBL lookups, and by every other service) previously
forwarded through the hosting provider's shared resolvers
(`46.38.225.230`/`46.38.252.230`). Confirmed via a live test: GMX's own
mail servers (`212.227.15.15`/`.19`, verified via reverse DNS as
`mout.gmx.net`) were being **dropped by `postscreen` on a false positive**
— Spamhaus's DNSBL was returning `127.255.255.254`, its documented
rate-limit sentinel ("this querying resolver has exceeded our free-tier
volume"), not a real listing. The shared resolver was almost certainly
pooling query volume from other tenants of the same hosting provider,
pushing it over Spamhaus's threshold.

Fix:
- Removed the `forwarders {}` block from `named.conf.options` — `bind9`
  now does full iterative/recursive resolution itself (root hints →
  TLD → authoritative) for `trusted` clients (`localhost` + `localnets`)
  instead of relaying to the shared upstream.
- Repointed `/etc/systemd/resolved.conf`'s `DNS=` from the shared
  provider IPs to `127.0.0.1` (i.e. `bind9`), then `systemctl restart
  systemd-resolved`. The `127.0.0.53` stub listener and `/etc/resolv.conf`
  symlink were left untouched, so Docker's DNS-detection logic (which
  special-cases the stub) is unaffected.
- (An `unbound` install was tried first as a separate local resolver, but
  `bind9` already listens on `127.0.0.1:53`/`[::1]:53` — `listen-on-v6 {
  any; }` plus BIND's default of listening on all IPv4 interfaces — so
  they couldn't coexist on the same port. Reusing `bind9` for both roles
  was simpler than fighting that conflict.)

Verified: both GMX IPs now correctly resolve as **not listed**;
`postscreen` now `PASS`es them. A real end-to-end test email
("Claude email test #01") delivered successfully to both `jens@` and
`otto2022@jayfield.org` after the fix.

**Known caveat**: direct recursive resolution occasionally hits a single
slow/unresponsive server somewhere in a delegation chain, where a
well-provisioned forwarding resolver's warm cache usually wouldn't. Seen
once during testing — a transient PTR lookup timeout caused one retry to
soft-fail (`450`, not `550`) before a second retry succeeded seconds
later. Not expected to be a recurring problem, but worth knowing if a
one-off `450 4.7.25 Client host rejected` shows up in `mail.log`.

## Apache virtual hosts

| ServerName | :80 | :443 | Notes |
|---|---|---|---|
| `dyndns.jayfield.org` | → redirect to :443 | Basic Auth (`Restricted Content`), `DocumentRoot /var/www/dyndns` (PHP) | Also the effective **default vhost** for unmatched hostnames (e.g. bare `alpha.jayfield.org` falls through here) |
| `jayfield.org` | → redirect to :443 | Static site (`/var/www/html`) | Previously had no `:80` vhost of its own — fell through to the `dyndns` default and redirected to the wrong domain; fixed |
| `www.jayfield.org` | → redirect to :443 | Static site (`/var/www/html`) | Renewal fixed to use the `apache` certbot plugin (was `standalone`, which required stopping Apache) |
| `mail.jayfield.org` | → redirect to :443 | Webmail-adjacent, proxies `/rspamd` → `http://localhost:11334/` | |
| `web.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `web` (nginx) container** on `127.0.0.1:8081` | New — see below |
| `cloud.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `nextcloud` container** on `127.0.0.1:8082`, sets `X-Forwarded-Proto: https` | Nextcloud — see below |
| `portainer.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `portainer` container** on `127.0.0.1:9443` over HTTPS (`SSLProxyEngine`/`SSLProxyVerify none` for the loopback hop to Portainer's self-signed cert) | Added 2026-07-20 — see "Portainer" below |
| `git.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `gitea` container** on `127.0.0.1:8083`, sets `X-Forwarded-Proto: https` | Added 2026-07-26 — migrated from `vlda-01`, see "Gitea" below |
| default (`_default_:443`, `000-default` on :80) | Apache's stock placeholder page | — | |

All Let's Encrypt certs (9 total, one per hostname/group) auto-renew via
`certbot.timer`; verified with `certbot renew --dry-run`.

All 8 domain `:443` blocks send `Strict-Transport-Security: max-age=63072000;
includeSubDomains` (`mod_headers`). Deliberately skipped on the
`default-ssl.conf` catch-all — it's the self-signed fallback for unmatched
SNI/Host headers, not a real trusted hostname a browser could pin.

## Docker services (visible in Portainer)

| Container | Image | Ports | Purpose |
|---|---|---|---|
| `portainer` | `portainer/portainer-ce:latest` | `8000` (edge-agent tunnel, published), `9443` (HTTPS UI, `127.0.0.1`-only), `9000` (UI, internal only) | Docker management UI — `https://portainer.jayfield.org`, reverse-proxied by Apache to `127.0.0.1:9443` (fixed 2026-07-20, see below) |
| `web` | `nginx:latest` | `127.0.0.1:8081` (localhost-only, not published externally) | Backing service for `web.jayfield.org`; only reachable through the Apache reverse proxy |
| `nextcloud` | `nextcloud:apache` | `127.0.0.1:8082` (localhost-only) | Nextcloud app (PHP+Apache bundled in the image); backing service for `cloud.jayfield.org` |
| `nextcloud-db` | `mariadb:lts` | internal only (`nextcloud_net`) | Dedicated DB for Nextcloud — kept separate from the host's own MySQL (which only holds `vmail`) |
| `nextcloud-redis` | `redis:alpine` | internal only (`nextcloud_net`) | File locking + distributed cache + PHP session storage for Nextcloud. Dedicated container rather than reusing the host's native `redis-server`, since that instance is bound to `127.0.0.1`/`::1` only with no auth — reusing it would have meant loosening a shared service's network exposure just to save one small container |
| `gitea` | `gitea/gitea:1.27.0` | `127.0.0.1:8083` (localhost-only) | Git hosting, migrated from `vlda-01` 2026-07-26 — see "Gitea" below |

Data persists in the `portainer_data` named volume, independent of the
container itself — updating Portainer means recreating the container
against a freshly pulled image while keeping this volume and the
`/var/run/docker.sock` bind mount; the old container is kept renamed
(not deleted) until the new one is verified healthy.

Two early-test containers (`my-apache-app`, a stopped `httpd`, and
`clever_jepsen`, a stale `hello-world` container from initial Docker
setup) have been removed as cleanup.

### `web.jayfield.org` request flow

```
Internet → Apache :80/:443 (web.jayfield.org vhost)
             → Let's Encrypt TLS termination
             → ProxyPass http://127.0.0.1:8081/
                 → Docker container "web" (nginx) on the docker0 bridge
```

The container is intentionally bound to `127.0.0.1` only, so it cannot be
reached directly on port 8081 from the internet — Apache is the only path in.
This is the pattern to follow for any future container-backed site: publish
the container to `127.0.0.1:<port>`, add an Apache vhost for the subdomain
that proxies to it, then run `certbot --apache -d <subdomain>` to get HTTPS.

### Portainer (`portainer.jayfield.org`)

```
Internet → Apache :80/:443 (portainer.jayfield.org vhost)
             → Let's Encrypt TLS termination
             → ProxyPass https://127.0.0.1:9443/ (SSLProxyVerify none —
                 backend keeps Portainer's own self-signed cert, but that
                 hop never leaves the host)
                 → Docker container "portainer" HTTPS UI, published to
                   127.0.0.1 only
```

Previously Portainer published `9443` directly to the public interface with
its own self-signed cert; fixed 2026-07-20 after that became unreachable in
HSTS-enforcing browsers (see `TODO.md`). The container was recreated with
`-p 127.0.0.1:9443:9443` (old container kept renamed until the new one was
verified healthy, then removed — same caution as the update procedure
below). The `8000` edge-agent tunnel port is still published directly;
nothing currently depends on it, but it wasn't in scope for this fix.

### Nextcloud (`cloud.jayfield.org`)

Minimal, data-hosting-focused deployment — core Files app only. No
Collabora/OnlyOffice, Talk, full-text search, external storage backends,
or LDAP/SSO; those can be layered on later without re-architecting.

```
Internet → Apache :80/:443 (cloud.jayfield.org vhost)
             → Let's Encrypt TLS termination, X-Forwarded-Proto: https
             → ProxyPass http://127.0.0.1:8082/
                 → container "nextcloud" (nextcloud:apache)
                     ↕ private network "nextcloud_net"
                 → container "nextcloud-db" (mariadb:lts)
                 → container "nextcloud-redis" (redis:alpine)
```

- **DNS**: `cloud` A record added to `/etc/bind/db.jayfield.org`
  (serial bumped 2022032213 → 2022032214).
- **Data volume**: bind-mounted at `/srv/nextcloud/data` on the host
  (owned by uid/gid `33`, matching `www-data` both on the host and
  inside the container) rather than an opaque named Docker volume — so
  it's directly inspectable/backupable with normal tools (`du`,
  `rsync`, `borg`).
- **App volume**: named Docker volume `nextcloud_html`, holds Nextcloud
  core + `config.php`; survives image upgrades.
- **DB volume**: named Docker volume `nextcloud_db`.
- **Auto-install**: done via the image's env-var-driven installer
  (`MYSQL_*`, `NEXTCLOUD_ADMIN_USER`/`NEXTCLOUD_ADMIN_PASSWORD`,
  `NEXTCLOUD_TRUSTED_DOMAINS`, `TRUSTED_PROXIES`,
  `OVERWRITEPROTOCOL=https`, `REDIS_HOST`) rather than leaving an
  unconfigured setup wizard reachable on the public internet.
- **Credentials**: DB root/user passwords and the initial admin
  password are in `/root/.nextcloud-credentials` on the server
  (`600 root:root`) — not committed anywhere, not in this repo.
- **Cron**: Nextcloud's background-job mode is set to `cron` (`occ
  background:cron`); a root crontab entry runs
  `docker exec -u www-data nextcloud php -f /var/www/html/cron.php`
  every 5 minutes. No sidecar cron container.
- **PHP limits**: image defaults are already generous for file hosting
  (512M `memory_limit`/`upload_max_filesize`/`post_max_size`, unlimited
  `max_execution_time`) — no tuning needed for this pass.

### Gitea (`git.jayfield.org`)

Migrated 2026-07-26 from `vlda-01` (home intranet server) per
`SYNC-PLAN.md`'s build order — first item on that plan, "no open
questions." Full migration, not a sync: `vlda-01`'s Gitea container was
stopped, its entire `/data` volume (SQLite DB, all repos, LFS objects,
host SSH keys) copied straight across via a piped `ssh ... tar | ssh ...
tar` (no intermediate disk, ~4GB), and brought up on alpha as a permanent
container — same `web`/`cloud`/`portainer` pattern (publish to
`127.0.0.1:<port>`, Apache vhost, `certbot --apache`).

```
Internet → Apache :80/:443 (git.jayfield.org vhost)
             → Let's Encrypt TLS termination, X-Forwarded-Proto: https
             → ProxyPass http://127.0.0.1:8083/
                 → container "gitea" (gitea/gitea:1.27.0)
                     → SQLite DB + repos, bind-mounted at /srv/gitea/data
```

- **Version pinned, not `:latest`**: `gitea/gitea:1.27.0`, matching what
  was actually running on `vlda-01` — a version-compatible move first,
  upgrading is a separate later decision.
- **Data**: bind-mounted at `/srv/gitea/data` on the host (matching the
  `/srv/nextcloud/data` convention — directly inspectable/backupable),
  not an opaque named volume.
- **Ownership**: the source data was owned by uid/gid `1000` on
  `vlda-01`, but `1000` is already `vmail` on alpha (Dovecot's mailbox
  user) — remapped everything to a dedicated `1010:1010` (`chown -R`,
  container run with `-e USER_UID=1010 -e USER_GID=1010`) to avoid any
  collision with or confusion around the real `vmail` account.
- **SQLite, not the host's MariaDB**: `vlda-01`'s Gitea was already
  self-contained SQLite (`DB_TYPE = sqlite3`), so no DB export/import or
  extra container was needed — the DB file just came along inside `/data`.
- **`app.ini` updated for the new home**: `DOMAIN`/`SSH_DOMAIN`/`ROOT_URL`
  repointed from the old LAN IP (`192.168.22.90`) to `git.jayfield.org`;
  `DISABLE_SSH` set to `true` — the existing repo remotes already used
  HTTP(S), not `git@.../ssh://`, so git-over-SSH was dropped rather than
  opening yet another public port to preserve a mode nothing was actually
  using. HTTPS clone/push covers the same ground through the same Apache
  TLS termination every other service uses.
- **Reverse-proxy scheme awareness**: without `X-Forwarded-Proto: https`
  (added to the vhost, same as `cloud.jayfield.org`), Gitea's API
  correctly used `ROOT_URL` for some fields but generated `http://` (not
  `https://`) `clone_url`/`html_url` for repos — since the Apache→container
  hop itself is plain HTTP and Gitea otherwise has no way to know the
  original request arrived over TLS. Fixed and verified via the API.
- **Verified end-to-end**: all previously-existing repos present via the
  API (including this `docs-alpha.jayfield.org` repo), admin user (`jayfield`,
  migrated over from `vlda-01`) intact; a real `git clone` over HTTPS
  reproduced identical commit history to the working copy; this repo's own
  `origin` remote repointed to `https://git.jayfield.org/...` and a real
  `git push` succeeded against it.
- **`vlda-01`'s copy**: left stopped (container not removed, data not
  deleted) as a cold backup — reversible if ever needed; not decommissioned
  outright.
- **Admin user renamed `jayfield` → `jens` 2026-07-26** (password rotated at
  the same time), via Gitea's own web admin panel (Site Administration →
  User Accounts → Edit Account) rather than a direct DB/API edit — that's
  the one code path that correctly moves the on-disk repo storage
  directories and sets up a redirect together, rather than risking an
  inconsistent rename. Verified: new credentials authenticate
  (`GET /api/v1/user` → `"username":"jens"`), both the new canonical path
  (`/jens/docs-alpha.jayfield.org`) and the old one (`/jayfield/...`, via
  Gitea's automatic redirect) return `200`, and this repo's local `origin`
  remote and stored git credentials were updated to match.

### Gitea backup to `vlda-01` (`SYNC-PLAN.md` item 2b)

Checked into this repo, split by which host each half runs on (they're on
different OSes with different conventions — see each `README.md`):
- Alpha side (Ubuntu 24.04): [`scripts/alpha/gitea-backup.sh`](scripts/alpha/gitea-backup.sh) /
  [`scripts/alpha/cron.d/gitea-backup`](scripts/alpha/cron.d/gitea-backup)
  — see `scripts/alpha/README.md`.
- `vlda-01` side (Unraid 7.3.2): [`scripts/clients/vlda-01/`](scripts/clients/vlda-01/)
  — see `scripts/clients/vlda-01/README.md`, which also covers the
  Unraid-specific persistence gotchas (RAM-backed root, VFAT `/boot`,
  Unassigned Devices) that shaped how this had to be built.

Built 2026-07-26. A point-in-time backup, not a hot standby/live mirror —
now that alpha is the canonical live Gitea, this keeps `vlda-01`'s copy
from just going stale after the migration. Same network direction as
everything else in this project: alpha has no route into the home LAN,
so the pull is **initiated by `vlda-01`**, never alpha reaching in.

**Why `gitea dump`, not a raw `rsync` of `/srv/gitea/data`**: the DB is a
live SQLite file; copying it while Gitea is writing risks shipping a torn
copy. `gitea dump` produces one consistent archive (DB + repos + LFS +
config together) — also what Gitea's own restore procedure expects.

**On alpha** — `/usr/local/sbin/gitea-backup.sh`, `/etc/cron.d/gitea-backup`
(daily `02:30`, clear of logrotate's midnight timer):
- `docker exec -u git gitea gitea dump -f /data/backups/gitea-dump-<timestamp>.zip --skip-index`
  into `/srv/gitea/data/backups/` (inside the existing bind mount — already
  host-inspectable, no extra volume). Timestamped to the second, not just
  the date: `gitea dump` refuses to overwrite an existing filename, which
  a plain per-day name would hit on any manual re-run (found this the hard
  way testing it).
- Writes a `sha256sum`-format `.sha256` file alongside each dump, so
  `vlda-01` can verify integrity after transfer before trusting it enough
  to delete the alpha-side copy (see below).
- Applies a POSIX ACL (`setfacl -m u:giteabackup:r`) to each new archive
  **and its checksum file** so the pull account below can read them.
  **Has to happen explicitly per file** — a default ACL on the directory
  alone isn't enough: `gitea dump` creates each file `600`, and the
  resulting ACL mask neuters the inherited grant (`getfacl` shows
  `#effective:---` if this step is skipped).
- Prunes to the last 14 archives (dump + checksum pairs together) — a
  **safety net only**: in normal operation `vlda-01` deletes each dump
  right after a verified pull, well before this would ever trigger. Only
  matters if `vlda-01` is offline long enough for dumps to pile up
  unswept.

**Transport — a dedicated, restricted account on alpha**, not a shared or
personal credential:
- System account `giteabackup` (`/usr/sbin/nologin`), member of the
  existing `sftp` group — reuses the `Match Group sftp` /
  `ForceCommand internal-sftp` block already in `sshd_config` for
  `ftp_jens`/`ftp_alex`; no new sshd config needed.
- Chrooted to `/var/sftp/giteabackup/`, containing only a bind mount of
  `/srv/gitea/data/backups`. **Read-write** at the filesystem level (`rw`
  in `/etc/fstab` — needed so `vlda-01` can delete a dump after verifying
  it), but the *account's* actual privilege is narrower than that sounds:
  the directory-level ACL grants `giteabackup` `rwx` (needed for
  delete/unlink, which is a directory operation in POSIX), while each
  individual file's ACL only grants `r`. Verified the boundary actually
  holds: an SFTP `rm` on an existing dump succeeds, but an SFTP `put`
  attempting to overwrite one fails with "Permission denied" — it can
  delete a file wholesale, never tamper with one in place.
- Key-only auth: a dedicated ed25519 keypair generated on `vlda-01`
  (private key never leaves it), `authorized_keys` further restricted
  (`restrict,no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty`)
  on top of the chroot/`ForceCommand`.
- **Plain SFTP, not real `rsync`**: `ForceCommand internal-sftp` blocks a
  genuine `rsync --server` invocation from the client, so the transport
  is SFTP `get`/`rm`, not rsync.

**On `vlda-01`** — via the **User Scripts** plugin, not a raw
crontab/`/usr/local/sbin` script: Unraid's `/` and `/usr` are RAM-backed
(`rootfs`/`overlay`) and don't survive a reboot — only `/boot` (the actual
USB stick) does. The pull script lives at
`/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script`,
scheduled daily `03:30` via `schedule.json`/`customSchedule.cron`
(`update_cron` folds it into the live `/etc/cron.d/root` alongside
Unraid's own generated jobs) — the same plugin already used for this
box's other scheduled maintenance, no new persistence mechanism invented.
The SSH keypair needed no special handling: Unraid already auto-mirrors
`/root/.ssh` to `/boot/config/ssh/root/` on its own.
- Lists what's actually on alpha via SFTP, fetches only filenames not
  already present locally (dump + its `.sha256`, when one exists — older
  dumps from before checksums existed won't have one, and are just kept
  as unverified without touching alpha's copy) — a run after several days
  offline catches up on all of them, not just the newest, without
  re-downloading anything already pulled.
- **Verifies each freshly-fetched dump's checksum before deleting
  anything on alpha.** A match triggers an SFTP `rm` of both the dump and
  its checksum file on alpha; a mismatch deletes the *bad local copy*
  instead and leaves alpha's copy alone for a retry next run — a failed
  or partial transfer never results in either end trusting a corrupt file
  or losing the only good copy.
- **Also runs once at every array startup** (`gitea-backup-pull-onboot`,
  a one-line wrapper around the same script, registered with
  `"frequency": "boot"` — `schedule.json` only supports one schedule per
  script path, so this needed a second entry rather than combining it with
  the daily one), in addition to the daily `03:30` schedule — so a restart
  after being offline catches up right away instead of waiting for the
  next scheduled time. Fired via the User Scripts plugin's own
  `disks_mounted` boot-event hook, which calls `startSchedule.php boot`
  once per real boot; verified by invoking that exact call manually
  (without a real reboot) and confirming the wrapper actually ran the
  underlying script. **Gotcha hit twice building this**: a script that
  tries to directly `exec` another file living on `/boot` fails with
  "Permission denied", since `/boot` is VFAT and can never carry a Unix
  execute bit — the wrapper has to invoke it via `bash <path>` instead of
  relying on the file's own (nonexistent) `+x`.
- Lands in `/mnt/disks/WSD2L840/git/` — an **Unassigned Device** (external
  disk, not part of the protected array), not the RAM-backed root — keeps
  the last 30 locally, now the real long-term history given alpha deletes
  its own copies right after a verified pull.
- **The device isn't guaranteed to already be mounted** when the script
  runs, so it resolves the disk by filesystem label (`findfs
  LABEL=WSD2L840`, not a hardcoded `/dev/sdX` — USB/Unassigned-Device
  paths aren't stable across reboots) and mounts it itself via the
  Unassigned Devices plugin's own `rc.unassigned mount` (confirmed
  idempotent — safe to call even when already mounted). Explicitly
  verifies the result is a real mountpoint (`mountpoint -q`) before writing
  anything — refuses to run rather than risk silently dumping backups onto
  the array if the disk is unplugged or fails to mount.
- **Unmounts it again when done — but only if this run is the one that
  mounted it.** Checks the mount state *before* mounting and remembers it;
  if the disk was already mounted for some other reason, this script
  leaves it mounted afterward rather than pulling it out from under
  whatever else was using it. Handled via an `EXIT` trap so it fires
  reliably on any exit path, not just success. One gotcha found while
  testing this: the script itself `cd`s into the mounted directory, so
  without explicitly `cd`ing back out first, the unmount would fail with
  "target is busy" (the shell's own working directory still being on the
  device it's trying to unmount).
- **Also checks for any *other* process still using the disk before
  unmounting, and gives up quietly (no error) if it finds one.** This
  turned out to be necessary rather than just nice-to-have: the
  Unassigned Devices plugin's own `rc.unassigned umount` always does a
  *lazy* (`-l`) unmount and reports "success" unconditionally — verified
  live that it does **not** refuse, error, or even notice when another
  process still has the mountpoint open (a single-device call is never
  the "force" path that would otherwise kill blocking processes first).
  So the script runs its own `fuser -m` check first and simply skips
  calling `rc.unassigned umount` at all if anything turns up, logging a
  plain informational line rather than an error and leaving the disk
  mounted for next time. Verified all three states end-to-end: already
  mounted before the run → untouched either way; not mounted, nothing
  else using it → mounted, used, and cleanly unmounted again; not
  mounted, but something else `cd`s into it mid-run → mounted, used, and
  correctly left mounted with no error.

**Restore path, if ever needed**: manual and deliberate — stop the target
Gitea container, unpack the chosen dump's DB/repos/etc. into `/data` per
Gitea's documented restore steps, restart, verify. Not automated.

**Verified end-to-end**, including the full checksum-verify-then-delete
flow with a genuinely fresh dump (not just a re-run against existing
files): dump + checksum produced on alpha, pulled to `vlda-01` via the
actual scheduled-invocation code path (`startCustom.php`), checksum
verified locally (`sha256sum -c` → `OK`), both files then deleted from
alpha (confirmed empty afterward) — all through the real production
script, not a manual approximation of it.

### Password vault sync (KeePass, via Nextcloud)

Each user's KeePass database now syncs through Nextcloud's own WebDAV
endpoint instead of a bespoke Apache-WebDAV + SFTP-upload pipeline:

| User | Nextcloud account | Vault file | WebDAV URL |
|---|---|---|---|
| `jens` | `jens` | `sicher_jens.kdbx` | `https://cloud.jayfield.org/remote.php/dav/files/jens/sicher_jens.kdbx` |
| `alex` | `alex` | `sicher_alex.kdbx` | `https://cloud.jayfield.org/remote.php/dav/files/alex/sicher_alex.kdbx` |

- Each `.kdbx` lives at the root of that user's Nextcloud Files storage
  (`/srv/nextcloud/data/<user>/files/`, `640 www-data:www-data`), picked
  up via `occ files:scan <user>`.
- Verified 2026-07-19 with a raw WebDAV `PROPFIND`/`GET`/`PUT`/`GET`
  round trip against the `alex` account — correct `Content-Length`,
  matching SHA-256 before and after a write. (KeePassXC has no native
  WebDAV client and there's no display on this host to drive a real
  KeePass session, so this HTTP-level check is the practical
  equivalent — it exercises the same verbs a WebDAV-aware sync client
  or plugin would use.)
- A `600`-permission backup copy of each vault also still lives in the
  respective user's home dir (`/home/jens/sicher_jens.kdbx`,
  `/home/alex/sicher_alex.kdbx`) on this host.
- **Superseded setup**: an Apache `mod_dav_fs` container (`docker-apache2-php` /
  `my-httpd` image, HTTP Basic Auth via `.htpasswd`) running on the admin
  workstation used to serve the vault directly, with files uploaded to
  this host via the chrooted `ftp_jens`/`ftp_alex` SFTP accounts under
  `/var/sftp/`. That local container/image has been removed (it ran with
  `--rm` and had no cron/systemd autostart, so nothing else depended on
  it). The `ftp_jens`/`ftp_alex` accounts remain on this host (chroot and
  files untouched) but are password-locked (`passwd -l`) since nothing
  uses them anymore — reversible with `passwd -u` if SFTP is ever needed
  again.

## Public access / authentication overview

Audited 2026-07-20 — which public hostnames require credentials:

| Host | Requires credentials? | Notes |
|---|---|---|
| `jayfield.org` / `www.jayfield.org` | No (intentionally public static site) | Was accidentally 401'ing every visitor via a stray `.htaccess` — fixed, see `TODO.md` |
| `dyndns.jayfield.org` | Yes — HTTP Basic Auth, one dedicated account per hostname | Vhost-level `<Directory>` block, intentional. Had an OS command injection in its update scripts (fixed 2026-07-20) and no real per-user auth binding (fixed 2026-07-26 with per-host TSIG keys — see `TODO.md`) |
| `mail.jayfield.org` | N/A — root is `403` (no content served there) | |
| `mail.jayfield.org/rspamd` | Yes — rspamd's own login page | |
| `cloud.jayfield.org` (Nextcloud) | Yes — redirects to `/login` | |
| `portainer.jayfield.org` | Yes — admin account already initialized (`/api/users/admin/check` → `204`, no open setup wizard) | |
| `web.jayfield.org` | No | Currently just the stock nginx welcome page (no real content deployed yet), so nothing sensitive is exposed — but worth revisiting before real content goes there if it's meant to stay public |

## Open items / known quirks

- **Intranet migration from `vlda-01` (files, Gitea, Jellyfin, Dogecoin
  node/wallet)** — Gitea done (see "Gitea" and "Gitea backup to `vlda-01`"
  above); files (Syncthing), Jellyfin, and the Dogecoin node/wallet are
  still planning-only, not started. See `SYNC-PLAN.md` for the full
  per-service design, open questions, and build order.

- **PTR/EHLO mismatch (workaround applied, real fix still open)**: the
  PTR record for this host's IP (`89.58.8.149`) resolves to
  `alpha.jayfield.org`, but Postfix's `myhostname` was `mail.jayfield.org`
  (matching the MX record) — a mismatch some strict receivers penalize.
  2026-07-19: `myhostname` changed to `alpha.jayfield.org` to match the
  existing PTR (verified live: `EHLO` → `250-alpha.jayfield.org`); see
  SETUP.md's mail-stack section. The real fix is repointing the PTR to
  `mail.jayfield.org` and then switching `myhostname` back — not done
  yet, since PTR isn't controlled from this host's own DNS zone. Checked
  2026-07-19: the ISP's control panel has no self-service PTR/rDNS
  option at all, so this needs a support ticket to the ISP asking them
  to set the PTR for `89.58.8.149` to `mail.jayfield.org` (worth noting
  in the ticket that the forward record already resolves correctly —
  `mail.jayfield.org` → `89.58.8.149` — since some providers require
  that before they'll set rDNS). If the ISP can't or won't do it, the
  current `myhostname = alpha.jayfield.org` workaround is a fine
  permanent setup as-is, not just a stopgap — PTR and EHLO already
  agree, which is what actually matters for deliverability.
- **Distributed low-and-slow scan on `apache-noscript`**: found 2026-07-19
  while auditing fail2ban jails for the `sshd` timing-evasion pattern below
  — a cluster of ~20 IPs across three `/24`s (`91.230.225.0/24`,
  `185.192.71.0/24`, `185.92.25.0/24`) each probed exactly once, never
  repeating. No per-IP `findtime`/`maxretry` tuning can catch this by
  design — it needs subnet-level (CIDR) banning instead, which risks
  blocking legitimate traffic sharing that block (e.g. carrier NAT). Left
  as routine scan noise for now; revisit if it escalates.
  **Follow-up 2026-07-26**: rechecked — the original three `/24`s are gone
  from current logs entirely (that specific campaign appears to have moved
  on). `apache-noscript` shows normal jail activity (42 total bans, 3
  currently banned) from ordinary repeat-offender IPs already caught by
  per-IP thresholds, not a new distributed single-shot pattern. No
  escalation; still fine to leave as-is.

- **No real nameserver redundancy**: 2026-07-19, `ns1.jayfield.org` and
  `ns2.jayfield.org` were registered as this domain's NS records at the
  registrar. Delegation checked and confirmed consistent — the `.org`
  registry (checked against two separate registry servers), this host's
  own authoritative zone, and both `ns1`/`ns2` themselves (queried
  directly, `aa` flag set, matching SOA serial `2022032214` at the time
  of that check — serial has since moved to `2022032216` from the
  SPF/DMARC/DKIM fix below, unrelated to NS/glue) all agree.
  But both names glue to the *same* IP (`89.58.8.149`, i.e. this same
  host) — RFC 2182 §3 recommends at least two nameservers not on the
  same network segment, and this setup has zero real redundancy: if
  `alpha` goes down, both "nameservers" disappear together. Tried
  unregistering `ns2` at the registrar since it wasn't adding real
  redundancy anyway — confirmed the ISP/registrar requires a minimum of
  two NS records at the delegation layer and rejected the change, so
  both stay registered as-is. Redundancy risk remains open; the only
  real fix is a second nameserver on genuinely separate infrastructure.

(Previously resolved: legacy `/portainer` proxy paths on the
default vhost and `jayfield.org`, stale DNS records `test`, `kacke`,
`pupsi`, `mausi`, `hase`, `super`, and a disabled `nextcloud.conf` vhost
plus its `a2dissite` marker file — a leftover from a Nextcloud install
that was disabled at some point but never actually deployed (no
`/var/www/nextcloud`, no package, no DB, no cert) — all cleaned up.
Also: `mnt-vlda-01.mount`'s recurring boot failure — root-caused to a
dormant WireGuard tunnel (`alpha_at_moon`, routing the same
`192.168.22.0/24` home-LAN range the CIFS target lived on) — resolved
2026-07-19 by removing all three together rather than reviving the
tunnel: `wireguard`/`wireguard-tools` purged, `nginx` +
`libnginx-mod-rtmp` purged (an unrelated, previously-uncatalogued RTMP
streaming service on `:1935`, found via the same diff pass, also unused
and removed), and `samba`/`cifs-utils` purged along with the `fstab`
entry and stored CIFS credentials. Configs backed up to
`/root/removed-configs-backup/` on the server before deletion, in case
any of the three are wanted again. Zero failed systemd units after.
Also: KeePass vault sync migrated 2026-07-19 from local Apache-WebDAV +
SFTP-upload to Nextcloud's built-in WebDAV — see "Password vault sync"
above. Also: SPF, DMARC, and both DKIM (`2021`/`2022._domainkey`) TXT
records in `db.jayfield.org` were silently broken — root cause was
**unquoted TXT rdata in the zone file**: BIND's zone parser treats an
unquoted `;` as a start-of-comment (so `v=DMARC1; p=reject;` was being
served as just `"v=DMARC1"`, and both DKIM keys as just `"v=DKIM1"` with
the `k=rsa; p=<key>` portion silently dropped — meaning DKIM signature
verification had been failing for all outbound mail), and an unquoted
multi-token SPF line was being split into multiple concatenated
character-strings instead of one, merging in an unrelated
`google-site-verification=` token and invalidating the SPF syntax.
Fixed 2026-07-19 by quoting every TXT value (splitting the long DKIM
keys across two `<=255`-byte quoted strings each, since a single
character-string is capped at 255 bytes — see `named-checkzone`'s
"text string too long" if this is missed) and separating
`google-site-verification` into its own TXT record. SOA serial bumped
`2022032214` → `2022032216`. Verified directly against the authoritative
server (`dig ... @89.58.8.149`) post-reload; public resolvers catch up
as the old records' TTLs (up to 3600s) expire.
Also 2026-07-19: audited for phpMyAdmin/Adminer/webmin/cPanel/Plesk/
ISPConfig/phpPgAdmin residue (dpkg logs, `rc`-state packages, filesystem-wide
filename search, Apache configs, MySQL users/DBs, crontabs, Docker
images/volumes) — clean; the only name hits were stock `fail2ban`/`augeas`
package files, not evidence of a prior install. Two stray public files
(`ShowKeyPlus_x64.zip`, `MediaCreationTool1909.exe` — unrelated pirated
Windows tooling, plus an empty `test` file) were found sitting in
`/var/www/html/pub`, publicly served off the default vhost; moved to
`/home/jens/pub` and the now-empty `pub/` directory removed from the web
root. Also: SSH hardened (`PermitRootLogin no`, `PasswordAuthentication no`,
key-only) and the `sshd` fail2ban jail's `findtime` widened from the
30-minute default to 24h (`maxretry` 3→4, `bantime` 4h→24h) after
`47.76.192.176` was found grinding root/`jayfield` SSH logins roughly every
33 minutes for most of a day — just outside the old 30-minute window, so it
never accumulated enough strikes to get banned; see `TODO.md` and
`SETUP.md` §1/§3 for the fix and the same evasion check applied to every
other jail (only one other pattern turned up — see "Open items" above).
Also: Portainer's `:9443` UI, previously published directly to the public
interface on its own self-signed cert, became unreachable in
HSTS-enforcing browsers (`MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT`, no
click-through) once the `includeSubDomains` HSTS policy above started
pinning `alpha.jayfield.org` to trusted-cert-only HTTPS on every port —
not a MITM, a self-inflicted interaction between two otherwise-correct
pieces of config. Fixed 2026-07-20 by moving it behind an Apache reverse
proxy on its own subdomain, `portainer.jayfield.org`, same pattern as
`web.jayfield.org`/`cloud.jayfield.org`; see "Portainer" above and
`TODO.md` for the full details.
Also: an audit of which public hostnames require credentials (prompted by
the Portainer/HSTS work above) turned up an OS command injection in the
dyndns update scripts (`/var/www/dyndns/nsupdate*.php` built a shell
command by directly interpolating unsanitized `$_GET` input into
`exec()`) — gated by the vhost's Basic Auth, so not anonymously
exploitable, but a real privilege escalation for anyone with valid dyndns
credentials. Fixed 2026-07-20 by rewriting the update path to use
`proc_open()` with array-form arguments (no shell involved at all) plus
strict hostname/IP validation. That same audit also surfaced (a) a
regression from the 2026-07-19 TSIG key hardening that had silently
broken legitimate dyndns updates — fixed with a POSIX ACL scoped to just
that key file, not group membership, since the `bind` group also owns
`rndc.key` — and (b) that access logging for every real vhost on this
host is currently a no-op, and (c) that the dyndns app has no real
per-user auth binding.
(b) fixed 2026-07-25 — see `TODO.md` for the root cause and the
per-vhost `CustomLog` fix. (c) fixed 2026-07-26: every dyndns hostname
now has its own dedicated `.htpasswd` account and TSIG key, with BIND's
`update-policy` (not just an app-level check) refusing any update where
the authenticated user doesn't match the requested hostname — see
`TODO.md` for the full rollout, including the now-retired shared key
and legacy fallback.