docs-alpha.jayfield.org/README.md

alpha.jayfield.org — Web Services Overview

Host: alpha.jayfield.org (89.58.8.149), Ubuntu 24.04 LTS (upgraded from 22.04 on 2026-07-26 — see TODO.md for the full rollout). SSH: ssh -p 10022 jens@alpha.jayfield.org

All administration in this repo's notes is performed remotely against this host over SSH (port 10022) — there is no local checkout of any service config; changes are made directly on the server and recorded here afterward.

This host runs two layers of web services: native services managed by system Apache, and containerized services managed by Docker (visible in Portainer). Apache reverse-proxies into the Docker layer where needed.

Domain namespace

jayfield.org covers two distinct things under one name:

• The root domain and its public subdomains (alpha, www, mail, dyndns, cloud, web, portainer, git, ...) all point at this host, 89.58.8.149 — everything this doc is about.
• home.jayfield.org is a separate, deliberately private namespace for the home intranet (vlda-01 and other home-LAN devices, see SYNC-PLAN.md) — not part of this host's authoritative zone, not publicly resolvable, kept off public DNS by design.

Native services (systemd / apt)

Service	Role
apache2 (2.4.58)	Terminates HTTP/HTTPS for all vhosts, reverse-proxies to backends (Docker containers, rspamd, Portainer)
bind9	Authoritative DNS for the jayfield.org zone (ns1/ns2.jayfield.org are both this host); also the box's own recursive resolver — see below
docker	Runs the containerized web services (see below)
certbot (apt, 2.9.0)	Let's Encrypt cert issuance/renewal, via certbot.timer (systemd)
Postfix (master)	SMTP, ports 25/587, on the public IP + localhost
Dovecot	IMAP/IMAPS (143/993) + ManageSieve (4190)
rspamd (4.1.3, rspamd.com PPA — not the OS's own older native package)	Spam filtering for the mail stack, localhost-only (11332–11334), proxied at mail.jayfield.org/rspamd
redis-server	Localhost-only (6379); backs rspamd's Bayes classifier, history, and reputation caching
mariadb (10.11.14, noble-native)	Holds vmail (Postfix/Dovecot virtual accounts) — see TODO.md for the 10.3→10.6→10.11 migration history

Python: python3.12.3 (noble's default). Previously python3.10.12 on 22.04, itself fixed from a stalled python3.8 leftover of the original incomplete 20.04→22.04 upgrade (breaking apt's certbot via a pytz/zoneinfo incompatibility) — that whole incomplete-upgrade saga is now fully resolved, see TODO.md.

Post-boot sanity check

Checked into this repo: scripts/alpha/sanity-check.sh / scripts/alpha/sanity-check.service (Ubuntu 24.04 — see scripts/alpha/README.md).

/usr/local/sbin/sanity-check.sh (added 2026-07-26, after the 24.04 upgrade turned up two real post-upgrade breakages — see TODO.md) checks that the whole stack is actually healthy, not just "systemd says active": package/dpkg consistency, disk space, SSH hardening (PermitRootLogin/PasswordAuthentication), BIND + DNSSEC validation + the dyndns zone, the full mail stack including a live rspamc symbols scan (catches a phantom-active rspamd, which is exactly what happened during the 24.04 upgrade) and whether anything is actually listening on Postfix's configured milter port, MariaDB/vmail data + Postfix's SQL lookups, fail2ban's jail count and the custom findtime hardening, all 7 public vhosts by expected HTTP status code (plus a genuine Gitea API check, not just a static-page response), and all 6 Docker containers.

Runs automatically once per boot via sanity-check.service (a oneshot unit, systemctl enabled, After= all the relevant services plus a 10s settle delay before running) — covers OS upgrades, kernel updates, and any other change that ends in a reboot, which in practice has been all of them so far. Also safe to run manually after a change that doesn't reboot (a plain apt upgrade, a config edit): sudo /usr/local/sbin/sanity-check.sh. Each run logs to /var/log/sanity-check/<timestamp>.log and emails a pass/warn/fail summary to jens@jayfield.org via the local mail stack (no external dependency). Exits non-zero if anything actually failed (warnings, like routine pending package upgrades, don't fail the run).

Weekly attacker summary

Checked into this repo: scripts/alpha/attacker-check.py / scripts/alpha/cron.d/attacker-check (Ubuntu 24.04 — see scripts/alpha/README.md).

/usr/local/sbin/attacker-check.py (added 2026-07-26) parses fail2ban.log (current + recent rotations, gzipped or not — reads enough history to cover a full week regardless of exactly when logrotate's daily midnight run lands relative to the cron job) for the past 7 days and turns it into two append-only CSVs for longitudinal analysis, plus a human-readable weekly summary emailed to root@jayfield.org and saved as a dated report file:

• /var/log/attacker-check/per_ip_weekly.csv — one row per (jail, IP) per week: event count, first/last seen, /24 subnet, and interval statistics (mean, stdev) between repeat hits.
• /var/log/attacker-check/per_jail_weekly.csv — weekly rollup per jail: total events, distinct IPs, distinct /24s, new bans.
• /var/log/attacker-check/reports/<week-ending-date>.txt — the full human-readable report for that run, one file per date.

root@jayfield.org didn't resolve to anything before this (not in /etc/aliases, not in the vmail aliases table) — added it as a new vmail.aliases row → jens@jayfield.org, matching the existing postmaster/hostmaster/webmaster/wlan pattern.

Two detection heuristics run on top of the raw data, both modeled directly on real campaigns found earlier this engagement (see TODO.md):

• Distributed single-shot scans: flags any /24 subnet with ≥5 distinct IPs averaging ≤3 events each against the same jail — the shape of the apache-noscript campaign found 2026-07-19.
• Regular-timing evasion: flags any IP with ≥3 repeat hits on the same jail whose inter-hit timing has a coefficient of variation under 15% — the shape of the sshd/dovecot low-and-slow campaigns found and fixed 2026-07-19/2026-07-25 (attackers spacing requests just past a jail's findtime window). A first live run against this week's real logs correctly re-identified the same dovecot campaign IPs from that earlier manual investigation, with matching ~22–28 minute intervals.

Runs weekly via /etc/cron.d/attacker-check, Sunday 06:00 (clear of logrotate's daily midnight timer). Safe to also run manually at any time: sudo /usr/local/sbin/attacker-check.py.

System users

User	uid	Home	Purpose
jens	1001	/home/jens	Primary sudo admin account — all administration in this repo is done as this user over SSH
alex	1005	/home/alex	Secondary, non-sudo personal account (added 2026-07-19)

Both also exist as separate Nextcloud accounts (see "Password vault sync" below) and as chrooted, SFTP-only accounts ftp_jens/ftp_alex under /var/sftp/ — the latter currently password-locked, not deleted (see below).

DNS resolver architecture

Fixed 2026-07-19: the box's own outbound DNS resolution (used by Postfix's postscreen for DNSBL lookups, and by every other service) previously forwarded through the hosting provider's shared resolvers (46.38.225.230/46.38.252.230). Confirmed via a live test: GMX's own mail servers (212.227.15.15/.19, verified via reverse DNS as mout.gmx.net) were being dropped by postscreen on a false positive — Spamhaus's DNSBL was returning 127.255.255.254, its documented rate-limit sentinel ("this querying resolver has exceeded our free-tier volume"), not a real listing. The shared resolver was almost certainly pooling query volume from other tenants of the same hosting provider, pushing it over Spamhaus's threshold.

Fix:

• Removed the forwarders {} block from named.conf.options — bind9 now does full iterative/recursive resolution itself (root hints → TLD → authoritative) for trusted clients (localhost + localnets) instead of relaying to the shared upstream.
• Repointed /etc/systemd/resolved.conf's DNS= from the shared provider IPs to 127.0.0.1 (i.e. bind9), then systemctl restart systemd-resolved. The 127.0.0.53 stub listener and /etc/resolv.conf symlink were left untouched, so Docker's DNS-detection logic (which special-cases the stub) is unaffected.
• (An unbound install was tried first as a separate local resolver, but bind9 already listens on 127.0.0.1:53/[::1]:53 — listen-on-v6 { any; } plus BIND's default of listening on all IPv4 interfaces — so they couldn't coexist on the same port. Reusing bind9 for both roles was simpler than fighting that conflict.)

Verified: both GMX IPs now correctly resolve as not listed; postscreen now PASSes them. A real end-to-end test email ("Claude email test #01") delivered successfully to both jens@ and otto2022@jayfield.org after the fix.

Known caveat: direct recursive resolution occasionally hits a single slow/unresponsive server somewhere in a delegation chain, where a well-provisioned forwarding resolver's warm cache usually wouldn't. Seen once during testing — a transient PTR lookup timeout caused one retry to soft-fail (450, not 550) before a second retry succeeded seconds later. Not expected to be a recurring problem, but worth knowing if a one-off 450 4.7.25 Client host rejected shows up in mail.log.

Apache virtual hosts

ServerName	:80	:443	Notes
dyndns.jayfield.org	→ redirect to :443	Basic Auth (Restricted Content), DocumentRoot /var/www/dyndns (PHP)	Also the effective default vhost for unmatched hostnames (e.g. bare alpha.jayfield.org falls through here)
jayfield.org	→ redirect to :443	Static site (/var/www/html)	Previously had no :80 vhost of its own — fell through to the dyndns default and redirected to the wrong domain; fixed
www.jayfield.org	→ redirect to :443	Static site (/var/www/html)	Renewal fixed to use the apache certbot plugin (was standalone, which required stopping Apache)
mail.jayfield.org	→ redirect to :443	Webmail-adjacent, proxies /rspamd → http://localhost:11334/
web.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker web (nginx) container on 127.0.0.1:8081	New — see below
cloud.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker nextcloud container on 127.0.0.1:8082, sets X-Forwarded-Proto: https	Nextcloud — see below
portainer.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker portainer container on 127.0.0.1:9443 over HTTPS (SSLProxyEngine/SSLProxyVerify none for the loopback hop to Portainer's self-signed cert)	Added 2026-07-20 — see "Portainer" below
git.jayfield.org	→ redirect to :443	Reverse-proxies to the Docker gitea container on 127.0.0.1:8083, sets X-Forwarded-Proto: https	Added 2026-07-26 — migrated from vlda-01, see "Gitea" below
default (_default_:443, 000-default on :80)	Apache's stock placeholder page	—

All Let's Encrypt certs (9 total, one per hostname/group) auto-renew via certbot.timer; verified with certbot renew --dry-run.

All 8 domain :443 blocks send Strict-Transport-Security: max-age=63072000; includeSubDomains (mod_headers). Deliberately skipped on the default-ssl.conf catch-all — it's the self-signed fallback for unmatched SNI/Host headers, not a real trusted hostname a browser could pin.

Docker services (visible in Portainer)

Container	Image	Ports	Purpose
portainer	portainer/portainer-ce:latest	8000 (edge-agent tunnel, published), 9443 (HTTPS UI, 127.0.0.1-only), 9000 (UI, internal only)	Docker management UI — https://portainer.jayfield.org, reverse-proxied by Apache to 127.0.0.1:9443 (fixed 2026-07-20, see below)
web	nginx:latest	127.0.0.1:8081 (localhost-only, not published externally)	Backing service for web.jayfield.org; only reachable through the Apache reverse proxy
nextcloud	nextcloud:apache	127.0.0.1:8082 (localhost-only)	Nextcloud app (PHP+Apache bundled in the image); backing service for cloud.jayfield.org
nextcloud-db	mariadb:lts	internal only (nextcloud_net)	Dedicated DB for Nextcloud — kept separate from the host's own MySQL (which only holds vmail)
nextcloud-redis	redis:alpine	internal only (nextcloud_net)	File locking + distributed cache + PHP session storage for Nextcloud. Dedicated container rather than reusing the host's native redis-server, since that instance is bound to 127.0.0.1/::1 only with no auth — reusing it would have meant loosening a shared service's network exposure just to save one small container
gitea	gitea/gitea:1.27.0	127.0.0.1:8083 (localhost-only)	Git hosting, migrated from vlda-01 2026-07-26 — see "Gitea" below

Data persists in the portainer_data named volume, independent of the container itself — updating Portainer means recreating the container against a freshly pulled image while keeping this volume and the /var/run/docker.sock bind mount; the old container is kept renamed (not deleted) until the new one is verified healthy.

Two early-test containers (my-apache-app, a stopped httpd, and clever_jepsen, a stale hello-world container from initial Docker setup) have been removed as cleanup.

web.jayfield.org request flow

Internet → Apache :80/:443 (web.jayfield.org vhost)
             → Let's Encrypt TLS termination
             → ProxyPass http://127.0.0.1:8081/
                 → Docker container "web" (nginx) on the docker0 bridge

The container is intentionally bound to 127.0.0.1 only, so it cannot be reached directly on port 8081 from the internet — Apache is the only path in. This is the pattern to follow for any future container-backed site: publish the container to 127.0.0.1:<port>, add an Apache vhost for the subdomain that proxies to it, then run certbot --apache -d <subdomain> to get HTTPS.

Portainer (portainer.jayfield.org)

Internet → Apache :80/:443 (portainer.jayfield.org vhost)
             → Let's Encrypt TLS termination
             → ProxyPass https://127.0.0.1:9443/ (SSLProxyVerify none —
                 backend keeps Portainer's own self-signed cert, but that
                 hop never leaves the host)
                 → Docker container "portainer" HTTPS UI, published to
                   127.0.0.1 only

Previously Portainer published 9443 directly to the public interface with its own self-signed cert; fixed 2026-07-20 after that became unreachable in HSTS-enforcing browsers (see TODO.md). The container was recreated with -p 127.0.0.1:9443:9443 (old container kept renamed until the new one was verified healthy, then removed — same caution as the update procedure below). The 8000 edge-agent tunnel port is still published directly; nothing currently depends on it, but it wasn't in scope for this fix.

Nextcloud (cloud.jayfield.org)

Minimal, data-hosting-focused deployment — core Files app only. No Collabora/OnlyOffice, Talk, full-text search, external storage backends, or LDAP/SSO; those can be layered on later without re-architecting.

Internet → Apache :80/:443 (cloud.jayfield.org vhost)
             → Let's Encrypt TLS termination, X-Forwarded-Proto: https
             → ProxyPass http://127.0.0.1:8082/
                 → container "nextcloud" (nextcloud:apache)
                     ↕ private network "nextcloud_net"
                 → container "nextcloud-db" (mariadb:lts)
                 → container "nextcloud-redis" (redis:alpine)

• DNS: cloud A record added to /etc/bind/db.jayfield.org (serial bumped 2022032213 → 2022032214).
• Data volume: bind-mounted at /srv/nextcloud/data on the host (owned by uid/gid 33, matching www-data both on the host and inside the container) rather than an opaque named Docker volume — so it's directly inspectable/backupable with normal tools (du, rsync, borg).
• App volume: named Docker volume nextcloud_html, holds Nextcloud core + config.php; survives image upgrades.
• DB volume: named Docker volume nextcloud_db.
• Auto-install: done via the image's env-var-driven installer (MYSQL_*, NEXTCLOUD_ADMIN_USER/NEXTCLOUD_ADMIN_PASSWORD, NEXTCLOUD_TRUSTED_DOMAINS, TRUSTED_PROXIES, OVERWRITEPROTOCOL=https, REDIS_HOST) rather than leaving an unconfigured setup wizard reachable on the public internet.
• Credentials: DB root/user passwords and the initial admin password are in /root/.nextcloud-credentials on the server (600 root:root) — not committed anywhere, not in this repo.
• Cron: Nextcloud's background-job mode is set to cron (occ background:cron); a root crontab entry runs docker exec -u www-data nextcloud php -f /var/www/html/cron.php every 5 minutes. No sidecar cron container.
• PHP limits: image defaults are already generous for file hosting (512M memory_limit/upload_max_filesize/post_max_size, unlimited max_execution_time) — no tuning needed for this pass.

Gitea (git.jayfield.org)

Migrated 2026-07-26 from vlda-01 (home intranet server) per SYNC-PLAN.md's build order — first item on that plan, "no open questions." Full migration, not a sync: vlda-01's Gitea container was stopped, its entire /data volume (SQLite DB, all repos, LFS objects, host SSH keys) copied straight across via a piped ssh ... tar | ssh ... tar (no intermediate disk, ~4GB), and brought up on alpha as a permanent container — same web/cloud/portainer pattern (publish to 127.0.0.1:<port>, Apache vhost, certbot --apache).

Internet → Apache :80/:443 (git.jayfield.org vhost)
             → Let's Encrypt TLS termination, X-Forwarded-Proto: https
             → ProxyPass http://127.0.0.1:8083/
                 → container "gitea" (gitea/gitea:1.27.0)
                     → SQLite DB + repos, bind-mounted at /srv/gitea/data

• Version pinned, not :latest: gitea/gitea:1.27.0, matching what was actually running on vlda-01 — a version-compatible move first, upgrading is a separate later decision.
• Data: bind-mounted at /srv/gitea/data on the host (matching the /srv/nextcloud/data convention — directly inspectable/backupable), not an opaque named volume.
• Ownership: the source data was owned by uid/gid 1000 on vlda-01, but 1000 is already vmail on alpha (Dovecot's mailbox user) — remapped everything to a dedicated 1010:1010 (chown -R, container run with -e USER_UID=1010 -e USER_GID=1010) to avoid any collision with or confusion around the real vmail account.
• SQLite, not the host's MariaDB: vlda-01's Gitea was already self-contained SQLite (DB_TYPE = sqlite3), so no DB export/import or extra container was needed — the DB file just came along inside /data.
• app.ini updated for the new home: DOMAIN/SSH_DOMAIN/ROOT_URL repointed from the old LAN IP (192.168.22.90) to git.jayfield.org; DISABLE_SSH set to true — the existing repo remotes already used HTTP(S), not git@.../ssh://, so git-over-SSH was dropped rather than opening yet another public port to preserve a mode nothing was actually using. HTTPS clone/push covers the same ground through the same Apache TLS termination every other service uses.
• Reverse-proxy scheme awareness: without X-Forwarded-Proto: https (added to the vhost, same as cloud.jayfield.org), Gitea's API correctly used ROOT_URL for some fields but generated http:// (not https://) clone_url/html_url for repos — since the Apache→container hop itself is plain HTTP and Gitea otherwise has no way to know the original request arrived over TLS. Fixed and verified via the API.
• Verified end-to-end: all previously-existing repos present via the API (including this docs-alpha.jayfield.org repo), admin user (jayfield, migrated over from vlda-01) intact; a real git clone over HTTPS reproduced identical commit history to the working copy; this repo's own origin remote repointed to https://git.jayfield.org/... and a real git push succeeded against it.
• vlda-01's copy: left stopped (container not removed, data not deleted) as a cold backup — reversible if ever needed; not decommissioned outright.
• Admin user renamed jayfield → jens 2026-07-26 (password rotated at the same time), via Gitea's own web admin panel (Site Administration → User Accounts → Edit Account) rather than a direct DB/API edit — that's the one code path that correctly moves the on-disk repo storage directories and sets up a redirect together, rather than risking an inconsistent rename. Verified: new credentials authenticate (GET /api/v1/user → "username":"jens"), both the new canonical path (/jens/docs-alpha.jayfield.org) and the old one (/jayfield/..., via Gitea's automatic redirect) return 200, and this repo's local origin remote and stored git credentials were updated to match.

Gitea backup to vlda-01 (SYNC-PLAN.md item 2b)

Checked into this repo, split by which host each half runs on (they're on different OSes with different conventions — see each README.md):

• Alpha side (Ubuntu 24.04): scripts/alpha/gitea-backup.sh / scripts/alpha/cron.d/gitea-backup — see scripts/alpha/README.md.
• vlda-01 side (Unraid 7.3.2): scripts/clients/vlda-01/ — see scripts/clients/vlda-01/README.md, which also covers the Unraid-specific persistence gotchas (RAM-backed root, VFAT /boot, Unassigned Devices) that shaped how this had to be built.

Built 2026-07-26. A point-in-time backup, not a hot standby/live mirror — now that alpha is the canonical live Gitea, this keeps vlda-01's copy from just going stale after the migration. Same network direction as everything else in this project: alpha has no route into the home LAN, so the pull is initiated by vlda-01, never alpha reaching in.

Why gitea dump, not a raw rsync of /srv/gitea/data: the DB is a live SQLite file; copying it while Gitea is writing risks shipping a torn copy. gitea dump produces one consistent archive (DB + repos + LFS + config together) — also what Gitea's own restore procedure expects.

On alpha — /usr/local/sbin/gitea-backup.sh, /etc/cron.d/gitea-backup (daily 02:30, clear of logrotate's midnight timer):

• docker exec -u git gitea gitea dump -f /data/backups/gitea-dump-<timestamp>.zip --skip-index into /srv/gitea/data/backups/ (inside the existing bind mount — already host-inspectable, no extra volume). Timestamped to the second, not just the date: gitea dump refuses to overwrite an existing filename, which a plain per-day name would hit on any manual re-run (found this the hard way testing it).
• Writes a sha256sum-format .sha256 file alongside each dump, so vlda-01 can verify integrity after transfer before trusting it enough to delete the alpha-side copy (see below).
• Applies a POSIX ACL (setfacl -m u:giteabackup:r) to each new archive and its checksum file so the pull account below can read them. Has to happen explicitly per file — a default ACL on the directory alone isn't enough: gitea dump creates each file 600, and the resulting ACL mask neuters the inherited grant (getfacl shows #effective:--- if this step is skipped).
• Prunes to the last 14 archives (dump + checksum pairs together) — a safety net only: in normal operation vlda-01 deletes each dump right after a verified pull, well before this would ever trigger. Only matters if vlda-01 is offline long enough for dumps to pile up unswept.

Transport — a dedicated, restricted account on alpha, not a shared or personal credential:

• System account giteabackup (/usr/sbin/nologin), member of the existing sftp group — reuses the Match Group sftp / ForceCommand internal-sftp block already in sshd_config for ftp_jens/ftp_alex; no new sshd config needed.
• Chrooted to /var/sftp/giteabackup/, containing only a bind mount of /srv/gitea/data/backups. Read-write at the filesystem level (rw in /etc/fstab — needed so vlda-01 can delete a dump after verifying it), but the account's actual privilege is narrower than that sounds: the directory-level ACL grants giteabackup rwx (needed for delete/unlink, which is a directory operation in POSIX), while each individual file's ACL only grants r. Verified the boundary actually holds: an SFTP rm on an existing dump succeeds, but an SFTP put attempting to overwrite one fails with "Permission denied" — it can delete a file wholesale, never tamper with one in place.
• Key-only auth: a dedicated ed25519 keypair generated on vlda-01 (private key never leaves it), authorized_keys further restricted (restrict,no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty) on top of the chroot/ForceCommand.
• Plain SFTP, not real rsync: ForceCommand internal-sftp blocks a genuine rsync --server invocation from the client, so the transport is SFTP get/rm, not rsync.

On vlda-01 — via the User Scripts plugin, not a raw crontab//usr/local/sbin script: Unraid's / and /usr are RAM-backed (rootfs/overlay) and don't survive a reboot — only /boot (the actual USB stick) does. The pull script lives at /boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script, scheduled daily 03:30 via schedule.json/customSchedule.cron (update_cron folds it into the live /etc/cron.d/root alongside Unraid's own generated jobs) — the same plugin already used for this box's other scheduled maintenance, no new persistence mechanism invented. The SSH keypair needed no special handling: Unraid already auto-mirrors /root/.ssh to /boot/config/ssh/root/ on its own.

• Lists what's actually on alpha via SFTP, fetches only filenames not already present locally (dump + its .sha256, when one exists — older dumps from before checksums existed won't have one, and are just kept as unverified without touching alpha's copy) — a run after several days offline catches up on all of them, not just the newest, without re-downloading anything already pulled.
• Verifies each freshly-fetched dump's checksum before deleting anything on alpha. A match triggers an SFTP rm of both the dump and its checksum file on alpha; a mismatch deletes the bad local copy instead and leaves alpha's copy alone for a retry next run — a failed or partial transfer never results in either end trusting a corrupt file or losing the only good copy.
• Also runs once at every array startup (gitea-backup-pull-onboot, a one-line wrapper around the same script, registered with "frequency": "boot" — schedule.json only supports one schedule per script path, so this needed a second entry rather than combining it with the daily one), in addition to the daily 03:30 schedule — so a restart after being offline catches up right away instead of waiting for the next scheduled time. Fired via the User Scripts plugin's own disks_mounted boot-event hook, which calls startSchedule.php boot once per real boot; verified by invoking that exact call manually (without a real reboot) and confirming the wrapper actually ran the underlying script. Gotcha hit twice building this: a script that tries to directly exec another file living on /boot fails with "Permission denied", since /boot is VFAT and can never carry a Unix execute bit — the wrapper has to invoke it via bash <path> instead of relying on the file's own (nonexistent) +x.
• Lands in /mnt/disks/WSD2L840/git/ — an Unassigned Device (external disk, not part of the protected array), not the RAM-backed root — keeps the last 30 locally, now the real long-term history given alpha deletes its own copies right after a verified pull.
• The device isn't guaranteed to already be mounted when the script runs, so it resolves the disk by filesystem label (findfs LABEL=WSD2L840, not a hardcoded /dev/sdX — USB/Unassigned-Device paths aren't stable across reboots) and mounts it itself via the Unassigned Devices plugin's own rc.unassigned mount (confirmed idempotent — safe to call even when already mounted). Explicitly verifies the result is a real mountpoint (mountpoint -q) before writing anything — refuses to run rather than risk silently dumping backups onto the array if the disk is unplugged or fails to mount.
• Unmounts it again when done — but only if this run is the one that mounted it. Checks the mount state before mounting and remembers it; if the disk was already mounted for some other reason, this script leaves it mounted afterward rather than pulling it out from under whatever else was using it. Handled via an EXIT trap so it fires reliably on any exit path, not just success. One gotcha found while testing this: the script itself cds into the mounted directory, so without explicitly cding back out first, the unmount would fail with "target is busy" (the shell's own working directory still being on the device it's trying to unmount). Verified both directions: already mounted → stays mounted after; not mounted → gets mounted, used, and unmounted again, with no "busy" error.

Restore path, if ever needed: manual and deliberate — stop the target Gitea container, unpack the chosen dump's DB/repos/etc. into /data per Gitea's documented restore steps, restart, verify. Not automated.

Verified end-to-end, including the full checksum-verify-then-delete flow with a genuinely fresh dump (not just a re-run against existing files): dump + checksum produced on alpha, pulled to vlda-01 via the actual scheduled-invocation code path (startCustom.php), checksum verified locally (sha256sum -c → OK), both files then deleted from alpha (confirmed empty afterward) — all through the real production script, not a manual approximation of it.

Password vault sync (KeePass, via Nextcloud)

Each user's KeePass database now syncs through Nextcloud's own WebDAV endpoint instead of a bespoke Apache-WebDAV + SFTP-upload pipeline:

User	Nextcloud account	Vault file	WebDAV URL
jens	jens	sicher_jens.kdbx	https://cloud.jayfield.org/remote.php/dav/files/jens/sicher_jens.kdbx
alex	alex	sicher_alex.kdbx	https://cloud.jayfield.org/remote.php/dav/files/alex/sicher_alex.kdbx

• Each .kdbx lives at the root of that user's Nextcloud Files storage (/srv/nextcloud/data/<user>/files/, 640 www-data:www-data), picked up via occ files:scan <user>.
• Verified 2026-07-19 with a raw WebDAV PROPFIND/GET/PUT/GET round trip against the alex account — correct Content-Length, matching SHA-256 before and after a write. (KeePassXC has no native WebDAV client and there's no display on this host to drive a real KeePass session, so this HTTP-level check is the practical equivalent — it exercises the same verbs a WebDAV-aware sync client or plugin would use.)
• A 600-permission backup copy of each vault also still lives in the respective user's home dir (/home/jens/sicher_jens.kdbx, /home/alex/sicher_alex.kdbx) on this host.
• Superseded setup: an Apache mod_dav_fs container (docker-apache2-php / my-httpd image, HTTP Basic Auth via .htpasswd) running on the admin workstation used to serve the vault directly, with files uploaded to this host via the chrooted ftp_jens/ftp_alex SFTP accounts under /var/sftp/. That local container/image has been removed (it ran with --rm and had no cron/systemd autostart, so nothing else depended on it). The ftp_jens/ftp_alex accounts remain on this host (chroot and files untouched) but are password-locked (passwd -l) since nothing uses them anymore — reversible with passwd -u if SFTP is ever needed again.

Public access / authentication overview

Audited 2026-07-20 — which public hostnames require credentials:

Host	Requires credentials?	Notes
jayfield.org / www.jayfield.org	No (intentionally public static site)	Was accidentally 401'ing every visitor via a stray .htaccess — fixed, see TODO.md
dyndns.jayfield.org	Yes — HTTP Basic Auth, one dedicated account per hostname	Vhost-level <Directory> block, intentional. Had an OS command injection in its update scripts (fixed 2026-07-20) and no real per-user auth binding (fixed 2026-07-26 with per-host TSIG keys — see TODO.md)
mail.jayfield.org	N/A — root is 403 (no content served there)
mail.jayfield.org/rspamd	Yes — rspamd's own login page
cloud.jayfield.org (Nextcloud)	Yes — redirects to /login
portainer.jayfield.org	Yes — admin account already initialized (/api/users/admin/check → 204, no open setup wizard)
web.jayfield.org	No	Currently just the stock nginx welcome page (no real content deployed yet), so nothing sensitive is exposed — but worth revisiting before real content goes there if it's meant to stay public

Open items / known quirks

• Intranet migration from vlda-01 (files, Gitea, Jellyfin, Dogecoin node/wallet) — planning stage only, not started. See SYNC-PLAN.md for the full per-service design, open questions, and build order.

• PTR/EHLO mismatch (workaround applied, real fix still open): the PTR record for this host's IP (89.58.8.149) resolves to alpha.jayfield.org, but Postfix's myhostname was mail.jayfield.org (matching the MX record) — a mismatch some strict receivers penalize. 2026-07-19: myhostname changed to alpha.jayfield.org to match the existing PTR (verified live: EHLO → 250-alpha.jayfield.org); see SETUP.md's mail-stack section. The real fix is repointing the PTR to mail.jayfield.org and then switching myhostname back — not done yet, since PTR isn't controlled from this host's own DNS zone. Checked 2026-07-19: the ISP's control panel has no self-service PTR/rDNS option at all, so this needs a support ticket to the ISP asking them to set the PTR for 89.58.8.149 to mail.jayfield.org (worth noting in the ticket that the forward record already resolves correctly — mail.jayfield.org → 89.58.8.149 — since some providers require that before they'll set rDNS). If the ISP can't or won't do it, the current myhostname = alpha.jayfield.org workaround is a fine permanent setup as-is, not just a stopgap — PTR and EHLO already agree, which is what actually matters for deliverability.

• Distributed low-and-slow scan on apache-noscript: found 2026-07-19 while auditing fail2ban jails for the sshd timing-evasion pattern below — a cluster of ~20 IPs across three /24s (91.230.225.0/24, 185.192.71.0/24, 185.92.25.0/24) each probed exactly once, never repeating. No per-IP findtime/maxretry tuning can catch this by design — it needs subnet-level (CIDR) banning instead, which risks blocking legitimate traffic sharing that block (e.g. carrier NAT). Left as routine scan noise for now; revisit if it escalates. Follow-up 2026-07-26: rechecked — the original three /24s are gone from current logs entirely (that specific campaign appears to have moved on). apache-noscript shows normal jail activity (42 total bans, 3 currently banned) from ordinary repeat-offender IPs already caught by per-IP thresholds, not a new distributed single-shot pattern. No escalation; still fine to leave as-is.

• No real nameserver redundancy: 2026-07-19, ns1.jayfield.org and ns2.jayfield.org were registered as this domain's NS records at the registrar. Delegation checked and confirmed consistent — the .org registry (checked against two separate registry servers), this host's own authoritative zone, and both ns1/ns2 themselves (queried directly, aa flag set, matching SOA serial 2022032214 at the time of that check — serial has since moved to 2022032216 from the SPF/DMARC/DKIM fix below, unrelated to NS/glue) all agree. But both names glue to the same IP (89.58.8.149, i.e. this same host) — RFC 2182 §3 recommends at least two nameservers not on the same network segment, and this setup has zero real redundancy: if alpha goes down, both "nameservers" disappear together. Tried unregistering ns2 at the registrar since it wasn't adding real redundancy anyway — confirmed the ISP/registrar requires a minimum of two NS records at the delegation layer and rejected the change, so both stay registered as-is. Redundancy risk remains open; the only real fix is a second nameserver on genuinely separate infrastructure.

(Previously resolved: legacy /portainer proxy paths on the default vhost and jayfield.org, stale DNS records test, kacke, pupsi, mausi, hase, super, and a disabled nextcloud.conf vhost plus its a2dissite marker file — a leftover from a Nextcloud install that was disabled at some point but never actually deployed (no /var/www/nextcloud, no package, no DB, no cert) — all cleaned up. Also: mnt-vlda-01.mount's recurring boot failure — root-caused to a dormant WireGuard tunnel (alpha_at_moon, routing the same 192.168.22.0/24 home-LAN range the CIFS target lived on) — resolved 2026-07-19 by removing all three together rather than reviving the tunnel: wireguard/wireguard-tools purged, nginx + libnginx-mod-rtmp purged (an unrelated, previously-uncatalogued RTMP streaming service on :1935, found via the same diff pass, also unused and removed), and samba/cifs-utils purged along with the fstab entry and stored CIFS credentials. Configs backed up to /root/removed-configs-backup/ on the server before deletion, in case any of the three are wanted again. Zero failed systemd units after. Also: KeePass vault sync migrated 2026-07-19 from local Apache-WebDAV + SFTP-upload to Nextcloud's built-in WebDAV — see "Password vault sync" above. Also: SPF, DMARC, and both DKIM (2021/2022._domainkey) TXT records in db.jayfield.org were silently broken — root cause was unquoted TXT rdata in the zone file: BIND's zone parser treats an unquoted ; as a start-of-comment (so v=DMARC1; p=reject; was being served as just "v=DMARC1", and both DKIM keys as just "v=DKIM1" with the k=rsa; p=<key> portion silently dropped — meaning DKIM signature verification had been failing for all outbound mail), and an unquoted multi-token SPF line was being split into multiple concatenated character-strings instead of one, merging in an unrelated google-site-verification= token and invalidating the SPF syntax. Fixed 2026-07-19 by quoting every TXT value (splitting the long DKIM keys across two <=255-byte quoted strings each, since a single character-string is capped at 255 bytes — see named-checkzone's "text string too long" if this is missed) and separating google-site-verification into its own TXT record. SOA serial bumped 2022032214 → 2022032216. Verified directly against the authoritative server (dig ... @89.58.8.149) post-reload; public resolvers catch up as the old records' TTLs (up to 3600s) expire. Also 2026-07-19: audited for phpMyAdmin/Adminer/webmin/cPanel/Plesk/ ISPConfig/phpPgAdmin residue (dpkg logs, rc-state packages, filesystem-wide filename search, Apache configs, MySQL users/DBs, crontabs, Docker images/volumes) — clean; the only name hits were stock fail2ban/augeas package files, not evidence of a prior install. Two stray public files (ShowKeyPlus_x64.zip, MediaCreationTool1909.exe — unrelated pirated Windows tooling, plus an empty test file) were found sitting in /var/www/html/pub, publicly served off the default vhost; moved to /home/jens/pub and the now-empty pub/ directory removed from the web root. Also: SSH hardened (PermitRootLogin no, PasswordAuthentication no, key-only) and the sshd fail2ban jail's findtime widened from the 30-minute default to 24h (maxretry 3→4, bantime 4h→24h) after 47.76.192.176 was found grinding root/jayfield SSH logins roughly every 33 minutes for most of a day — just outside the old 30-minute window, so it never accumulated enough strikes to get banned; see TODO.md and SETUP.md §1/§3 for the fix and the same evasion check applied to every other jail (only one other pattern turned up — see "Open items" above). Also: Portainer's :9443 UI, previously published directly to the public interface on its own self-signed cert, became unreachable in HSTS-enforcing browsers (MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT, no click-through) once the includeSubDomains HSTS policy above started pinning alpha.jayfield.org to trusted-cert-only HTTPS on every port — not a MITM, a self-inflicted interaction between two otherwise-correct pieces of config. Fixed 2026-07-20 by moving it behind an Apache reverse proxy on its own subdomain, portainer.jayfield.org, same pattern as web.jayfield.org/cloud.jayfield.org; see "Portainer" above and TODO.md for the full details. Also: an audit of which public hostnames require credentials (prompted by the Portainer/HSTS work above) turned up an OS command injection in the dyndns update scripts (/var/www/dyndns/nsupdate*.php built a shell command by directly interpolating unsanitized $_GET input into exec()) — gated by the vhost's Basic Auth, so not anonymously exploitable, but a real privilege escalation for anyone with valid dyndns credentials. Fixed 2026-07-20 by rewriting the update path to use proc_open() with array-form arguments (no shell involved at all) plus strict hostname/IP validation. That same audit also surfaced (a) a regression from the 2026-07-19 TSIG key hardening that had silently broken legitimate dyndns updates — fixed with a POSIX ACL scoped to just that key file, not group membership, since the bind group also owns rndc.key — and (b) that access logging for every real vhost on this host is currently a no-op, and (c) that the dyndns app has no real per-user auth binding. (b) fixed 2026-07-25 — see TODO.md for the root cause and the per-vhost CustomLog fix. (c) fixed 2026-07-26: every dyndns hostname now has its own dedicated .htpasswd account and TSIG key, with BIND's update-policy (not just an app-level check) refusing any update where the authenticated user doesn't match the requested hostname — see TODO.md for the full rollout, including the now-retired shared key and legacy fallback.