commit 92d871f2da2031c3f4bed40a8ad3740322381f40 Author: Jens Ahrensfeld Date: Sun Jul 19 23:09:27 2026 +0200 Initial docs: alpha.jayfield.org server documentation Records the host's service inventory, setup runbook, mail account list, sync plan, package diff vs. clean install, and hardening TODOs, including today's SSH hardening (key-only, no root login) and the fail2ban sshd findtime fix for a low-and-slow brute-force evasion pattern. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01RdTyvEJkfNLDVAWt8WQ6X9 diff --git a/DIFF.md b/DIFF.md new file mode 100644 index 0000000..6e3c0b4 --- /dev/null +++ b/DIFF.md @@ -0,0 +1,220 @@ +# alpha.jayfield.org — Diff vs. a Clean Ubuntu 22.04 Install + +Generated 2026-07-19. Methodology and caveats up front; findings below, +organized in three tiers of decreasing mechanical certainty and increasing +manual judgment. + +**Update, same day**: three of the findings below (RTMP/`nginx`, the +`wireguard` tunnel, and `samba`/`cifs-utils`) were removed shortly after +this diff was run — see the annotations inline and `README.md`'s known +quirks for the current state. Left in place here as a record of what was +found and why it was removed; this file is a point-in-time investigation, +not a live inventory (`README.md` is the live inventory). + +**Baseline used**: the official `jammy-server-cloudimg-amd64` manifest +(`cloud-images.ubuntu.com/jammy/current/`) for package comparisons, and +freshly-downloaded `.deb`s from the exact same Ubuntu archive this host +uses (matching installed versions exactly — e.g. `bind9 9.18.39-0ubuntu0.22.04.4`, +`postfix 3.6.4-1ubuntu1.4`) for config-file comparisons. One honest caveat: +several signs suggest this host was actually provisioned from the +interactive Server ISO installer, not the cloud image (`installation-report`, +`tasksel`, `popularity-contest`, and `language-pack-en` are all manually +installed — none of these appear on a cloud-init-provisioned instance). +The cloud image manifest is still the best available reference point, but +treat tier 1's line count as approximate, not exact. + +--- + +## Tier 1 — Package diff + +968 packages installed vs. 600 in the baseline manifest. Most of that gap +is transitive dependencies (installing `apache2` alone pulls in a dozen +supporting libraries nobody explicitly chose). Filtering to packages that +are both (a) not in the baseline and (b) marked `apt-mark showmanual` +(i.e. an actual administrative decision, not a dependency) gives **122 +packages** — the real, meaningful list. Grouped by what they're for: + +| Purpose | Packages | +|---|---| +| DNS | `bind9` | +| Mail | `postfix`, `postfix-mysql`, `dovecot-core`, `dovecot-imapd`, `dovecot-lmtpd`, `dovecot-managesieved`, `dovecot-mysql`, `dovecot-sieve`, `rspamd`, `mailutils`, `libmailutils6` | +| Database | `mariadb-server` (+ `mariadb-server-10.3`/`galera-3` — stale leftovers, see note below) | +| Web | `apache2`, `php8.0` + extensions, `libapache2-mod-php7.4`, `libapache2-mod-php8.0`, ~~`nginx`~~ (see Tier 2 — was **not** a web server here; **removed** same day, was unused RTMP) | +| TLS | `certbot`, `python3-certbot-apache` | +| Docker | `docker-ce`, `docker-ce-cli`, `docker-buildx-plugin`, `docker-compose-plugin`, `containerd.io` | +| Security | `fail2ban` | +| File sharing | ~~`samba`, `cifs-utils`~~ (→ the `vlda-01` CIFS mount; **removed** same day along with the dead tunnel below) | +| **VPN — not previously documented** | ~~`wireguard`~~ (**removed** same day — dormant, root-caused the CIFS failure, config backed up to `/root/removed-configs-backup/`) | +| Utilities | `at`, `net-tools`, `unzip`, `lz4`, `ltrace`, `bsdmainutils`, `resolvconf`, `qemu-guest-agent` | +| ISO-installer artifacts | `installation-report`, `tasksel`, `popularity-contest`, `language-pack-en`, `language-selector-common` | + +**Stale/leftover packages worth a cleanup pass**: `mariadb-server-10.3`, +`mariadb-client-10.3`, `galera-3`, `libssl1.1`, `libicu66`, `php8.0` (vs. +the active `php7.4` module), `libperl5.30` — all remnants of the +18.04→20.04→22.04 upgrade history already noted in `README.md` (same +lineage as the `python3.8`/certbot issue documented there). Not actively +harmful, just disk/attack-surface bloat from packages no longer in active +use. Candidate for a future `apt autoremove` pass — worth checking what's +actually still linked against before purging. + +**24 packages "missing" vs. baseline are not removals** — they're +expected absences for a BIOS-booted QEMU VM with no physical hardware: +`grub-efi-*`, `shim-signed`, `mokutil`, `efibootmgr` (this host boots via +legacy BIOS/`bios_grub`, confirmed in the disk-resize work — never needed +EFI tooling), plus `libtss2-*` (TPM), `fuse3`/`libntfs-3g89` (physical +filesystem drivers), `systemd-hwe-hwdb` (hardware enablement) — none of +this is relevant to a cloud VM regardless of who administers it. + +--- + +## Tier 2 — Config file diff (live file vs. pristine package default) + +Real `diff -u` output against genuinely pristine files pulled from the +matching package version in Ubuntu's archive — not reconstructed from +memory. Only the files with a real static default to diff against are +listed; Postfix's `main.cf`/`master.cf` and Dovecot's `dovecot.conf` are +**not** static conffiles (Postfix generates them via debconf at install +time; Dovecot's shipped file is a first-install template only), so "diff" +for those is a semantic comparison, covered in `SETUP.md` instead. + +### `/etc/bind/named.conf.options` +Exactly what's documented in `TODO.md`/`README.md` — `version "unknown"`, +the `rate-limit` block, and the `forwarders` block removed in favor of +full recursion (with an inline comment explaining why). Plus pre-existing +(2022) additions: `allow-transfer`, `allow-query { any; }`, +`allow-recursion { trusted; }`, `querylog yes`, the `trusted` ACL. + +### `/etc/ssh/sshd_config` +- `Port 10022` (from `#Port 22`) +- `PasswordAuthentication yes` and `PermitRootLogin yes` **explicitly set** — + the shipped default leaves both commented out (relying on OpenSSH's own + safer compiled-in defaults). Flagged already in `SETUP.md` §1 as a + hardening gap worth reconsidering. +- `ChallengeResponseAuthentication no` — uses the pre-8.7 directive name; + current templates use `KbdInteractiveAuthentication`. Cosmetic, but a + dating signal (this file predates a later OpenSSH directive rename). +- **New finding**: an entire `Match Group sftp` block — chrooted, + no-shell SFTP-only access (`ChrootDirectory /var/sftp/%u`, tunneling and + forwarding all disabled). Three accounts use it: `ftpuser`, `ftp_jens`, + `ftp_alex` (`/var/sftp/{ftpuser,ftp_jens,ftp_alex}/`). This is a + legitimate, separate access mechanism from `jens`'s full shell account — + not documented anywhere before this diff. + +### `/etc/systemd/resolved.conf` +Exactly the `DNS=127.0.0.1` change documented in `README.md`'s resolver +section, plus pre-existing `DNSSEC=allow-downgrade`, `LLMNR=no`, etc. + +### `/etc/apache2/apache2.conf` +- ``: `AllowOverride all` (default: `None`) — permits + `.htaccess` files to override server config anywhere under `/var/www`. + Broader than the stock posture; worth checking whether any `.htaccess` + files actually exist and rely on this before tightening it. +- A custom `fail2ban` `LogFormat` — ties into the `apache-auth` jail. +- **New finding, worth a closer look**: a global block appended at the + end of the file — + ``` + + SetHandler application/x-httpd-php + + ``` + The regex `\.* ` (zero or more literal dots, unanchored) matches nearly + any filename, including ones with zero dots. Applied globally (outside + any `` scope), this routes essentially all requests through + the PHP handler, not just `.php` files. Whether this has real + consequences depends on what mod_php does with non-PHP content (likely + passes it through as literal text), but it's a broad, unusual, and + previously-undocumented directive — worth understanding the original + intent before deciding whether to scope it down to `\.php$`. + +### `/etc/fail2ban/jail.conf` +This host edits `jail.conf` directly rather than using `jail.local` for +overrides — the intended pattern is to leave the vendor file untouched +and override in `jail.local` so future package upgrades don't silently +clobber local changes. Real customizations found: `bantime` 10m→**4h**, +`findtime` 10m→**30m**, `maxretry` 5→**3**, `port = 10022` on the +`sshd`/`dropbear`/`selinux-ssh` jails (matching the moved SSH port), +`sender = root@jayfield.org`, and several jails explicitly `enabled = true` +(`sshd`, `postfix`, `postfix-rbl`, `dovecot`, `sieve`, `pam-generic`, and +others) that ship disabled by default. The remaining diff noise (action- +syntax `%(banaction)s[...]` reformatting, comment typo fixes) is from +fail2ban's own upstream template evolving between minor releases, not a +local change. + +### `/etc/nginx/nginx.conf` — **removed 2026-07-19, same day as this diff** +Confirmed Tier 1's finding cleanly: the entire `http {}` block from the +stock config was **removed** — no web serving capability at all — and +`include /etc/nginx/rtmp.conf;` was the only addition. This nginx instance +existed solely to run `libnginx-mod-rtmp` as a live-streaming ingest server +(`rtmp.conf`: `listen 1935; application live { live on; record off; }`, +no publish authentication configured). Entirely separate from the +Docker `web` (nginx) container and from Apache — no overlap, no conflict, +just a second, unrelated nginx process that had gone uncatalogued. No +active connections or recent use found before removal; purged +(`nginx nginx-core nginx-common libnginx-mod-rtmp`) along with `/etc/nginx`. + +### `/etc/apache2/ports.conf` +No diff — matches stock exactly (Apache's own `Listen 80`/mod_ssl's +`Listen 443` addition). + +--- + +## Tier 3 — Changes with no package-level "default" to diff against + +Not mechanically diffable — either data (not config), or infrastructure +this host's package manager doesn't track at all. + +- **Disk partitioning**: root partition was 9.7GB of a 172GB disk for + years (see `README.md`) — grown to 157GB this session via + `growpart`/`resize2fs`. No "default" partition layout to diff against; + this is provisioning-time state. +- **DNS zone content** (`/etc/bind/db.jayfield.org`): all records — + there's no baseline zone file to diff against, it's 100% custom data. +- **MySQL `vmail` database**: schema + data for mail accounts/aliases/domains + (see `SETUP.md` §3 for the schema). Data, not config. +- **TLS certificates**: 7 Let's Encrypt certs, all custom to this domain. +- **Docker layer**: `portainer`, `web` (nginx), and the Nextcloud stack + (`nextcloud`, `nextcloud-db`, `nextcloud-redis`) — none of this exists + in a base install; it's entirely additive infrastructure documented in + `README.md`. +- **WireGuard — removed 2026-07-19, same day as this diff** + (`/etc/wireguard/alpha_at_moon.conf`): a client tunnel named + "alpha at moon" — `Address = 10.8.0.8/24`, peer endpoint + `vpn.dyndns.jayfield.org:51820`, `AllowedIPs = 10.8.0.0/24, + 192.168.22.0/24`. Was inactive (`wg-quick@wg0` not running) at the time + of this diff. Almost certainly *why* the `mnt-vlda-01.mount` failure + documented in `README.md` happened — `vlda-01` (the CIFS target) was + presumably only reachable via this same `192.168.22.0/24` range. + Decision made: rather than reviving the tunnel, removed it along with + the dead CIFS mount and unused Samba packages. Config backed up to + `/root/removed-configs-backup/alpha_at_moon.conf.bak` on the server + (`600 root:root`) in case the tunnel is wanted again later. +- **SFTP chroot jail** (`/var/sftp/`): three accounts (`ftpuser`, + `ftp_jens`, `ftp_alex`) in the `sftp` group, each chrooted to their own + directory with no shell — a controlled file-drop mechanism separate + from `jens`'s admin SSH access. Purpose/owners of these accounts not + established from config alone. +- **Users, groups, sudoers**: not diffed — no baseline "clean install" + user set to compare against beyond the default `ubuntu`/first-boot user, + which this host doesn't appear to still have. +- **Cron jobs**: `certbot renew` (weekly), Nextcloud's `cron.php` (every + 5 min) — both documented in `README.md`, both additive, no baseline. + +--- + +## Summary: what this diff actually surfaced + +Beyond confirming the changes already tracked in `README.md`/`TODO.md`, +this exercise found **four previously-undocumented things**: + +1. ~~A live RTMP streaming server (`nginx` + `libnginx-mod-rtmp` on + `:1935`)~~ — **removed** same day; unused, no auth on the publish + endpoint, no reason found to keep it. +2. ~~A dormant WireGuard tunnel to a home network~~ — **removed** same + day, along with the CIFS mount it was almost certainly gating and the + Samba packages that supported it. Config backed up first. +3. A chrooted SFTP-only access path for three accounts, separate from the + main admin login. Left as-is — appears to be legitimate, ongoing + infrastructure, not a stray leftover. +4. Two Apache config items worth a second look: `AllowOverride all` on + `/var/www`, and a broad `` PHP handler that's wider + than it probably needs to be. Still open — not yet addressed. diff --git a/MAIL-ACCOUNTS.md b/MAIL-ACCOUNTS.md new file mode 100644 index 0000000..04e5618 --- /dev/null +++ b/MAIL-ACCOUNTS.md @@ -0,0 +1,122 @@ +# alpha.jayfield.org — Adding Email Accounts (How-To) + +Accounts, aliases, and domains for the mail stack (Postfix + Dovecot) all +live in one place: the `vmail` MySQL database on this host. There's no +admin UI — everything below is a `mysql` insert plus one hash-generation +command. See `SETUP.md` §3 for how this schema was built in the first +place; this doc is just the day-to-day "add a mailbox" recipe. + +**No service restart is ever needed for account changes.** Postfix and +Dovecot query MySQL live on every delivery/login (`auth_cache_size` is +unset/disabled on this host — verified in `/etc/dovecot/conf.d/10-auth.conf`) +so a new row is usable immediately. + +## 1. Prerequisite: the domain must already exist + +```bash +sudo mysql vmail -e "SELECT * FROM domains;" +``` +Currently just `jayfield.org`. If you ever need a second domain, insert it +here first (`INSERT INTO vmail.domains (domain) VALUES ('newdomain.tld');`) +— but note that's the *easy* part; a real second domain also needs its own +DNS zone/MX/SPF/DMARC records and DKIM selector (see `SETUP.md` §2/§3), a +TLS cert covering it, and its own `mynetworks`/`inet_interfaces` +consideration. Out of scope for just adding a mailbox — this doc assumes +`jayfield.org`. + +## 2. Generate the password hash + +Never put a plaintext password in the `accounts` table — hash it first: +```bash +sudo doveadm pw -s SHA512-CRYPT +``` +(You'll be prompted twice, interactively — don't pass the password as a +CLI argument, it'd land in shell history.) Copy the full +`{SHA512-CRYPT}$6$...` output; that's the exact string that goes in the +`password` column. + +You may see `Error: net_connect_unix(/run/dovecot/stats-writer) failed: +Permission denied` printed before the hash — that's `doveadm` trying to +report stats, unrelated to the hash it prints right after; harmless, not a +sign the hash is wrong. + +## 3. Insert the account + +```bash +sudo mysql vmail -e "INSERT INTO accounts (username, domain, password, quota, enabled, sendonly) VALUES ('newuser', 'jayfield.org', '{SHA512-CRYPT}\$6\$...', 2048, 1, 0);" +``` +- `quota` is in **MB** (existing accounts use `2048` = 2GB — match that + unless there's a reason not to). +- `enabled = 1` — required for the account to authenticate or receive mail + at all. Setting it to `0` disables the account without deleting it (see + §6). +- `sendonly = 0` — normal mailbox: can send *and* receive. Set to `1` only + for a send-only address that should never accept incoming mail (see §5). + +That's it — no mailbox directory to create by hand. Dovecot auto-creates +`/var/vmail/mailboxes/jayfield.org//mail/` (owned `vmail:vmail`, +`700`) the first time the account logs in or receives mail. + +## 4. Client configuration (no webmail — this stack is IMAP/SMTP only) + +| Setting | Value | +|---|---| +| IMAP (recommended) | `mail.jayfield.org` : `993`, implicit TLS | +| IMAP (STARTTLS) | `mail.jayfield.org` : `143` | +| SMTP submission | `mail.jayfield.org` : `587`, STARTTLS + SASL (PLAIN) auth required | +| Username | `@jayfield.org` (full address, not just the local part) | +| Password | the plaintext password you hashed in §2 | +| Port 25 | **not for clients** — server-to-server only, no SASL, no relay (`reject_unauth_destination`) | + +`disable_plaintext_auth = yes` is set — clients must actually negotiate TLS +first; a client configured for plaintext IMAP/SMTP on the wrong port will +just fail to auth, not silently send credentials in the clear. + +## 5. Send-only accounts (`sendonly = 1`) + +Postfix's `recipient-access.cf` query rejects any inbound mail to a +`sendonly=1` address at `RCPT TO` time (`450`/`550` before it's even +accepted), while the account can still authenticate via SASL on `587` and +send outbound. Use this for something like a monitoring/alerting sender +address that should never actually receive replies. + +## 6. Aliases (forward one address to another mailbox) + +Aliases don't need their own mailbox or password — they're a pure +redirect, resolved at `RCPT TO` time via `aliases.cf`: +```bash +sudo mysql vmail -e "INSERT INTO aliases (source_username, source_domain, destination_username, destination_domain, enabled) VALUES ('sales', 'jayfield.org', 'jens', 'jayfield.org', 1);" +``` +This host already has `postmaster`/`hostmaster`/`webmaster`/`wlan` → +`jens` as standing aliases (RFC-required roles + a house one) — check +`SELECT * FROM aliases;` before adding a new one to avoid duplicates. + +## 7. Disabling / removing an account + +Prefer disabling over deleting — it's reversible and doesn't touch the +Maildir: +```bash +sudo mysql vmail -e "UPDATE accounts SET enabled = 0 WHERE username = 'olduser' AND domain = 'jayfield.org';" +``` +This blocks both authentication and inbound delivery immediately (both +`accounts.cf` and `recipient-access.cf` filter on `enabled = true`). The +actual mail data stays on disk at +`/var/vmail/mailboxes/jayfield.org/olduser/` until you decide to remove it +by hand — nothing auto-deletes it. + +## 8. Verify a new account end-to-end + +Don't trust the INSERT alone — confirm both directions actually work, +same discipline as `SETUP.md` §7: + +```bash +# IMAP auth works +doveadm auth test newuser@jayfield.org + +# Watch a real inbound + outbound test the same way §7 of SETUP.md describes: +# send TO newuser@jayfield.org from an external provider, confirm in +# /var/log/mail.log: +# dovecot: lmtp(newuser@jayfield.org)<...>: sieve: msgid=<...>: stored mail into mailbox 'INBOX' +# then send FROM the new account via a real mail client on 587, confirm: +# postfix/submission/smtpd[...]: sasl_method=PLAIN, sasl_username=newuser@jayfield.org +``` diff --git a/README.md b/README.md new file mode 100644 index 0000000..413f9b5 --- /dev/null +++ b/README.md @@ -0,0 +1,322 @@ +# alpha.jayfield.org — Web Services Overview + +Host: `alpha.jayfield.org` (`89.58.8.149`), Ubuntu 22.04 LTS. +SSH: `ssh -p 10022 jens@alpha.jayfield.org` + +All administration in this repo's notes is performed remotely against +this host over SSH (port `10022`) — there is no local checkout of any +service config; changes are made directly on the server and recorded +here afterward. + +This host runs two layers of web services: native services managed by +system Apache, and containerized services managed by Docker (visible in +Portainer). Apache reverse-proxies into the Docker layer where needed. + +## Native services (systemd / apt) + +| Service | Role | +|---|---| +| `apache2` (2.4.52) | Terminates HTTP/HTTPS for all vhosts, reverse-proxies to backends (Docker containers, rspamd, Portainer) | +| `bind9` | Authoritative DNS for the `jayfield.org` zone (`ns1`/`ns2.jayfield.org` are both this host); also the box's own recursive resolver — see below | +| `docker` | Runs the containerized web services (see below) | +| `certbot` (apt, 1.21.0) | Let's Encrypt cert issuance/renewal, via `certbot.timer` (systemd) | +| Postfix (`master`) | SMTP, ports 25/587, on the public IP + localhost | +| Dovecot | IMAP/IMAPS (143/993) + ManageSieve (4190) | +| rspamd | Spam filtering for the mail stack, localhost-only (11332–11334), proxied at `mail.jayfield.org/rspamd` | +| redis-server | Localhost-only (6379); backs rspamd's Bayes classifier, history, and reputation caching | + +Python: `python3.10.12` (fixed from a stalled `python3.8` leftover of an +incomplete 20.04→22.04 upgrade — this was breaking apt's certbot via a +`pytz`/`zoneinfo` incompatibility). + +## System users + +| User | uid | Home | Purpose | +|---|---|---|---| +| `jens` | 1001 | `/home/jens` | Primary sudo admin account — all administration in this repo is done as this user over SSH | +| `alex` | 1005 | `/home/alex` | Secondary, non-sudo personal account (added 2026-07-19) | + +Both also exist as separate Nextcloud accounts (see "Password vault sync" +below) and as chrooted, SFTP-only accounts `ftp_jens`/`ftp_alex` under +`/var/sftp/` — the latter currently password-locked, not deleted (see +below). + +## DNS resolver architecture + +Fixed 2026-07-19: the box's own outbound DNS resolution (used by Postfix's +`postscreen` for DNSBL lookups, and by every other service) previously +forwarded through the hosting provider's shared resolvers +(`46.38.225.230`/`46.38.252.230`). Confirmed via a live test: GMX's own +mail servers (`212.227.15.15`/`.19`, verified via reverse DNS as +`mout.gmx.net`) were being **dropped by `postscreen` on a false positive** +— Spamhaus's DNSBL was returning `127.255.255.254`, its documented +rate-limit sentinel ("this querying resolver has exceeded our free-tier +volume"), not a real listing. The shared resolver was almost certainly +pooling query volume from other tenants of the same hosting provider, +pushing it over Spamhaus's threshold. + +Fix: +- Removed the `forwarders {}` block from `named.conf.options` — `bind9` + now does full iterative/recursive resolution itself (root hints → + TLD → authoritative) for `trusted` clients (`localhost` + `localnets`) + instead of relaying to the shared upstream. +- Repointed `/etc/systemd/resolved.conf`'s `DNS=` from the shared + provider IPs to `127.0.0.1` (i.e. `bind9`), then `systemctl restart + systemd-resolved`. The `127.0.0.53` stub listener and `/etc/resolv.conf` + symlink were left untouched, so Docker's DNS-detection logic (which + special-cases the stub) is unaffected. +- (An `unbound` install was tried first as a separate local resolver, but + `bind9` already listens on `127.0.0.1:53`/`[::1]:53` — `listen-on-v6 { + any; }` plus BIND's default of listening on all IPv4 interfaces — so + they couldn't coexist on the same port. Reusing `bind9` for both roles + was simpler than fighting that conflict.) + +Verified: both GMX IPs now correctly resolve as **not listed**; +`postscreen` now `PASS`es them. A real end-to-end test email +("Claude email test #01") delivered successfully to both `jens@` and +`otto2022@jayfield.org` after the fix. + +**Known caveat**: direct recursive resolution occasionally hits a single +slow/unresponsive server somewhere in a delegation chain, where a +well-provisioned forwarding resolver's warm cache usually wouldn't. Seen +once during testing — a transient PTR lookup timeout caused one retry to +soft-fail (`450`, not `550`) before a second retry succeeded seconds +later. Not expected to be a recurring problem, but worth knowing if a +one-off `450 4.7.25 Client host rejected` shows up in `mail.log`. + +## Apache virtual hosts + +| ServerName | :80 | :443 | Notes | +|---|---|---|---| +| `dyndns.jayfield.org` | → redirect to :443 | Basic Auth (`Restricted Content`), `DocumentRoot /var/www/dyndns` (PHP) | Also the effective **default vhost** for unmatched hostnames (e.g. bare `alpha.jayfield.org` falls through here) | +| `jayfield.org` | → redirect to :443 | Static site (`/var/www/html`) | Previously had no `:80` vhost of its own — fell through to the `dyndns` default and redirected to the wrong domain; fixed | +| `www.jayfield.org` | → redirect to :443 | Static site (`/var/www/html`) | Renewal fixed to use the `apache` certbot plugin (was `standalone`, which required stopping Apache) | +| `mail.jayfield.org` | → redirect to :443 | Webmail-adjacent, proxies `/rspamd` → `http://localhost:11334/` | | +| `web.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `web` (nginx) container** on `127.0.0.1:8081` | New — see below | +| `cloud.jayfield.org` | → redirect to :443 | Reverse-proxies to the **Docker `nextcloud` container** on `127.0.0.1:8082`, sets `X-Forwarded-Proto: https` | Nextcloud — see below | +| default (`_default_:443`, `000-default` on :80) | Apache's stock placeholder page | — | | + +All Let's Encrypt certs (7 total, one per hostname/group) auto-renew via +`certbot.timer`; verified with `certbot renew --dry-run`. + +All 6 domain `:443` blocks send `Strict-Transport-Security: max-age=63072000; +includeSubDomains` (`mod_headers`). Deliberately skipped on the +`default-ssl.conf` catch-all — it's the self-signed fallback for unmatched +SNI/Host headers, not a real trusted hostname a browser could pin. + +## Docker services (visible in Portainer) + +| Container | Image | Ports | Purpose | +|---|---|---|---| +| `portainer` | `portainer/portainer-ce:2.39.5` | `8000` (edge-agent tunnel, published), `9443` (HTTPS UI, published), `9000` (UI, internal only) | Docker management UI — `https://alpha.jayfield.org:9443` | +| `web` | `nginx:latest` | `127.0.0.1:8081` (localhost-only, not published externally) | Backing service for `web.jayfield.org`; only reachable through the Apache reverse proxy | +| `nextcloud` | `nextcloud:apache` | `127.0.0.1:8082` (localhost-only) | Nextcloud app (PHP+Apache bundled in the image); backing service for `cloud.jayfield.org` | +| `nextcloud-db` | `mariadb:lts` | internal only (`nextcloud_net`) | Dedicated DB for Nextcloud — kept separate from the host's own MySQL (which only holds `vmail`) | +| `nextcloud-redis` | `redis:alpine` | internal only (`nextcloud_net`) | File locking + distributed cache + PHP session storage for Nextcloud. Dedicated container rather than reusing the host's native `redis-server`, since that instance is bound to `127.0.0.1`/`::1` only with no auth — reusing it would have meant loosening a shared service's network exposure just to save one small container | + +Data persists in the `portainer_data` named volume, independent of the +container itself — updating Portainer means recreating the container +against a freshly pulled image while keeping this volume and the +`/var/run/docker.sock` bind mount; the old container is kept renamed +(not deleted) until the new one is verified healthy. + +Two early-test containers (`my-apache-app`, a stopped `httpd`, and +`clever_jepsen`, a stale `hello-world` container from initial Docker +setup) have been removed as cleanup. + +### `web.jayfield.org` request flow + +``` +Internet → Apache :80/:443 (web.jayfield.org vhost) + → Let's Encrypt TLS termination + → ProxyPass http://127.0.0.1:8081/ + → Docker container "web" (nginx) on the docker0 bridge +``` + +The container is intentionally bound to `127.0.0.1` only, so it cannot be +reached directly on port 8081 from the internet — Apache is the only path in. +This is the pattern to follow for any future container-backed site: publish +the container to `127.0.0.1:`, add an Apache vhost for the subdomain +that proxies to it, then run `certbot --apache -d ` to get HTTPS. + +### Nextcloud (`cloud.jayfield.org`) + +Minimal, data-hosting-focused deployment — core Files app only. No +Collabora/OnlyOffice, Talk, full-text search, external storage backends, +or LDAP/SSO; those can be layered on later without re-architecting. + +``` +Internet → Apache :80/:443 (cloud.jayfield.org vhost) + → Let's Encrypt TLS termination, X-Forwarded-Proto: https + → ProxyPass http://127.0.0.1:8082/ + → container "nextcloud" (nextcloud:apache) + ↕ private network "nextcloud_net" + → container "nextcloud-db" (mariadb:lts) + → container "nextcloud-redis" (redis:alpine) +``` + +- **DNS**: `cloud` A record added to `/etc/bind/db.jayfield.org` + (serial bumped 2022032213 → 2022032214). +- **Data volume**: bind-mounted at `/srv/nextcloud/data` on the host + (owned by uid/gid `33`, matching `www-data` both on the host and + inside the container) rather than an opaque named Docker volume — so + it's directly inspectable/backupable with normal tools (`du`, + `rsync`, `borg`). +- **App volume**: named Docker volume `nextcloud_html`, holds Nextcloud + core + `config.php`; survives image upgrades. +- **DB volume**: named Docker volume `nextcloud_db`. +- **Auto-install**: done via the image's env-var-driven installer + (`MYSQL_*`, `NEXTCLOUD_ADMIN_USER`/`NEXTCLOUD_ADMIN_PASSWORD`, + `NEXTCLOUD_TRUSTED_DOMAINS`, `TRUSTED_PROXIES`, + `OVERWRITEPROTOCOL=https`, `REDIS_HOST`) rather than leaving an + unconfigured setup wizard reachable on the public internet. +- **Credentials**: DB root/user passwords and the initial admin + password are in `/root/.nextcloud-credentials` on the server + (`600 root:root`) — not committed anywhere, not in this repo. +- **Cron**: Nextcloud's background-job mode is set to `cron` (`occ + background:cron`); a root crontab entry runs + `docker exec -u www-data nextcloud php -f /var/www/html/cron.php` + every 5 minutes. No sidecar cron container. +- **PHP limits**: image defaults are already generous for file hosting + (512M `memory_limit`/`upload_max_filesize`/`post_max_size`, unlimited + `max_execution_time`) — no tuning needed for this pass. + +### Password vault sync (KeePass, via Nextcloud) + +Each user's KeePass database now syncs through Nextcloud's own WebDAV +endpoint instead of a bespoke Apache-WebDAV + SFTP-upload pipeline: + +| User | Nextcloud account | Vault file | WebDAV URL | +|---|---|---|---| +| `jens` | `jens` | `sicher_jens.kdbx` | `https://cloud.jayfield.org/remote.php/dav/files/jens/sicher_jens.kdbx` | +| `alex` | `alex` | `sicher_alex.kdbx` | `https://cloud.jayfield.org/remote.php/dav/files/alex/sicher_alex.kdbx` | + +- Each `.kdbx` lives at the root of that user's Nextcloud Files storage + (`/srv/nextcloud/data//files/`, `640 www-data:www-data`), picked + up via `occ files:scan `. +- Verified 2026-07-19 with a raw WebDAV `PROPFIND`/`GET`/`PUT`/`GET` + round trip against the `alex` account — correct `Content-Length`, + matching SHA-256 before and after a write. (KeePassXC has no native + WebDAV client and there's no display on this host to drive a real + KeePass session, so this HTTP-level check is the practical + equivalent — it exercises the same verbs a WebDAV-aware sync client + or plugin would use.) +- A `600`-permission backup copy of each vault also still lives in the + respective user's home dir (`/home/jens/sicher_jens.kdbx`, + `/home/alex/sicher_alex.kdbx`) on this host. +- **Superseded setup**: an Apache `mod_dav_fs` container (`docker-apache2-php` / + `my-httpd` image, HTTP Basic Auth via `.htpasswd`) running on the admin + workstation used to serve the vault directly, with files uploaded to + this host via the chrooted `ftp_jens`/`ftp_alex` SFTP accounts under + `/var/sftp/`. That local container/image has been removed (it ran with + `--rm` and had no cron/systemd autostart, so nothing else depended on + it). The `ftp_jens`/`ftp_alex` accounts remain on this host (chroot and + files untouched) but are password-locked (`passwd -l`) since nothing + uses them anymore — reversible with `passwd -u` if SFTP is ever needed + again. + +## Open items / known quirks + +- **Intranet migration from `vlda-01` (files, Gitea, Jellyfin, Dogecoin + node/wallet)** — planning stage only, not started. See `SYNC-PLAN.md` + for the full per-service design, open questions, and build order. + +- **PTR/EHLO mismatch (workaround applied, real fix still open)**: the + PTR record for this host's IP (`89.58.8.149`) resolves to + `alpha.jayfield.org`, but Postfix's `myhostname` was `mail.jayfield.org` + (matching the MX record) — a mismatch some strict receivers penalize. + 2026-07-19: `myhostname` changed to `alpha.jayfield.org` to match the + existing PTR (verified live: `EHLO` → `250-alpha.jayfield.org`); see + SETUP.md's mail-stack section. The real fix is repointing the PTR to + `mail.jayfield.org` and then switching `myhostname` back — not done + yet, since PTR isn't controlled from this host's own DNS zone. Checked + 2026-07-19: the ISP's control panel has no self-service PTR/rDNS + option at all, so this needs a support ticket to the ISP asking them + to set the PTR for `89.58.8.149` to `mail.jayfield.org` (worth noting + in the ticket that the forward record already resolves correctly — + `mail.jayfield.org` → `89.58.8.149` — since some providers require + that before they'll set rDNS). If the ISP can't or won't do it, the + current `myhostname = alpha.jayfield.org` workaround is a fine + permanent setup as-is, not just a stopgap — PTR and EHLO already + agree, which is what actually matters for deliverability. +- **Distributed low-and-slow scan on `apache-noscript`**: found 2026-07-19 + while auditing fail2ban jails for the `sshd` timing-evasion pattern below + — a cluster of ~20 IPs across three `/24`s (`91.230.225.0/24`, + `185.192.71.0/24`, `185.92.25.0/24`) each probed exactly once, never + repeating. No per-IP `findtime`/`maxretry` tuning can catch this by + design — it needs subnet-level (CIDR) banning instead, which risks + blocking legitimate traffic sharing that block (e.g. carrier NAT). Left + as routine scan noise for now; revisit if it escalates. + +- **No real nameserver redundancy**: 2026-07-19, `ns1.jayfield.org` and + `ns2.jayfield.org` were registered as this domain's NS records at the + registrar. Delegation checked and confirmed consistent — the `.org` + registry (checked against two separate registry servers), this host's + own authoritative zone, and both `ns1`/`ns2` themselves (queried + directly, `aa` flag set, matching SOA serial `2022032214` at the time + of that check — serial has since moved to `2022032216` from the + SPF/DMARC/DKIM fix below, unrelated to NS/glue) all agree. + But both names glue to the *same* IP (`89.58.8.149`, i.e. this same + host) — RFC 2182 §3 recommends at least two nameservers not on the + same network segment, and this setup has zero real redundancy: if + `alpha` goes down, both "nameservers" disappear together. Tried + unregistering `ns2` at the registrar since it wasn't adding real + redundancy anyway — confirmed the ISP/registrar requires a minimum of + two NS records at the delegation layer and rejected the change, so + both stay registered as-is. Redundancy risk remains open; the only + real fix is a second nameserver on genuinely separate infrastructure. + +(Previously resolved: legacy `/portainer` proxy paths on the +default vhost and `jayfield.org`, stale DNS records `test`, `kacke`, +`pupsi`, `mausi`, `hase`, `super`, and a disabled `nextcloud.conf` vhost +plus its `a2dissite` marker file — a leftover from a Nextcloud install +that was disabled at some point but never actually deployed (no +`/var/www/nextcloud`, no package, no DB, no cert) — all cleaned up. +Also: `mnt-vlda-01.mount`'s recurring boot failure — root-caused to a +dormant WireGuard tunnel (`alpha_at_moon`, routing the same +`192.168.22.0/24` home-LAN range the CIFS target lived on) — resolved +2026-07-19 by removing all three together rather than reviving the +tunnel: `wireguard`/`wireguard-tools` purged, `nginx` + +`libnginx-mod-rtmp` purged (an unrelated, previously-uncatalogued RTMP +streaming service on `:1935`, found via the same diff pass, also unused +and removed), and `samba`/`cifs-utils` purged along with the `fstab` +entry and stored CIFS credentials. Configs backed up to +`/root/removed-configs-backup/` on the server before deletion, in case +any of the three are wanted again. Zero failed systemd units after. +Also: KeePass vault sync migrated 2026-07-19 from local Apache-WebDAV + +SFTP-upload to Nextcloud's built-in WebDAV — see "Password vault sync" +above. Also: SPF, DMARC, and both DKIM (`2021`/`2022._domainkey`) TXT +records in `db.jayfield.org` were silently broken — root cause was +**unquoted TXT rdata in the zone file**: BIND's zone parser treats an +unquoted `;` as a start-of-comment (so `v=DMARC1; p=reject;` was being +served as just `"v=DMARC1"`, and both DKIM keys as just `"v=DKIM1"` with +the `k=rsa; p=` portion silently dropped — meaning DKIM signature +verification had been failing for all outbound mail), and an unquoted +multi-token SPF line was being split into multiple concatenated +character-strings instead of one, merging in an unrelated +`google-site-verification=` token and invalidating the SPF syntax. +Fixed 2026-07-19 by quoting every TXT value (splitting the long DKIM +keys across two `<=255`-byte quoted strings each, since a single +character-string is capped at 255 bytes — see `named-checkzone`'s +"text string too long" if this is missed) and separating +`google-site-verification` into its own TXT record. SOA serial bumped +`2022032214` → `2022032216`. Verified directly against the authoritative +server (`dig ... @89.58.8.149`) post-reload; public resolvers catch up +as the old records' TTLs (up to 3600s) expire. +Also 2026-07-19: audited for phpMyAdmin/Adminer/webmin/cPanel/Plesk/ +ISPConfig/phpPgAdmin residue (dpkg logs, `rc`-state packages, filesystem-wide +filename search, Apache configs, MySQL users/DBs, crontabs, Docker +images/volumes) — clean; the only name hits were stock `fail2ban`/`augeas` +package files, not evidence of a prior install. Two stray public files +(`ShowKeyPlus_x64.zip`, `MediaCreationTool1909.exe` — unrelated pirated +Windows tooling, plus an empty `test` file) were found sitting in +`/var/www/html/pub`, publicly served off the default vhost; moved to +`/home/jens/pub` and the now-empty `pub/` directory removed from the web +root. Also: SSH hardened (`PermitRootLogin no`, `PasswordAuthentication no`, +key-only) and the `sshd` fail2ban jail's `findtime` widened from the +30-minute default to 24h (`maxretry` 3→4, `bantime` 4h→24h) after +`47.76.192.176` was found grinding root/`jayfield` SSH logins roughly every +33 minutes for most of a day — just outside the old 30-minute window, so it +never accumulated enough strikes to get banned; see `TODO.md` and +`SETUP.md` §1/§3 for the fix and the same evasion check applied to every +other jail (only one other pattern turned up — see "Open items" above).) diff --git a/SETUP.md b/SETUP.md new file mode 100644 index 0000000..5e186be --- /dev/null +++ b/SETUP.md @@ -0,0 +1,732 @@ +# alpha.jayfield.org — Fresh-Install Setup Guide + +This is a from-scratch runbook for reproducing this host: Ubuntu 22.04 LTS +running authoritative DNS, a full mail stack, Apache-fronted web services, +and a Docker layer for containerized apps (Portainer, Nextcloud). It's +written in build order — each section depends on the ones before it. + +Reference: `README.md` documents the *current, live* state of this exact +server. This file documents *how to get there* from nothing. When the two +disagree, `README.md` is the source of truth for what's actually running. + +--- + +## 0. Before you start + +- **VPS with a real, stable public IPv4** (this host: `89.58.8.149`, static + netplan config, not DHCP) and ideally IPv6. You'll need reverse DNS (PTR) + for the mail IP pointed at your mail hostname *before* sending real mail — + ask your host to set it. **Don't assume a self-service option exists**: + this host's provider control panel has no PTR/rDNS field at all, so it + had to go through a support ticket instead (see §3's `myhostname` caveat + for the interim workaround this host is running while that's pending). +- **Partition the disk fully at provision time.** This host originally had a + 9.7GB root partition on a 172GB disk — ~161GB sat unallocated for years + before we caught it. If your provisioner's cloud image under-partitions, + fix it immediately: + ``` + sudo growpart /dev/sda 3 # extend the partition into free space + sudo resize2fs /dev/sda3 # grow the filesystem to fill it (online, no reboot) + ``` + Verify with `df -h /` and `sudo parted /dev/sda unit GB print free`. +- **Register the domain and delegate DNS to this host** before step 2 — + you need `ns1`/`ns2` NS records pointed here at the registrar for the + zone to actually resolve publicly. +- A non-root sudo user for all administration (this host: `jens`). Don't + do the following steps as root directly. + +--- + +## 1. Base system + +```bash +sudo apt update && sudo apt full-upgrade -y +sudo timedatectl set-timezone Europe/Berlin # match your locale +sudo apt install -y unattended-upgrades +sudo dpkg-reconfigure -plow unattended-upgrades +``` + +### SSH + +Move SSH off the default port to cut down on background credential-stuffing +noise (this host uses `10022`): + +``` +# /etc/ssh/sshd_config +Port 10022 +``` + +Prefer `PermitRootLogin no` and key-only auth (`PasswordAuthentication no`) +from the start, then add `fail2ban`'s `sshd` jail (already covered in §3) +as defense-in-depth rather than the only line of defense: + +``` +# /etc/ssh/sshd_config +PermitRootLogin no +PasswordAuthentication no +PubkeyAuthentication yes +``` + +(Fixed on this host 2026-07-19 — it had shipped with both `yes`. If you +have chrooted SFTP-only accounts via a `Match Group` block, e.g. for +dropbox-style uploads, that block can keep its own +`PasswordAuthentication yes` independently; it doesn't reopen this risk as +long as the matched accounts have no shell/forwarding/TTY.) + +```bash +sudo systemctl restart sshd +``` +Test the new port from a **second terminal before closing your current +session** — if `sshd_config` has a typo, you don't want to be locked out. + +--- + +## 2. DNS (BIND9) + +```bash +sudo apt install -y bind9 bind9-dnsutils bind9-utils +``` + +### Zone file + +`/etc/bind/named.conf.default-zones`: +``` +zone "jayfield.org" { + type master; + file "/etc/bind/db.jayfield.org"; +}; +``` + +`/etc/bind/db.jayfield.org` — minimal skeleton (see `README.md` for every +record currently live; SPF/DKIM/DMARC TXT records get added once the mail +stack in §3 generates a DKIM key): +``` +$TTL 300 +$ORIGIN jayfield.org. +@ IN SOA ns1.jayfield.org. hostmaster.jayfield.org. ( + 2024010100 ; Serial — bump on every edit + 3600 ; Refresh + 1800 ; Retry + 1209600 ; Expire + 600 ) ; Minimum + +@ IN A +@ IN NS ns1.jayfield.org. +@ IN NS ns2.jayfield.org. +ns1.jayfield.org. IN A +ns2.jayfield.org. IN A +@ IN MX 10 mail +mail IN A +``` + +### `named.conf.options` — the two security fixes + resolver architecture + +``` +options { + directory "/var/cache/bind"; + version "unknown"; // don't leak the exact BIND version + + rate-limit { // mitigate DNS reflection/amplification abuse + responses-per-second 10; + window 5; + }; + + dnssec-validation auto; + listen-on-v6 { any; }; + notify yes; + allow-transfer { ; }; + allow-query { any; }; // required for a public authoritative zone + allow-recursion { trusted; }; // recursion only for local/trusted clients + + // Deliberately NO forwarders block — see note below. + + querylog yes; + minimal-responses no; +}; + +acl "trusted" { + localhost; + localnets; +}; +``` + +**Don't add a `forwarders {}` block.** The obvious move is to forward +non-authoritative queries to your hosting provider's DNS servers — this +host originally did exactly that, and it caused a real production +incident: the provider's resolver is shared across many tenants, so its +combined query volume tripped Spamhaus's DNSBL rate limit +(`127.255.255.254`, a "your resolver is over quota" sentinel, not a real +blacklist entry). Postfix's `postscreen` misread that as "this sender is +blacklisted" and silently dropped legitimate inbound mail — confirmed live +with GMX's own mail servers getting rejected twice. With no `forwarders` +block, BIND does full recursive resolution itself (root hints → TLD → +authoritative) under its own dedicated IP, avoiding the shared-quota +problem entirely. See `README.md`'s "DNS resolver architecture" section +for the full incident writeup. + +```bash +sudo named-checkconf +sudo systemctl restart bind9 +``` + +### Point the system resolver at BIND + +`/etc/systemd/resolved.conf`: +``` +[Resolve] +DNS=127.0.0.1 +``` +```bash +sudo systemctl restart systemd-resolved +``` +This makes every service on the box (Postfix's DNSBL/PTR lookups included) +resolve through BIND's own recursion instead of a shared upstream. Leave +the `127.0.0.53` stub listener and `/etc/resolv.conf` symlink untouched — +Docker's DNS auto-detection specifically looks for that stub. + +Verify: +```bash +dig @127.0.0.1 google.com # general recursive resolution works +dig @127.0.0.1 jayfield.org +short # your own zone resolves +``` + +### TSIG for dynamic DNS (if you need a dyndns subdomain) + +```bash +sudo tsig-keygen -a hmac-sha256 dyndns.jayfield.org. | sudo tee /etc/bind/dyndns.jayfield.org.key +sudo chmod 640 /etc/bind/dyndns.jayfield.org.key +sudo chown root:bind /etc/bind/dyndns.jayfield.org.key # NOT 644 — this key is the only auth for dynamic updates +``` +Add a matching `key` block + `update-policy` (or a delegated zone) in +`named.conf.local` — see the actual working config on this host for the +exact `nsupdate` invocation pattern used by `/var/www/dyndns`. + +--- + +## 3. Mail stack (Postfix + Dovecot + rspamd + fail2ban) + +```bash +sudo apt install -y postfix postfix-mysql mariadb-server \ + dovecot-core dovecot-imapd dovecot-lmtpd dovecot-managesieved \ + dovecot-mysql dovecot-sieve \ + redis-server redis-tools \ + rspamd fail2ban +``` +During the `postfix` package install prompt, pick "Internet Site", +`mail.` as the system mail name. + +### MySQL: virtual mailbox schema + +```bash +sudo mysql -e "CREATE DATABASE vmail CHARACTER SET utf8;" +sudo mysql vmail <<'EOF' +CREATE TABLE domains ( + id int(10) unsigned NOT NULL AUTO_INCREMENT, + domain varchar(255) NOT NULL, + PRIMARY KEY (id), UNIQUE KEY domain (domain) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE accounts ( + id int(10) unsigned NOT NULL AUTO_INCREMENT, + username varchar(64) NOT NULL, + domain varchar(255) NOT NULL, + password varchar(255) NOT NULL, + quota int(10) unsigned DEFAULT 0, + enabled tinyint(1) DEFAULT 0, + sendonly tinyint(1) DEFAULT 0, + PRIMARY KEY (id), UNIQUE KEY username (username,domain), KEY domain (domain), + CONSTRAINT accounts_ibfk_1 FOREIGN KEY (domain) REFERENCES domains (domain) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE aliases ( + id int(10) unsigned NOT NULL AUTO_INCREMENT, + source_username varchar(64) NOT NULL, source_domain varchar(255) NOT NULL, + destination_username varchar(64) NOT NULL, destination_domain varchar(255) NOT NULL, + enabled tinyint(1) DEFAULT 0, + PRIMARY KEY (id), + UNIQUE KEY source_username (source_username,source_domain,destination_username,destination_domain), + KEY source_domain (source_domain), + CONSTRAINT aliases_ibfk_1 FOREIGN KEY (source_domain) REFERENCES domains (domain) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE tlspolicies ( + id int(10) unsigned NOT NULL AUTO_INCREMENT, + domain varchar(255) NOT NULL, + policy enum('none','may','encrypt','dane','dane-only','fingerprint','verify','secure') NOT NULL, + params varchar(255) DEFAULT NULL, + PRIMARY KEY (id), UNIQUE KEY domain (domain) +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +EOF +sudo mysql -e "CREATE USER 'vmail'@'localhost' IDENTIFIED BY ''; GRANT SELECT ON vmail.* TO 'vmail'@'localhost';" +sudo mysql -e "INSERT INTO vmail.domains (domain) VALUES ('jayfield.org');" +``` +Add real accounts via `INSERT INTO accounts ... ` with a `SHA512-CRYPT` +password hash (`doveadm pw -s SHA512-CRYPT`). See `MAIL-ACCOUNTS.md` for +the full day-to-day recipe (accounts, aliases, send-only addresses, +disabling an account, verification) — this is just the one-time schema +setup. + +**Host `'localhost'`, even though Postfix/Dovecot connect via TCP to +`127.0.0.1`** (see the `hosts =` / `connect =` lines below) — this looks +wrong but isn't. With `skip_name_resolve` at its default (`OFF`), MariaDB +reverse-resolves the connecting client's IP before matching it against +`mysql.user`'s `Host` column, rather than matching the raw IP. Since +`/etc/hosts` maps `127.0.0.1 → localhost` (standard on every Linux box), +a TCP connection from `127.0.0.1` resolves to the hostname `localhost` at +match time and hits the `vmail@localhost` grant — confirmed live 2026-07-19 +by connecting with a deliberately wrong password and reading the error: +`Access denied for user 'vmail'@'localhost'`, despite `mysql -h 127.0.0.1`. +A grant for `'vmail'@'127.0.0.1'` would also work (matching the raw IP +directly), but `'localhost'` is what's actually live on this host, so +that's what's documented here now — don't "fix" it to `127.0.0.1` thinking +the current one is broken. + +### Postfix SQL lookup maps + +`/etc/postfix/sql/*.cf` — create each, mode `640 root:root`, all pointing +at `hosts = 127.0.0.1`, `dbname = vmail`, `user = vmail`: + +| File | Query | +|---|---| +| `domains.cf` | `SELECT domain FROM domains WHERE domain='%s'` | +| `accounts.cf` | `select 1 as found from accounts where username='%u' and domain='%d' and enabled=true LIMIT 1;` | +| `aliases.cf` | `select concat(destination_username,'@',destination_domain) as destinations from aliases where source_username='%u' and source_domain='%d' and enabled=true;` | +| `recipient-access.cf` | `select if(sendonly=true,'REJECT','OK') AS access from accounts where username='%u' and domain='%d' and enabled=true LIMIT 1;` | +| `sender-login-maps.cf` | `select concat(username,'@',domain) as 'owns' from accounts where username='%u' AND domain='%d' and enabled=true union select concat(destination_username,'@',destination_domain) AS 'owns' from aliases where source_username='%u' and source_domain='%d' and enabled=true;` | +| `tls-policy.cf` | `SELECT policy, params FROM tlspolicies WHERE domain='%s'` | + +### `main.cf` — the parts that matter + +``` +myhostname = alpha.jayfield.org +mydomain = jayfield.org +mydestination = $myhostname localhost.$mydomain localhost +mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 +inet_interfaces = 127.0.0.1, ::1, +virtual_mailbox_domains = mysql:/etc/postfix/sql/domains.cf +virtual_mailbox_maps = mysql:/etc/postfix/sql/accounts.cf +virtual_alias_maps = mysql:/etc/postfix/sql/aliases.cf +virtual_transport = lmtp:unix:private/dovecot-lmtp + +# postscreen: pre-greet + DNSBL filtering before real smtpd handles anything +postscreen_dnsbl_sites = ix.dnsbl.manitu.net*2 zen.spamhaus.org*2 +postscreen_dnsbl_threshold = 2 +postscreen_dnsbl_action = drop +postscreen_greet_action = drop +postscreen_access_list = permit_mynetworks cidr:/etc/postfix/postscreen_access + +# no open relay: reject anything not from mynetworks or SASL-authenticated +smtpd_relay_restrictions = reject_non_fqdn_recipient reject_unknown_recipient_domain permit_mynetworks reject_unauth_destination +smtpd_recipient_restrictions = check_recipient_access mysql:/etc/postfix/sql/recipient-access.cf +smtpd_client_restrictions = permit_mynetworks check_client_access hash:/etc/postfix/without_ptr reject_unknown_client_hostname +smtpd_helo_required = yes +smtpd_helo_restrictions = permit_mynetworks reject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_unknown_helo_hostname +smtpd_data_restrictions = reject_unauth_pipelining + +# submission (587) uses separate mua_* restrictions — see master.cf below +mua_relay_restrictions = reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_mynetworks,permit_sasl_authenticated,reject +mua_client_restrictions = permit_mynetworks,permit_sasl_authenticated,reject +mua_sender_restrictions = permit_mynetworks,reject_non_fqdn_sender,reject_sender_login_mismatch,permit_sasl_authenticated,reject + +# TLS +smtpd_tls_cert_file = /etc/letsencrypt/live/mail.jayfield.org/fullchain.pem +smtpd_tls_key_file = /etc/letsencrypt/live/mail.jayfield.org/privkey.pem +smtpd_tls_security_level = may +smtp_tls_security_level = dane # opportunistic DANE validation for outbound +smtp_tls_policy_maps = mysql:/etc/postfix/sql/tls-policy.cf +smtp_dns_support_level = dnssec + +# rspamd milter integration +milter_default_action = accept +milter_protocol = 6 +smtpd_milters = inet:localhost:11332 +non_smtpd_milters = inet:localhost:11332 + +message_size_limit = 52428800 +recipient_delimiter = + +``` + +**`myhostname` must match the PTR record for the mail IP.** This host's +PTR for `89.58.8.149` resolves to `alpha.jayfield.org`, not +`mail.jayfield.org` — a PTR/EHLO mismatch that some strict receivers +penalize. The "correct" fix is repointing the PTR at your hosting +provider (a PTR record isn't set in this zone's own `db.jayfield.org`; +it lives in the provider's reverse-DNS zone for the IP block, usually +an rDNS field in their control panel or a support ticket) to +`mail.jayfield.org`, matching the MX record. Until that's done, +`myhostname` is set to `alpha.jayfield.org` here as a same-session +workaround so PTR and EHLO at least agree — verify with: +```bash +printf 'EHLO test.local\r\nQUIT\r\n' | nc -w 3 127.0.0.1 25 # expect "220 alpha.jayfield.org ..." +``` +Switch it back to `mail.jayfield.org` once the PTR is fixed upstream — +that's the more conventional setup (dedicated mail hostname for both +PTR and EHLO, matching the MX record) and is what the TLS cert +(`mail.jayfield.org`) already assumes. + +### `master.cf` — separate port 25 from port 587 + +``` +smtp inet n - y - 1 postscreen -o smtpd_sasl_auth_enable=no +smtpd pass - - y - - smtpd +dnsblog unix - - y - 0 dnsblog +tlsproxy unix - - y - 0 tlsproxy + +submission inet n - y - - smtpd + -o syslog_name=postfix/submission + -o smtpd_tls_security_level=encrypt + -o smtpd_sasl_auth_enable=yes + -o smtpd_sasl_type=dovecot + -o smtpd_sasl_path=private/auth + -o smtpd_sasl_security_options=noanonymous + -o smtpd_client_restrictions=$mua_client_restrictions + -o smtpd_sender_restrictions=$mua_sender_restrictions + -o smtpd_relay_restrictions=$mua_relay_restrictions + -o smtpd_sender_login_maps=mysql:/etc/postfix/sql/sender-login-maps.cf + -o smtpd_helo_required=no + -o smtpd_helo_restrictions= +``` +This is the whole point: port 25 never accepts SASL auth and never relays +to third parties (`reject_unauth_destination`); port 587 requires both TLS +and authentication before it will relay anywhere. + +### Dovecot + +``` +# /etc/dovecot/conf.d/*.conf — key settings +protocols = imap lmtp sieve +mail_location = maildir:~/mail:LAYOUT=fs +mail_uid = vmail +mail_gid = vmail +mail_home = /var/vmail/mailboxes/%d/%n +mail_privileged_group = vmail + +ssl = required +ssl_cert = +default_pass_scheme = SHA512-CRYPT +password_query = SELECT username, domain, password FROM accounts WHERE username='%n' AND domain='%d' AND enabled=true; +user_query = SELECT concat('*:storage=', quota, 'M') AS quota_rule FROM accounts WHERE username='%n' AND domain='%d' AND sendonly=false; +``` + +### rspamd + +``` +# /etc/rspamd/local.d/redis.conf +servers = "127.0.0.1"; + +# /etc/rspamd/local.d/worker-proxy.inc (the milter Postfix talks to) +bind_socket = "localhost:11332"; + +# /etc/rspamd/local.d/dkim_signing.conf +path = "/var/lib/rspamd/dkim/$selector.key"; +selector = "2022"; +allow_username_mismatch = true; +``` +Generate the DKIM key and publish it in DNS: +```bash +sudo mkdir -p /var/lib/rspamd/dkim +sudo rspamadm dkim_keygen -s 2022 -d jayfield.org -k /var/lib/rspamd/dkim/2022.key +sudo chown _rspamd:_rspamd /var/lib/rspamd/dkim/2022.key +``` +The command prints a DNS TXT record for `2022._domainkey.jayfield.org` — +add it to `db.jayfield.org`, bump the serial, `rndc reload`. + +Also add SPF and DMARC TXT records: +``` +@ 3600 IN TXT "v=spf1 a:mail.jayfield.org ?all" +_dmarc 3600 IN TXT "v=DMARC1; p=reject;" +``` + +**Always quote TXT rdata in the zone file — an unquoted `;` starts a +BIND comment, not a literal semicolon.** This host learned that the +hard way 2026-07-19: `_dmarc ... TXT v=DMARC1; p=reject;` (unquoted) +was silently being served as just `"v=DMARC1"` — BIND treated +`; p=reject;` as a trailing comment and dropped it — and both DKIM +records were serving only `"v=DKIM1"` the same way, with `k=rsa; p=` +and the actual public key silently missing (**DKIM signature +verification was failing for all outbound mail** as a result). A +separate, unquoted, semicolon-free SPF line had the opposite problem — +BIND split it on whitespace into multiple concatenated character-strings +and merged in an unrelated `google-site-verification=` token, which +broke SPF parsing (RFC 7208 concatenates a record's strings with no +added spaces). `named-checkzone` does **not** catch either mistake — it +happily loads a TXT record missing its policy tag or one with extra +strings, since both are syntactically valid TXT rdata. Verify TXT +records with `dig TXT @` after every edit, not just +`named-checkzone`. + +A single quoted character-string is also capped at 255 bytes — DKIM +public keys routinely exceed that. Split across multiple adjacent +quoted strings inside the same record instead (the DNS/DKIM spec +concatenates them back together): +``` +2022._domainkey 3600 IN TXT ("v=DKIM1; k=rsa; p=" "") +``` + +### fail2ban + +```bash +sudo systemctl enable --now fail2ban +``` +Enable jails for `sshd`, `postfix`, `postfix-sasl`, `postfix-rbl`, +`dovecot`, `sieve` (and the Apache ones once §4 is done) in +`/etc/fail2ban/jail.local`, all watching `/var/log/mail.log` / +`/var/log/auth.log` / `/var/log/apache2/*.log` as appropriate. + +**Widen `sshd`'s `findtime` beyond the 30-minute default.** Default +`findtime=30m`/`maxretry=3` only catches bursts — a bot that spaces login +attempts ~33 minutes apart never accumulates 3 strikes inside any 30-minute +window and is never banned at all (seen live on this host against +`47.76.192.176`; check `fail2ban-client status sshd` and grep +`/var/log/fail2ban.log` for `Found`/`Ban` counts per IP if a repeat +offender never shows up banned). Fix in its own drop-in rather than +editing the packaged jail file: +``` +# /etc/fail2ban/jail.d/sshd-findtime.local +[sshd] +findtime = 1d +maxretry = 4 +bantime = 1d +``` +`fail2ban-client reload sshd` applies it. Worth spot-checking the other +jails' `Found`/`Ban` ratios the same way — a *distributed* variant (many +different source IPs, each trying once) evades per-IP banning by design +and needs subnet-level blocking instead, which `findtime` tuning can't fix. + +### Bring it up + +```bash +sudo systemctl restart mariadb redis-server rspamd dovecot postfix +sudo postfix check && sudo postconf -n # sanity check +``` + +--- + +## 4. Web (Apache + Let's Encrypt) + +```bash +sudo apt install -y apache2 certbot python3-certbot-apache libapache2-mod-php7.4 +sudo a2enmod ssl proxy proxy_http rewrite headers +``` + +For each subdomain (static site, dyndns, mail webmail-proxy, any +Docker-backed app): create `/etc/apache2/sites-available/.conf` with +a `:80` vhost, `a2ensite`, then let certbot handle TLS + the redirect: + +```bash +sudo certbot --apache -d --non-interactive --agree-tos \ + -m webmaster@jayfield.org --redirect +``` +This creates the `-le-ssl.conf` SSL vhost, injects the redirect into the +`:80` block, and registers the cert with `certbot.timer` for auto-renewal. + +**Add HSTS to every real domain's SSL vhost** (skip the self-signed +`default-ssl.conf` catch-all — it has no real trusted hostname for a +browser to pin): +``` +Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains" +``` + +**Pattern for any Docker-backed subdomain** (see §5): publish the +container to `127.0.0.1:` only, never a public interface. The Apache +vhost does `ProxyPass / http://127.0.0.1:/` and +`ProxyPassReverse` to match, plus (if the app needs to know it's behind +TLS) `RequestHeader set X-Forwarded-Proto "https"`. + +**Give every vhost its own `:80` block**, even if it just redirects — a +domain with no `:80` vhost of its own silently falls through to whichever +vhost Apache treats as default, which may redirect to the *wrong* domain. +This host learned that the hard way with `jayfield.org`. + +--- + +## 5. Docker layer + +```bash +curl -fsSL https://get.docker.com | sudo sh +sudo usermod -aG docker jens +``` + +### Portainer + +```bash +sudo docker run -d --name portainer --restart=always \ + -p 8000:8000 -p 9443:9443 \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v portainer_data:/data \ + portainer/portainer-ce:latest +``` +`https://:9443` for the UI. `/var/run/docker.sock` access means +anyone with Portainer access effectively has root on the host — treat its +credentials accordingly. + +### Any future container-backed site (e.g. the `web` nginx container) + +```bash +sudo docker run -d --name --restart=always \ + -p 127.0.0.1::80 \ + +``` +Then wire it into Apache per §4's pattern. + +### Nextcloud (data-hosting-focused, minimal — core Files app only) + +```bash +sudo docker network create nextcloud_net + +sudo mkdir -p /srv/nextcloud/data +sudo chown 33:33 /srv/nextcloud/data # uid 33 = www-data, both on host and in the image +sudo chmod 750 /srv/nextcloud/data + +sudo docker run -d --name nextcloud-db --network nextcloud_net --restart=always \ + -e MYSQL_ROOT_PASSWORD="$(openssl rand -hex 24)" \ + -e MYSQL_DATABASE=nextcloud -e MYSQL_USER=nextcloud \ + -e MYSQL_PASSWORD="" \ + -v nextcloud_db:/var/lib/mysql \ + mariadb:lts + +sudo docker run -d --name nextcloud-redis --network nextcloud_net --restart=always \ + -v nextcloud_redis:/data redis:alpine + +sudo docker run -d --name nextcloud --network nextcloud_net --restart=always \ + -p 127.0.0.1:8082:80 \ + -v nextcloud_html:/var/www/html \ + -v /srv/nextcloud/data:/var/www/html/data \ + -e MYSQL_HOST=nextcloud-db -e MYSQL_DATABASE=nextcloud \ + -e MYSQL_USER=nextcloud -e MYSQL_PASSWORD="" \ + -e NEXTCLOUD_ADMIN_USER= -e NEXTCLOUD_ADMIN_PASSWORD="$(openssl rand -base64 18)" \ + -e NEXTCLOUD_TRUSTED_DOMAINS=cloud.jayfield.org \ + -e TRUSTED_PROXIES=127.0.0.1 -e OVERWRITEPROTOCOL=https \ + -e REDIS_HOST=nextcloud-redis \ + nextcloud:apache + +sudo docker exec -u www-data nextcloud php occ background:cron +(sudo crontab -l 2>/dev/null; echo '*/5 * * * * docker exec -u www-data nextcloud php -f /var/www/html/cron.php > /dev/null 2>&1') | sudo crontab - +``` +**Use dedicated `nextcloud-db`/`nextcloud-redis` containers, not the +host's own MariaDB/Redis.** The host's MySQL already serves the mail +stack's `vmail` database — keep app data separate from mail data. The +host's `redis-server` is bound to loopback with no auth for rspamd's +benefit; reusing it for Nextcloud would mean loosening a shared service's +exposure just to save one small container. + +Set the DNS `A` record and Apache vhost per §2/§4 before running +`certbot --apache -d cloud.jayfield.org`. + +### Per-user password vaults (KeePass, synced via Nextcloud WebDAV) + +Once Nextcloud is up, each admin/family member gets a matching Linux +user and Nextcloud account, and their KeePass `.kdbx` is placed directly +in their Nextcloud storage so it syncs over Nextcloud's own WebDAV — +no separate Apache-WebDAV or SFTP-upload pipeline needed. + +```bash +# 1. Linux user (if they don't already have one) +sudo useradd -m -s /bin/bash + +# 2. Nextcloud account — occ has no interactive TTY over a plain `docker exec` +# from an automated shell, so generate the password non-interactively: +PW=$(openssl rand -base64 18) +sudo docker exec -e OC_PASS="$PW" -u www-data nextcloud php occ user:add --password-from-env +echo "$PW" # hand this to the user, then have them change it on first login + +# 3. Drop the vault file into their Files root and pick it up +sudo mkdir -p /srv/nextcloud/data//files +sudo cp .kdbx /srv/nextcloud/data//files/sicher_.kdbx +sudo chown -R www-data:www-data /srv/nextcloud/data/ +sudo chmod 750 /srv/nextcloud/data/ +sudo chmod 640 /srv/nextcloud/data//files/sicher_.kdbx +sudo docker exec -u www-data nextcloud php occ files:scan +``` +Point KeePass (or any WebDAV-aware sync tool) at +`https://cloud.jayfield.org/remote.php/dav/files//sicher_.kdbx`, +authenticating with the Nextcloud account from step 2. Sanity-check a new +setup with a raw WebDAV round trip before trusting it: +```bash +curl -u ':' -X PROPFIND \ + https://cloud.jayfield.org/remote.php/dav/files//sicher_.kdbx \ + -H 'Depth: 0' # expect HTTP 207 +``` +Also keep a `600`-permission copy in the user's own home directory +(`/home//sicher_.kdbx`) as a local backup outside Nextcloud. + +If an old SFTP-upload path is being retired in favor of this, lock the +corresponding chrooted account rather than deleting it — +`sudo passwd -l ` refuses login while leaving the account, +chroot, and any files recoverable (`passwd -u` reverses it). + +--- + +## 6. Post-install hardening checklist + +Run through this after everything above is live: + +- [ ] `dig CH TXT version.bind @` returns `"unknown"`, not a real version +- [ ] `named.conf.options` has a `rate-limit {}` block +- [ ] TSIG key files (if any) are `640 root:bind`, never world-readable +- [ ] Every real HTTPS vhost sends `Strict-Transport-Security` +- [ ] Every vhost has its own `:80` redirect — none silently inherit the default +- [ ] `fail2ban-client status` shows jails for `sshd`, `postfix*`, `dovecot`, `apache-auth` +- [ ] `postconf -n | grep relay_restrictions` shows `reject_unauth_destination` — confirm you are not an open relay (test from an external host) +- [ ] SPF, DKIM, and DMARC (`p=reject` or at least `p=quarantine`) TXT records are live and verified against a real send (check `Authentication-Results` on a delivered test message) +- [ ] No `forwarders {}` in BIND, and `systemd-resolved`'s `DNS=` points at `127.0.0.1` — verify with `dig @127.0.0.1 ` and a spot-check DNSBL query for a known-good IP, confirming it doesn't return `127.255.255.254` +- [ ] Disk partition actually uses the full disk (`df -h /` vs. `lsblk`/`parted ... print free` should agree) +- [ ] `PermitRootLogin no` / `PasswordAuthentication no` — verify live with + `sshd -T`, not just the config file (see §1) +- [ ] `fail2ban-client status sshd` findtime is wide enough to catch + low-and-slow attempts, not just bursts (see §3) — spot-check + `/var/log/fail2ban.log` for any repeat-offender IP with several + `Found` entries but no `Ban` +- [ ] Any chrooted SFTP accounts (`/var/sftp/*`) no longer in active use are + password-locked (`sudo passwd -S ` shows `L`), not left active + +## 7. End-to-end verification + +Send a real test email from an external provider (Gmail, GMX, etc.) to a +real mailbox and confirm in `/var/log/mail.log`: +``` +postscreen[...]: PASS NEW [] +... +dovecot: lmtp(user@yourdomain)<...>: sieve: msgid=<...>: stored mail into mailbox 'INBOX' +``` +Then reply from that mailbox and confirm the outbound leg: +``` +postfix/submission/smtpd[...]: sasl_method=PLAIN, sasl_username=user@yourdomain +postfix/smtp[...]: to=<...>, relay=, status=sent +``` +This is the same verification loop that caught the DNSBL false-positive +bug documented in `README.md` — a real send-and-check beats trusting the +config on paper. diff --git a/SYNC-PLAN.md b/SYNC-PLAN.md new file mode 100644 index 0000000..b0bf676 --- /dev/null +++ b/SYNC-PLAN.md @@ -0,0 +1,111 @@ +# alpha.jayfield.org — Intranet Sync Plan (vlda-01 → alpha) + +**Status: planning only, nothing implemented yet.** This is a point-in-time +plan for later reference, not a live inventory (`README.md` is the live +inventory) and not a build runbook (`SETUP.md` is that). Update this file +as decisions get made and pieces get built; move finished work into +`README.md` once it's live, the way the KeePass/Nextcloud migration did. + +## Problem + +`vlda-01` (home intranet server, behind a home LAN, not 24/7 online) hosts +four things the household wants reliably available: plain files, Gitea, +Jellyfin, and a Dogecoin node/wallet used for daily transaction catchup. +Goal: use `alpha.jayfield.org` (public, 24/7, already running Docker + +Nextcloud) as the always-available side, without reproducing the failure +mode that got the old setup removed. + +**Why not just revive the old WireGuard tunnel + CIFS mount** (see +`README.md`'s "Previously resolved" history — `alpha_at_moon` tunnel + +`192.168.22.0/24` CIFS mount, purged 2026-07-19, configs backed up to +`/root/removed-configs-backup/` if ever needed as a reference): that setup +required `vlda-01` to be reachable *at the moment alpha wanted the data* — +a live mount. That's structurally incompatible with "not 24/7 online." Each +service below instead either pulls data to alpha ahead of time or migrates +outright, so alpha's availability stops depending on vlda-01's. + +## Per-service plan + +### 1. Files → Syncthing + +- Syncthing container on alpha, data lands at `/srv/intranet-data/` + (host-inspectable, matching the `/srv/nextcloud/data` pattern). Admin GUI + bound to `127.0.0.1:8384`, reverse-proxied through Apache like + Portainer/Nextcloud — never expose the setup UI directly. +- Syncthing client on `vlda-01` dials **out** to alpha — no inbound port + forwarding needed at home, unlike the old tunnel. Catches up automatically + whenever vlda-01 gets power + internet. +- Direction: start **send-only from vlda-01 → receive-only on alpha** + (vlda-01 stays authoritative, alpha is a live mirror). Revisit bidirectional + later once trusted. +- Firewall: open `22000/tcp` + `21027/udp` on alpha for Syncthing. +- Expose via Nextcloud: enable the built-in `files_external` app, mount + `/srv/intranet-data/` as local external storage — same web UI + (`cloud.jayfield.org`) already used for the KeePass vaults. + +**Status: not started.** + +### 2. Gitea → full migration, not sync + +Just repos + a small DB — no reason to keep this tied to vlda-01's uptime. + +- Stop the Gitea container on vlda-01, copy its data volume (repos + DB + + config) to alpha. +- Bring it up on alpha as a permanent container, following the exact + pattern already used for `web`/`cloud.jayfield.org`: publish to + `127.0.0.1:` → Apache vhost (`git.jayfield.org`) → `certbot --apache`. +- Decommission or keep vlda-01's copy as a cold backup, admin's choice. + +**Status: not started. No open questions — ready to build when wanted.** + +### 3. Jellyfin → scoping question, not yet planned + +Media libraries are often hundreds of GB to multiple TB, and local hardware +transcoding at home may not have an equivalent on the VPS. Before sketching +an approach, need: + +- Approximate library size. +- Is the actual goal *remote* streaming (away from home), or is Jellyfin + only ever watched on the home LAN? If the latter, migrating buys nothing — + Jellyfin only needs to be up when someone's actually watching, which is + whenever vlda-01 would be on anyway. If remote access is wanted, next step + is sizing alpha's disk/bandwidth budget for it. + +**Status: blocked on the above — no plan yet.** + +### 4. Dogecoin node/wallet → security-conscious split, needs sign-off + +**Do not migrate the wallet wholesale without explicit confirmation.** Per +the attacker report (`SendUserFile`'d 2026-07-19), this host takes on the +order of 10,000 automated SSH/scan attempts a month. Putting spending keys +on the same public, constantly-probed box is a materially different risk +than at home. + +Recommended design: +- Run `dogecoind` on alpha as a Docker container with **`disablewallet=1`** + — blockchain sync only, no private keys, no funds at risk. Stays + permanently caught up with the network regardless of vlda-01's uptime. +- Keep the actual wallet (`wallet.dat`, spending keys) only on vlda-01. + Point vlda-01's `dogecoin-cli`/GUI at alpha's node as its backend + (`-connect=`), with RPC reachable **only** over an authenticated tunnel + (SSH-forwarded or WireGuard) — never bind RPC to a public interface; an + open Dogecoin RPC port is exactly the kind of thing scanners probe for. +- Result: "daily transaction catchup" becomes near-instant once vlda-01 + turns on, since the chain is already synced — without private keys ever + touching the internet-facing box. + +**Open question, needs the user's explicit answer before implementing**: +go with the wallet-split design above, or is the fund exposure small enough +to just run the wallet on alpha too, for simplicity? Do not proceed with +either without confirmation — this is the one piece of this plan with real +financial/security stakes. + +## Suggested build order + +1. Gitea (no open questions, most self-contained, validates the general + "migrate a dockerized service to alpha + Apache vhost" pattern). +2. Files via Syncthing (validates the intermittent-sync approach before + trusting it with anything else). +3. Dogecoin node — **only after explicit sign-off on the wallet-split + question above.** +4. Jellyfin — **only after the scoping question above is answered.** diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..b65170e --- /dev/null +++ b/TODO.md @@ -0,0 +1,76 @@ +# alpha.jayfield.org — TODO + +Outstanding hardening items (see `README.md` for the full service overview). +Nothing here is urgent; all are low-risk, no-downtime changes. + +- [x] **Lock down the dyndns TSIG key file permissions** — done 2026-07-19 + `/etc/bind/dyndns.jayfield.org.key` was `644` (world-readable), holding + the shared secret that authenticates dynamic DNS updates for + `dyndns.jayfield.org` with no IP-based `allow-update` backing it up. + Fixed: `chmod 640` + `chown root:bind` (owner was already correct), + matching `rndc.key`'s permissions. + +- [x] **Suppress BIND version disclosure** — done 2026-07-19 + No `version` statement in `named.conf.options`, so + `dig CH TXT version.bind @ns1.jayfield.org` leaked the exact BIND + version (`9.18.39`) — a minor fingerprinting aid for attackers. + Fixed: added `version "unknown";` inside the `options {}` block, + validated with `named-checkconf`, and applied via `rndc reload`. + Verified `dig CH TXT version.bind` now returns `"unknown"`. + +- [x] **Add response rate limiting (RRL)** — done 2026-07-19 + No `rate-limit {}` block was configured. Since `allow-query { any; }` + is required for a public authoritative zone, RRL is the standard + mitigation against the server being used for spoofed-source DNS + reflection/amplification against third parties. + Fixed: added `rate-limit { responses-per-second 10; window 5; }` to + `named.conf.options`, validated with `named-checkconf`, and applied + via `rndc reload`. Started with conservative defaults (10/s, 5s + window) — revisit if legitimate high-volume resolvers get throttled. + +- [x] **Add HSTS headers to all vhosts** — done 2026-07-19 + No vhost sent a `Strict-Transport-Security` header, leaving a window + for on-path SSL-stripping on a user's very first plaintext request to + any domain. + Fixed: added `Header always set Strict-Transport-Security + "max-age=63072000; includeSubDomains"` to all 6 domain `:443` blocks + (`dyndns`, `jayfield.org`, `www`, `mail`, `web`, `cloud`) — skipped + `default-ssl.conf` since it's the self-signed catch-all with no real + trusted hostname for a browser to pin. Validated with + `apache2ctl configtest`, applied via `systemctl reload apache2`, + verified live with `curl -sI` against all 6 domains. + +- [x] **`jayfield.org` had no `:80` vhost of its own** — done 2026-07-19 + Bare `http://jayfield.org` fell through to the `dyndns` default vhost + and redirected to the wrong domain. + Fixed: added a `:80` block to `jayfield.org.conf` redirecting to + `https://jayfield.org/`, matching the `dyndns`/`mail` pattern. + Verified: `curl -sI http://jayfield.org/` → `302` with + `Location: https://jayfield.org/`. + +- [x] **SSH: disable root login + password auth** — done 2026-07-19 + `PermitRootLogin yes` and `PasswordAuthentication yes` were both still + enabled (the "known deviation" flagged in `SETUP.md` §1) — every + credential-stuffing bot on the internet got a real login prompt to + grind against. + Fixed: `PermitRootLogin no`, `PasswordAuthentication no`, + `PubkeyAuthentication yes` in `/etc/ssh/sshd_config` (backed up as + `sshd_config.bak.20260719225249`). The `Match Group sftp` block's own + `PasswordAuthentication yes` was left as-is — separate, already-chrooted, + no-shell/no-forwarding accounts, not part of this risk. Validated with + `sshd -t`, applied via `systemctl reload sshd`, verified live with + `sshd -T` and a fresh key-only connection before considering it done. + +- [x] **fail2ban: `sshd` jail was evadable via low-and-slow attempts** — done 2026-07-19 + Default `findtime=30m`/`maxretry=3` never triggered for `47.76.192.176`, + which had been hitting `root`/`jayfield` logins roughly every 33 minutes + for most of a day — just outside the 30-minute window, so failures kept + aging out before reaching 3 strikes. Confirmed via `fail2ban.log` + (`Found`/`Ban` counts) that no other jail showed the same per-IP timing + pattern; a separate, *distributed* single-shot scan on `apache-noscript` + was noted but left open (see `README.md`). + Fixed: `/etc/fail2ban/jail.d/sshd-findtime.local` overrides `sshd` to + `findtime=1d`, `maxretry=4`, `bantime=1d`, kept in its own drop-in + rather than editing the Debian-packaged jail file. Applied via + `fail2ban-client reload sshd`; `47.76.192.176` was banned on its very + next attempt after the change.