docs-alpha.jayfield.org/SETUP.md

# alpha.jayfield.org — Fresh-Install Setup Guide

This is a from-scratch runbook for reproducing this host: Ubuntu 24.04 LTS
(Noble Numbat) running authoritative DNS, a full mail stack, Apache-fronted
web services, and a Docker layer for containerized apps (Portainer,
Nextcloud, Gitea). It's written in build order — each section depends on
the ones before it.

This host itself was originally built on 22.04 and upgraded in place to
24.04 on 2026-07-26 (see `TODO.md` for the full rollout and the two
post-upgrade breakages it turned up, `fail2ban` and `rspamd`) — a fresh
build today should start directly on 24.04 rather than replaying that
upgrade path.

Reference: `README.md` documents the *current, live* state of this exact
server. This file documents *how to get there* from nothing. When the two
disagree, `README.md` is the source of truth for what's actually running.

---

## 0. Before you start

- **VPS with a real, stable public IPv4** (this host: `89.58.8.149`, static
  netplan config, not DHCP) and ideally IPv6. You'll need reverse DNS (PTR)
  for the mail IP pointed at your mail hostname *before* sending real mail —
  ask your host to set it. **Don't assume a self-service option exists**:
  this host's provider control panel has no PTR/rDNS field at all, so it
  had to go through a support ticket instead (see §3's `myhostname` caveat
  for the interim workaround this host is running while that's pending).
- **Partition the disk fully at provision time.** This host originally had a
  9.7GB root partition on a 172GB disk — ~161GB sat unallocated for years
  before we caught it. If your provisioner's cloud image under-partitions,
  fix it immediately:
  ```
  sudo growpart /dev/sda 3      # extend the partition into free space
  sudo resize2fs /dev/sda3      # grow the filesystem to fill it (online, no reboot)
  ```
  Verify with `df -h /` and `sudo parted /dev/sda unit GB print free`.
- **Register the domain and delegate DNS to this host** before step 2 —
  you need `ns1`/`ns2` NS records pointed here at the registrar for the
  zone to actually resolve publicly.
- A non-root sudo user for all administration (this host: `jens`). Don't
  do the following steps as root directly.

---

## 1. Base system

```bash
sudo apt update && sudo apt full-upgrade -y
sudo timedatectl set-timezone Europe/Berlin   # match your locale
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
```

### SSH

Move SSH off the default port to cut down on background credential-stuffing
noise (this host uses `10022`):

```
# /etc/ssh/sshd_config
Port 10022
```

Prefer `PermitRootLogin no` and key-only auth (`PasswordAuthentication no`)
from the start, then add `fail2ban`'s `sshd` jail (already covered in §3)
as defense-in-depth rather than the only line of defense:

```
# /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
```

(Fixed on this host 2026-07-19 — it had shipped with both `yes`. If you
have chrooted SFTP-only accounts via a `Match Group` block, e.g. for
dropbox-style uploads, that block can keep its own
`PasswordAuthentication yes` independently; it doesn't reopen this risk as
long as the matched accounts have no shell/forwarding/TTY.)

```bash
sudo systemctl restart sshd
```
Test the new port from a **second terminal before closing your current
session** — if `sshd_config` has a typo, you don't want to be locked out.

---

## 2. DNS (BIND9)

```bash
sudo apt install -y bind9 bind9-dnsutils bind9-utils
```

### Zone file

`/etc/bind/named.conf.default-zones`:
```
zone "jayfield.org" {
    type master;
    file "/etc/bind/db.jayfield.org";
};
```

`/etc/bind/db.jayfield.org` — minimal skeleton (see `README.md` for every
record currently live; SPF/DKIM/DMARC TXT records get added once the mail
stack in §3 generates a DKIM key):
```
$TTL 300
$ORIGIN jayfield.org.
@       IN      SOA     ns1.jayfield.org. hostmaster.jayfield.org. (
                                  2024010100 ; Serial — bump on every edit
                                  3600       ; Refresh
                                  1800       ; Retry
                                  1209600    ; Expire
                                  600 )      ; Minimum

@                       IN  A       <your public IP>
@                       IN  NS      ns1.jayfield.org.
@                       IN  NS      ns2.jayfield.org.
ns1.jayfield.org.       IN  A       <your public IP>
ns2.jayfield.org.       IN  A       <your public IP>
@                       IN  MX  10  mail
mail                    IN  A       <your public IP>
```

### `named.conf.options` — the two security fixes + resolver architecture

```
options {
	directory "/var/cache/bind";
	version "unknown";                       // don't leak the exact BIND version

	rate-limit {                             // mitigate DNS reflection/amplification abuse
		responses-per-second 10;
		window 5;
	};

	dnssec-validation auto;
	listen-on-v6 { any; };
	notify yes;
	allow-transfer { <your secondary NS IP, if any>; };
	allow-query { any; };                    // required for a public authoritative zone
	allow-recursion { trusted; };             // recursion only for local/trusted clients

	// Deliberately NO forwarders block — see note below.

	querylog yes;
	minimal-responses no;
};

acl "trusted" {
     localhost;
     localnets;
};
```

**Don't add a `forwarders {}` block.** The obvious move is to forward
non-authoritative queries to your hosting provider's DNS servers — this
host originally did exactly that, and it caused a real production
incident: the provider's resolver is shared across many tenants, so its
combined query volume tripped Spamhaus's DNSBL rate limit
(`127.255.255.254`, a "your resolver is over quota" sentinel, not a real
blacklist entry). Postfix's `postscreen` misread that as "this sender is
blacklisted" and silently dropped legitimate inbound mail — confirmed live
with GMX's own mail servers getting rejected twice. With no `forwarders`
block, BIND does full recursive resolution itself (root hints → TLD →
authoritative) under its own dedicated IP, avoiding the shared-quota
problem entirely. See `README.md`'s "DNS resolver architecture" section
for the full incident writeup.

```bash
sudo named-checkconf
sudo systemctl restart bind9
```

### Point the system resolver at BIND

`/etc/systemd/resolved.conf`:
```
[Resolve]
DNS=127.0.0.1
```
```bash
sudo systemctl restart systemd-resolved
```
This makes every service on the box (Postfix's DNSBL/PTR lookups included)
resolve through BIND's own recursion instead of a shared upstream. Leave
the `127.0.0.53` stub listener and `/etc/resolv.conf` symlink untouched —
Docker's DNS auto-detection specifically looks for that stub.

Verify:
```bash
dig @127.0.0.1 google.com          # general recursive resolution works
dig @127.0.0.1 jayfield.org +short # your own zone resolves
```

### TSIG for dynamic DNS (if you need a dyndns subdomain)

**Don't use one shared key for every hostname in the zone** — this host
originally did, and it meant any dyndns credential could update *any*
hostname, not just its own (found 2026-07-20, fixed 2026-07-26, see
`TODO.md`). Generate one key per hostname instead:

```bash
for h in <host1> <host2>; do
  sudo tsig-keygen -a hmac-sha512 "$h.dyndns.jayfield.org" | \
    sudo tee "/etc/bind/$h.dyndns.jayfield.org.key"
  sudo chown root:bind "/etc/bind/$h.dyndns.jayfield.org.key"
  sudo chmod 640 "/etc/bind/$h.dyndns.jayfield.org.key"
  sudo setfacl -m u:www-data:r "/etc/bind/$h.dyndns.jayfield.org.key"  # nsupdate runs as www-data
done
```

In `named.conf.default-zones`, `include` each key file and grant it
permission for *only* its own name via `update-policy` (not
`allow-update`, which can't be scoped per name):

```
include "/etc/bind/<host1>.dyndns.jayfield.org.key";
include "/etc/bind/<host2>.dyndns.jayfield.org.key";

zone "dyndns.jayfield.org" {
	type master;
	file "/var/cache/bind/db.dyndns.jayfield.org";
	update-policy {
		grant "<host1>.dyndns.jayfield.org" name <host1>.dyndns.jayfield.org. A AAAA TXT;
		grant "<host2>.dyndns.jayfield.org" name <host2>.dyndns.jayfield.org. A AAAA TXT;
	};
};
```

Give each hostname its own `.htpasswd` account too, and have the PHP
select the TSIG key based on `$_SERVER['PHP_AUTH_USER']` (the
*authenticated* identity) rather than whatever hostname the request
claims — that way BIND itself refuses a cross-host update even if the
app-level check were ever bypassed. See the live
`/var/www/dyndns/nsupdate*.php` on this host for the exact pattern, and
`TODO.md`'s 2026-07-26 dyndns entries for the full rollout writeup
(including how to phase in a legacy fallback for devices that need
reconfiguring first, without a hard cutover).

---

## 3. Mail stack (Postfix + Dovecot + rspamd + fail2ban)

**Use the `rspamd.com` repo, not `apt install rspamd` straight from
Ubuntu's own archive** — noble's native `rspamd` package is older than
upstream's own repo build (this host runs `4.1.3` from `rspamd.com`).
This isn't just a version-freshness preference: this exact gap caused a
real incident during this host's 22.04→24.04 upgrade, where
`do-release-upgrade` silently *removed* rspamd entirely (its apt source
was in the old, disabled deb822 `.sources` format and didn't carry
forward) rather than upgrading it — `systemctl status rspamd` still
showed old cached "active" state for a while after, masking that it
had stopped scanning mail at all. Add the repo for your release
explicitly, verify it's actually enabled, then install:
```bash
curl -sS https://rspamd.com/apt-stable/gpg.key | sudo gpg --dearmor -o /usr/share/keyrings/rspamd.gpg
echo "deb [signed-by=/usr/share/keyrings/rspamd.gpg] http://rspamd.com/apt-stable/ $(lsb_release -cs) main" | \
  sudo tee /etc/apt/sources.list.d/rspamd.sources.list
sudo apt update
sudo apt install -y postfix postfix-mysql mariadb-server \
  dovecot-core dovecot-imapd dovecot-lmtpd dovecot-managesieved \
  dovecot-mysql dovecot-sieve \
  redis-server redis-tools \
  rspamd fail2ban
```
During the `postfix` package install prompt, pick "Internet Site",
`mail.<yourdomain>` as the system mail name.

After any future release upgrade, don't trust `systemctl is-active
rspamd` alone — confirm it's genuinely scanning mail with `rspamc
symbols <<< "test"` (a phantom-active-but-not-scanning rspamd is
exactly what happened here; this check is now automated in
`scripts/alpha/sanity-check.sh`, which runs after every boot).

### MySQL: virtual mailbox schema

```bash
sudo mysql -e "CREATE DATABASE vmail CHARACTER SET utf8;"
sudo mysql vmail <<'EOF'
CREATE TABLE domains (
  id int(10) unsigned NOT NULL AUTO_INCREMENT,
  domain varchar(255) NOT NULL,
  PRIMARY KEY (id), UNIQUE KEY domain (domain)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE accounts (
  id int(10) unsigned NOT NULL AUTO_INCREMENT,
  username varchar(64) NOT NULL,
  domain varchar(255) NOT NULL,
  password varchar(255) NOT NULL,
  quota int(10) unsigned DEFAULT 0,
  enabled tinyint(1) DEFAULT 0,
  sendonly tinyint(1) DEFAULT 0,
  PRIMARY KEY (id), UNIQUE KEY username (username,domain), KEY domain (domain),
  CONSTRAINT accounts_ibfk_1 FOREIGN KEY (domain) REFERENCES domains (domain)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE aliases (
  id int(10) unsigned NOT NULL AUTO_INCREMENT,
  source_username varchar(64) NOT NULL, source_domain varchar(255) NOT NULL,
  destination_username varchar(64) NOT NULL, destination_domain varchar(255) NOT NULL,
  enabled tinyint(1) DEFAULT 0,
  PRIMARY KEY (id),
  UNIQUE KEY source_username (source_username,source_domain,destination_username,destination_domain),
  KEY source_domain (source_domain),
  CONSTRAINT aliases_ibfk_1 FOREIGN KEY (source_domain) REFERENCES domains (domain)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE tlspolicies (
  id int(10) unsigned NOT NULL AUTO_INCREMENT,
  domain varchar(255) NOT NULL,
  policy enum('none','may','encrypt','dane','dane-only','fingerprint','verify','secure') NOT NULL,
  params varchar(255) DEFAULT NULL,
  PRIMARY KEY (id), UNIQUE KEY domain (domain)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
EOF
sudo mysql -e "CREATE USER 'vmail'@'localhost' IDENTIFIED BY '<strong password>'; GRANT SELECT ON vmail.* TO 'vmail'@'localhost';"
sudo mysql -e "INSERT INTO vmail.domains (domain) VALUES ('jayfield.org');"
```
Add real accounts via `INSERT INTO accounts ... ` with a `SHA512-CRYPT`
password hash (`doveadm pw -s SHA512-CRYPT`). See `MAIL-ACCOUNTS.md` for
the full day-to-day recipe (accounts, aliases, send-only addresses,
disabling an account, verification) — this is just the one-time schema
setup.

**Host `'localhost'`, even though Postfix/Dovecot connect via TCP to
`127.0.0.1`** (see the `hosts =` / `connect =` lines below) — this looks
wrong but isn't. With `skip_name_resolve` at its default (`OFF`), MariaDB
reverse-resolves the connecting client's IP before matching it against
`mysql.user`'s `Host` column, rather than matching the raw IP. Since
`/etc/hosts` maps `127.0.0.1 → localhost` (standard on every Linux box),
a TCP connection from `127.0.0.1` resolves to the hostname `localhost` at
match time and hits the `vmail@localhost` grant — confirmed live 2026-07-19
by connecting with a deliberately wrong password and reading the error:
`Access denied for user 'vmail'@'localhost'`, despite `mysql -h 127.0.0.1`.
A grant for `'vmail'@'127.0.0.1'` would also work (matching the raw IP
directly), but `'localhost'` is what's actually live on this host, so
that's what's documented here now — don't "fix" it to `127.0.0.1` thinking
the current one is broken.

### Postfix SQL lookup maps

`/etc/postfix/sql/*.cf` — create each, mode `640 root:root`, all pointing
at `hosts = 127.0.0.1`, `dbname = vmail`, `user = vmail`:

| File | Query |
|---|---|
| `domains.cf` | `SELECT domain FROM domains WHERE domain='%s'` |
| `accounts.cf` | `select 1 as found from accounts where username='%u' and domain='%d' and enabled=true LIMIT 1;` |
| `aliases.cf` | `select concat(destination_username,'@',destination_domain) as destinations from aliases where source_username='%u' and source_domain='%d' and enabled=true;` |
| `recipient-access.cf` | `select if(sendonly=true,'REJECT','OK') AS access from accounts where username='%u' and domain='%d' and enabled=true LIMIT 1;` |
| `sender-login-maps.cf` | `select concat(username,'@',domain) as 'owns' from accounts where username='%u' AND domain='%d' and enabled=true union select concat(destination_username,'@',destination_domain) AS 'owns' from aliases where source_username='%u' and source_domain='%d' and enabled=true;` |
| `tls-policy.cf` | `SELECT policy, params FROM tlspolicies WHERE domain='%s'` |

### `main.cf` — the parts that matter

```
myhostname = alpha.jayfield.org
mydomain = jayfield.org
mydestination = $myhostname localhost.$mydomain localhost
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
inet_interfaces = 127.0.0.1, ::1, <your public IP>
virtual_mailbox_domains = mysql:/etc/postfix/sql/domains.cf
virtual_mailbox_maps = mysql:/etc/postfix/sql/accounts.cf
virtual_alias_maps = mysql:/etc/postfix/sql/aliases.cf
virtual_transport = lmtp:unix:private/dovecot-lmtp

# postscreen: pre-greet + DNSBL filtering before real smtpd handles anything
postscreen_dnsbl_sites = ix.dnsbl.manitu.net*2 zen.spamhaus.org*2
postscreen_dnsbl_threshold = 2
postscreen_dnsbl_action = drop
postscreen_greet_action = drop
postscreen_access_list = permit_mynetworks cidr:/etc/postfix/postscreen_access

# no open relay: reject anything not from mynetworks or SASL-authenticated
smtpd_relay_restrictions = reject_non_fqdn_recipient reject_unknown_recipient_domain permit_mynetworks reject_unauth_destination
smtpd_recipient_restrictions = check_recipient_access mysql:/etc/postfix/sql/recipient-access.cf
smtpd_client_restrictions = permit_mynetworks check_client_access hash:/etc/postfix/without_ptr reject_unknown_client_hostname
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks reject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_unknown_helo_hostname
smtpd_data_restrictions = reject_unauth_pipelining

# submission (587) uses separate mua_* restrictions — see master.cf below
mua_relay_restrictions = reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_mynetworks,permit_sasl_authenticated,reject
mua_client_restrictions = permit_mynetworks,permit_sasl_authenticated,reject
mua_sender_restrictions = permit_mynetworks,reject_non_fqdn_sender,reject_sender_login_mismatch,permit_sasl_authenticated,reject

# TLS
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.jayfield.org/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.jayfield.org/privkey.pem
smtpd_tls_security_level = may
smtp_tls_security_level = dane          # opportunistic DANE validation for outbound
smtp_tls_policy_maps = mysql:/etc/postfix/sql/tls-policy.cf
smtp_dns_support_level = dnssec

# rspamd milter integration
milter_default_action = accept
milter_protocol = 6
smtpd_milters = inet:localhost:11332
non_smtpd_milters = inet:localhost:11332

message_size_limit = 52428800
recipient_delimiter = +
```

**`myhostname` must match the PTR record for the mail IP.** This host's
PTR for `89.58.8.149` resolves to `alpha.jayfield.org`, not
`mail.jayfield.org` — a PTR/EHLO mismatch that some strict receivers
penalize. The "correct" fix is repointing the PTR at your hosting
provider (a PTR record isn't set in this zone's own `db.jayfield.org`;
it lives in the provider's reverse-DNS zone for the IP block, usually
an rDNS field in their control panel or a support ticket) to
`mail.jayfield.org`, matching the MX record. Until that's done,
`myhostname` is set to `alpha.jayfield.org` here as a same-session
workaround so PTR and EHLO at least agree — verify with:
```bash
printf 'EHLO test.local\r\nQUIT\r\n' | nc -w 3 127.0.0.1 25   # expect "220 alpha.jayfield.org ..."
```
Switch it back to `mail.jayfield.org` once the PTR is fixed upstream —
that's the more conventional setup (dedicated mail hostname for both
PTR and EHLO, matching the MX record) and is what the TLS cert
(`mail.jayfield.org`) already assumes.

### `master.cf` — separate port 25 from port 587

```
smtp      inet  n       -       y       -       1       postscreen -o smtpd_sasl_auth_enable=no
smtpd     pass  -       -       y       -       -       smtpd
dnsblog   unix  -       -       y       -       0       dnsblog
tlsproxy  unix  -       -       y       -       0       tlsproxy

submission inet n       -       y       -       -       smtpd
    -o syslog_name=postfix/submission
    -o smtpd_tls_security_level=encrypt
    -o smtpd_sasl_auth_enable=yes
    -o smtpd_sasl_type=dovecot
    -o smtpd_sasl_path=private/auth
    -o smtpd_sasl_security_options=noanonymous
    -o smtpd_client_restrictions=$mua_client_restrictions
    -o smtpd_sender_restrictions=$mua_sender_restrictions
    -o smtpd_relay_restrictions=$mua_relay_restrictions
    -o smtpd_sender_login_maps=mysql:/etc/postfix/sql/sender-login-maps.cf
    -o smtpd_helo_required=no
    -o smtpd_helo_restrictions=
```
This is the whole point: port 25 never accepts SASL auth and never relays
to third parties (`reject_unauth_destination`); port 587 requires both TLS
and authentication before it will relay anywhere.

### Dovecot

```
# /etc/dovecot/conf.d/*.conf — key settings
protocols = imap lmtp sieve
mail_location = maildir:~/mail:LAYOUT=fs
mail_uid = vmail
mail_gid = vmail
mail_home = /var/vmail/mailboxes/%d/%n
mail_privileged_group = vmail

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.jayfield.org/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.jayfield.org/privkey.pem
disable_plaintext_auth = yes   # default — do not override to "no"

passdb { driver = sql; args = /etc/dovecot/dovecot-sql.conf }
userdb { driver = sql; args = /etc/dovecot/dovecot-sql.conf }

service auth {
  unix_listener /var/spool/postfix/private/auth { group = postfix; mode = 0660; user = postfix }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp { group = postfix; mode = 0660; user = postfix }
  user = vmail
}
service managesieve-login {
  inet_listener sieve { port = 4190 }
}

plugin {
  sieve = file:/var/vmail/sieve/%d/%n/scripts;active=/var/vmail/sieve/%d/%n/active-script.sieve
  sieve_before = /var/vmail/sieve/global/spam-global.sieve
  imapsieve_mailbox1_name = Spam
  imapsieve_mailbox1_causes = COPY
  imapsieve_mailbox1_before = file:/var/vmail/sieve/global/learn-spam.sieve
  imapsieve_mailbox2_name = *
  imapsieve_mailbox2_from = Spam
  imapsieve_mailbox2_causes = COPY
  imapsieve_mailbox2_before = file:/var/vmail/sieve/global/learn-ham.sieve
}
```
`/etc/dovecot/dovecot-sql.conf`:
```
driver = mysql
connect = host=127.0.0.1 dbname=vmail user=vmail password=<same as above>
default_pass_scheme = SHA512-CRYPT
password_query = SELECT username, domain, password FROM accounts WHERE username='%n' AND domain='%d' AND enabled=true;
user_query = SELECT concat('*:storage=', quota, 'M') AS quota_rule FROM accounts WHERE username='%n' AND domain='%d' AND sendonly=false;
```

### rspamd

```
# /etc/rspamd/local.d/redis.conf
servers = "127.0.0.1";

# /etc/rspamd/local.d/worker-proxy.inc  (the milter Postfix talks to)
bind_socket = "localhost:11332";

# /etc/rspamd/local.d/dkim_signing.conf
path = "/var/lib/rspamd/dkim/$selector.key";
selector = "2022";
allow_username_mismatch = true;
```
Generate the DKIM key and publish it in DNS:
```bash
sudo mkdir -p /var/lib/rspamd/dkim
sudo rspamadm dkim_keygen -s 2022 -d jayfield.org -k /var/lib/rspamd/dkim/2022.key
sudo chown _rspamd:_rspamd /var/lib/rspamd/dkim/2022.key
```
The command prints a DNS TXT record for `2022._domainkey.jayfield.org` —
add it to `db.jayfield.org`, bump the serial, `rndc reload`.

Also add SPF and DMARC TXT records:
```
@         3600 IN TXT "v=spf1 a:mail.jayfield.org ?all"
_dmarc    3600 IN TXT "v=DMARC1; p=reject;"
```

**Always quote TXT rdata in the zone file — an unquoted `;` starts a
BIND comment, not a literal semicolon.** This host learned that the
hard way 2026-07-19: `_dmarc ... TXT v=DMARC1; p=reject;` (unquoted)
was silently being served as just `"v=DMARC1"` — BIND treated
`; p=reject;` as a trailing comment and dropped it — and both DKIM
records were serving only `"v=DKIM1"` the same way, with `k=rsa; p=`
and the actual public key silently missing (**DKIM signature
verification was failing for all outbound mail** as a result). A
separate, unquoted, semicolon-free SPF line had the opposite problem —
BIND split it on whitespace into multiple concatenated character-strings
and merged in an unrelated `google-site-verification=` token, which
broke SPF parsing (RFC 7208 concatenates a record's strings with no
added spaces). `named-checkzone` does **not** catch either mistake — it
happily loads a TXT record missing its policy tag or one with extra
strings, since both are syntactically valid TXT rdata. Verify TXT
records with `dig TXT <name> @<ns>` after every edit, not just
`named-checkzone`.

A single quoted character-string is also capped at 255 bytes — DKIM
public keys routinely exceed that. Split across multiple adjacent
quoted strings inside the same record instead (the DNS/DKIM spec
concatenates them back together):
```
2022._domainkey 3600 IN TXT ("v=DKIM1; k=rsa; p=<first ~236 chars>" "<rest of the base64 key>")
```

### fail2ban

```bash
sudo systemctl enable --now fail2ban
```
Enable jails for `sshd`, `postfix`, `postfix-sasl`, `postfix-rbl`,
`dovecot`, `sieve` (and the Apache ones once §4 is done) in
`/etc/fail2ban/jail.local`, all watching `/var/log/mail.log` /
`/var/log/auth.log` / `/var/log/apache2/*.log` as appropriate.

**Widen `findtime` beyond the 30-minute default on every credential-auth
jail, not just `sshd`.** Default `findtime=30m`/`maxretry=3` only catches
bursts — a bot that spaces login attempts ~22-33 minutes apart never
accumulates 3 strikes inside any 30-minute window and is never banned at
all. Seen live on this host twice with the *identical* pacing trick:
first against `sshd` (`47.76.192.176`, ~33min spacing), then — because
only `sshd` got fixed at the time — against `dovecot`
(`47.95.196.60`/`82.157.132.42`/`120.48.118.132`/others, ~22-25min
spacing, coordinated multi-IP campaign, ran for days unbanned). Don't
wait for it to recur in a third jail: apply the same override to every
jail guarding a real credential (`sshd`, `dovecot`, `postfix-sasl`,
`sieve`, `courier-auth` if in use) at provisioning time, each in its own
drop-in rather than editing the packaged jail file:
```
# /etc/fail2ban/jail.d/sshd-findtime.local
[sshd]
findtime = 1d
maxretry = 4
bantime  = 1d
```
```
# /etc/fail2ban/jail.d/dovecot-findtime.local
[dovecot]
findtime = 1d
maxretry = 4
bantime  = 1d
```
`fail2ban-client reload <jail>` applies each one. To check whether a jail
still has the gap: `fail2ban-client status <jail>` for who's currently
banned, cross-referenced against grepping the jail's own log
(`/var/log/auth.log`, `/var/log/mail.log`, etc.) for per-IP hit counts —
if an IP has dozens of failures spread evenly a few minutes past
`findtime` apart and never shows up banned, that's this pattern. Also
worth spot-checking the Apache jails' `Found`/`Ban` ratios the same way —
a *distributed* variant (many different source IPs, each trying once)
evades per-IP banning by design and needs subnet-level blocking instead,
which `findtime` tuning can't fix.

### Bring it up

```bash
sudo systemctl restart mariadb redis-server rspamd dovecot postfix
sudo postfix check && sudo postconf -n   # sanity check
```

---

## 4. Web (Apache + Let's Encrypt)

```bash
sudo apt install -y apache2 certbot python3-certbot-apache
sudo a2enmod ssl proxy proxy_http rewrite headers
```

**`php7.4` isn't in noble's default archives** — this host's `dyndns`
app (see below) is written against 7.4, not noble's own default PHP.
Pull it from the `ondrej/php` PPA instead, targeting `noble` (not
`focal`/`jammy`, despite `ppa.launchpad.net`'s URLs and some
mirrors/filenames still saying "focal" from older releases of this same
PPA — check the actual `Suites:` line, not the filename):
```bash
sudo add-apt-repository -y ppa:ondrej/php
sudo apt update
sudo apt install -y libapache2-mod-php7.4
```
Confirm the source is genuinely built for this release before trusting
it, not just present: `apt-cache policy libapache2-mod-php7.4` should
show a `noble` (not `jammy`) candidate. **This host's own 22.04→24.04
upgrade left this exact package on stale jammy-built binaries for a
while** — `do-release-upgrade` repoints the PPA's `Suites:` to the new
codename automatically, but doesn't by itself pull down matching
rebuilt packages for third-party PPAs; a plain `apt full-upgrade` after
the release upgrade completes is what actually finishes the job. Check
`apt list --upgradable` after any future release upgrade rather than
assuming a completed `do-release-upgrade` run means everything is
current.

For each subdomain (static site, dyndns, mail webmail-proxy, any
Docker-backed app): create `/etc/apache2/sites-available/<name>.conf` with
a `:80` vhost, `a2ensite`, then let certbot handle TLS + the redirect:

```bash
sudo certbot --apache -d <subdomain> --non-interactive --agree-tos \
  -m webmaster@jayfield.org --redirect
```
This creates the `-le-ssl.conf` SSL vhost, injects the redirect into the
`:80` block, and registers the cert with `certbot.timer` for auto-renewal.

**Add HSTS to every real domain's SSL vhost** (skip the self-signed
`default-ssl.conf` catch-all — it has no real trusted hostname for a
browser to pin):
```
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
```

**Pattern for any Docker-backed subdomain** (see §5): publish the
container to `127.0.0.1:<port>` only, never a public interface. The Apache
vhost does `ProxyPass / http://127.0.0.1:<port>/` and
`ProxyPassReverse` to match, plus (if the app needs to know it's behind
TLS) `RequestHeader set X-Forwarded-Proto "https"`.

**Give every vhost its own `:80` block**, even if it just redirects — a
domain with no `:80` vhost of its own silently falls through to whichever
vhost Apache treats as default, which may redirect to the *wrong* domain.
This host learned that the hard way with `jayfield.org`.

---

## 5. Docker layer

```bash
curl -fsSL https://get.docker.com | sudo sh
sudo usermod -aG docker jens
```

### Portainer

```bash
sudo docker run -d --name portainer --restart=always \
  -p 8000:8000 -p 9443:9443 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v portainer_data:/data \
  portainer/portainer-ce:latest
```
`https://<host>:9443` for the UI. `/var/run/docker.sock` access means
anyone with Portainer access effectively has root on the host — treat its
credentials accordingly.

### Any future container-backed site (e.g. the `web` nginx container)

```bash
sudo docker run -d --name <name> --restart=always \
  -p 127.0.0.1:<port>:80 \
  <image>
```
Then wire it into Apache per §4's pattern.

### Gitea

```bash
sudo mkdir -p /srv/gitea/data
# pick a uid/gid that doesn't collide with an existing host account -
# check with `getent passwd <uid>` first (e.g. this host's vmail user
# is already 1000:1000, so 1010:1010 was used instead)
sudo chown 1010:1010 /srv/gitea/data

sudo docker run -d --name gitea --restart=always \
  -v /srv/gitea/data:/data \
  -p 127.0.0.1:8083:3000 \
  -e USER_UID=1010 -e USER_GID=1010 \
  -e TZ=Europe/Berlin \
  gitea/gitea:<pin an exact version, not :latest>
```
First run creates `/data/gitea/conf/app.ini` (or bring in an existing one,
e.g. when migrating from elsewhere — see `README.md`'s "Gitea" section for
how this host's instance was migrated from `vlda-01`). Key settings if
setting up fresh via the web installer, or to check/fix after migrating
an existing `app.ini`:
```ini
[server]
DOMAIN = git.jayfield.org
SSH_DOMAIN = git.jayfield.org
ROOT_URL = https://git.jayfield.org/
DISABLE_SSH = true   ; HTTPS-only, avoids exposing a second public port
                      ; for a protocol nothing here actually uses
```
Set the DNS `A` record and Apache vhost per §2/§4 before running
`certbot --apache -d git.jayfield.org`. **Also add
`RequestHeader set X-Forwarded-Proto "https"` to the `:443` vhost** (same
as Nextcloud below) — without it, Gitea can't tell the Apache→container
hop was originally HTTPS and generates `http://` clone URLs even though
`ROOT_URL` says otherwise.

### Nextcloud (data-hosting-focused, minimal — core Files app only)

```bash
sudo docker network create nextcloud_net

sudo mkdir -p /srv/nextcloud/data
sudo chown 33:33 /srv/nextcloud/data    # uid 33 = www-data, both on host and in the image
sudo chmod 750 /srv/nextcloud/data

sudo docker run -d --name nextcloud-db --network nextcloud_net --restart=always \
  -e MYSQL_ROOT_PASSWORD="$(openssl rand -hex 24)" \
  -e MYSQL_DATABASE=nextcloud -e MYSQL_USER=nextcloud \
  -e MYSQL_PASSWORD="<generated>" \
  -v nextcloud_db:/var/lib/mysql \
  mariadb:lts

sudo docker run -d --name nextcloud-redis --network nextcloud_net --restart=always \
  -v nextcloud_redis:/data redis:alpine

sudo docker run -d --name nextcloud --network nextcloud_net --restart=always \
  -p 127.0.0.1:8082:80 \
  -v nextcloud_html:/var/www/html \
  -v /srv/nextcloud/data:/var/www/html/data \
  -e MYSQL_HOST=nextcloud-db -e MYSQL_DATABASE=nextcloud \
  -e MYSQL_USER=nextcloud -e MYSQL_PASSWORD="<same as above>" \
  -e NEXTCLOUD_ADMIN_USER=<user> -e NEXTCLOUD_ADMIN_PASSWORD="$(openssl rand -base64 18)" \
  -e NEXTCLOUD_TRUSTED_DOMAINS=cloud.jayfield.org \
  -e TRUSTED_PROXIES=127.0.0.1 -e OVERWRITEPROTOCOL=https \
  -e REDIS_HOST=nextcloud-redis \
  nextcloud:apache

sudo docker exec -u www-data nextcloud php occ background:cron
(sudo crontab -l 2>/dev/null; echo '*/5 * * * * docker exec -u www-data nextcloud php -f /var/www/html/cron.php > /dev/null 2>&1') | sudo crontab -
```
**Use dedicated `nextcloud-db`/`nextcloud-redis` containers, not the
host's own MariaDB/Redis.** The host's MySQL already serves the mail
stack's `vmail` database — keep app data separate from mail data. The
host's `redis-server` is bound to loopback with no auth for rspamd's
benefit; reusing it for Nextcloud would mean loosening a shared service's
exposure just to save one small container.

Set the DNS `A` record and Apache vhost per §2/§4 before running
`certbot --apache -d cloud.jayfield.org`.

### Per-user password vaults (KeePass, synced via Nextcloud WebDAV)

Once Nextcloud is up, each admin/family member gets a matching Linux
user and Nextcloud account, and their KeePass `.kdbx` is placed directly
in their Nextcloud storage so it syncs over Nextcloud's own WebDAV —
no separate Apache-WebDAV or SFTP-upload pipeline needed.

```bash
# 1. Linux user (if they don't already have one)
sudo useradd -m -s /bin/bash <user>

# 2. Nextcloud account — occ has no interactive TTY over a plain `docker exec`
#    from an automated shell, so generate the password non-interactively:
PW=$(openssl rand -base64 18)
sudo docker exec -e OC_PASS="$PW" -u www-data nextcloud php occ user:add --password-from-env <user>
echo "$PW"   # hand this to the user, then have them change it on first login

# 3. Drop the vault file into their Files root and pick it up
sudo mkdir -p /srv/nextcloud/data/<user>/files
sudo cp <source>.kdbx /srv/nextcloud/data/<user>/files/sicher_<user>.kdbx
sudo chown -R www-data:www-data /srv/nextcloud/data/<user>
sudo chmod 750 /srv/nextcloud/data/<user>
sudo chmod 640 /srv/nextcloud/data/<user>/files/sicher_<user>.kdbx
sudo docker exec -u www-data nextcloud php occ files:scan <user>
```
Point KeePass (or any WebDAV-aware sync tool) at
`https://cloud.jayfield.org/remote.php/dav/files/<user>/sicher_<user>.kdbx`,
authenticating with the Nextcloud account from step 2. Sanity-check a new
setup with a raw WebDAV round trip before trusting it:
```bash
curl -u '<user>:<password>' -X PROPFIND \
  https://cloud.jayfield.org/remote.php/dav/files/<user>/sicher_<user>.kdbx \
  -H 'Depth: 0'   # expect HTTP 207
```
Also keep a `600`-permission copy in the user's own home directory
(`/home/<user>/sicher_<user>.kdbx`) as a local backup outside Nextcloud.

If an old SFTP-upload path is being retired in favor of this, lock the
corresponding chrooted account rather than deleting it —
`sudo passwd -l <sftp-user>` refuses login while leaving the account,
chroot, and any files recoverable (`passwd -u` reverses it).

---

## 6. Post-install hardening checklist

Run through this after everything above is live:

- [ ] `dig CH TXT version.bind @<host>` returns `"unknown"`, not a real version
- [ ] `named.conf.options` has a `rate-limit {}` block
- [ ] TSIG key files (if any) are `640 root:bind`, never world-readable
- [ ] Every real HTTPS vhost sends `Strict-Transport-Security`
- [ ] Every vhost has its own `:80` redirect — none silently inherit the default
- [ ] `fail2ban-client status` shows jails for `sshd`, `postfix*`, `dovecot`, `apache-auth`
- [ ] `postconf -n | grep relay_restrictions` shows `reject_unauth_destination` — confirm you are not an open relay (test from an external host)
- [ ] SPF, DKIM, and DMARC (`p=reject` or at least `p=quarantine`) TXT records are live and verified against a real send (check `Authentication-Results` on a delivered test message)
- [ ] No `forwarders {}` in BIND, and `systemd-resolved`'s `DNS=` points at `127.0.0.1` — verify with `dig @127.0.0.1 <any domain>` and a spot-check DNSBL query for a known-good IP, confirming it doesn't return `127.255.255.254`
- [ ] Disk partition actually uses the full disk (`df -h /` vs. `lsblk`/`parted ... print free` should agree)
- [ ] `PermitRootLogin no` / `PasswordAuthentication no` — verify live with
      `sshd -T`, not just the config file (see §1)
- [ ] `fail2ban-client status <jail>` findtime is wide enough to catch
      low-and-slow attempts, not just bursts, for every credential jail
      (`sshd`, `dovecot`, `postfix-sasl`, not just `sshd`) — see §3 —
      spot-check `/var/log/fail2ban.log` for any repeat-offender IP with
      several `Found` entries but no `Ban`
- [ ] Any chrooted SFTP accounts (`/var/sftp/*`) no longer in active use are
      password-locked (`sudo passwd -S <user>` shows `L`), not left active

## 7. End-to-end verification

Send a real test email from an external provider (Gmail, GMX, etc.) to a
real mailbox and confirm in `/var/log/mail.log`:
```
postscreen[...]: PASS NEW [<sender IP>]
...
dovecot: lmtp(user@yourdomain)<...>: sieve: msgid=<...>: stored mail into mailbox 'INBOX'
```
Then reply from that mailbox and confirm the outbound leg:
```
postfix/submission/smtpd[...]: sasl_method=PLAIN, sasl_username=user@yourdomain
postfix/smtp[...]: to=<...>, relay=<remote MX>, status=sent
```
This is the same verification loop that caught the DNSBL false-positive
bug documented in `README.md` — a real send-and-check beats trusting the
config on paper.