Check in scripts/configs, organized and documented by host OS

Every script/config built this session (sanity-check, attacker-check, the Gitea backup pipeline on both ends) now lives in this repo as reference copies, not just described in prose - previously they only existed on the live servers. Split into scripts/alpha/ (Ubuntu 24.04.4 LTS) and scripts/clients/vlda-01/ (Unraid 7.3.2), each with its own README stating the exact OS/kernel and a file-by-file map to deployed paths, since a script written for one host's conventions doesn't just work unchanged on the other - the vlda-01 README in particular documents the persistence gotchas that actually broke earlier attempts (RAM- backed root filesystem, VFAT /boot with no execute bit, Unassigned Device auto-mount). Root README.md now links directly to these files from each relevant section instead of only describing them. Explicitly not included: the dedicated SSH private key vlda-01 uses to authenticate to alpha - noted in clients/vlda-01/README.md to regenerate rather than ever commit one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-26 17:21:05 +02:00
parent e88b815775
commit e9b51dbffe
16 changed files with 869 additions and 0 deletions
@@ -0,0 +1,36 @@
+# Scripts and configs — checked-in reference copies
+
+This directory holds the actual scripts and config files running on
+`alpha.jayfield.org` and its backup client(s), checked into git so they're
+versioned and reviewable alongside the prose docs that explain *why* they
+exist (`README.md`, `TODO.md`, `SYNC-PLAN.md` in the repo root).
+
+**These are reference copies, not a deployment source.** Editing a file
+here does not change anything on the live server — there is no CI/CD or
+sync mechanism pulling from this repo onto either host (matching this
+whole repo's existing convention: "changes are made directly on the
+server and recorded here afterward", see the root `README.md`). After
+changing something on a live host, re-fetch it into this directory so the
+checked-in copy doesn't drift from reality.
+
+## Layout
+
+- `alpha/` — scripts and configs that run **on `alpha.jayfield.org`
+  itself** (Ubuntu 24.04 LTS — see `alpha/README.md` for the exact
+  version/kernel).
+- `clients/` — scripts and configs that run on other hosts this project
+  administers, each in its own subdirectory named after the host. Currently
+  just `clients/vlda-01/` (Unraid — see its own `README.md`), but structured
+  to hold more if that ever grows (see `SYNC-PLAN.md` for what else lives
+  on `vlda-01`).
+
+**Why this split matters**: `alpha` and its clients run different, often
+very different, operating systems (Ubuntu vs. Unraid so far) — a script
+written for one's conventions (systemd, `/etc/cron.d`, standard persistent
+filesystem) will not just drop into the other unchanged (Unraid's `/` and
+`/usr` are RAM-backed and don't survive a reboot, its own scheduling is
+via the User Scripts plugin, not raw crontab — see `clients/vlda-01/README.md`
+for the specifics that tripped this up while building it). Each
+subdirectory's `README.md` states the exact OS/version a script was
+written for, so anyone reusing these later — including future-me — knows
+whether a given script's assumptions still hold.
@@ -0,0 +1,40 @@
+# Scripts running on `alpha.jayfield.org`
+
+**OS**: Ubuntu 24.04.4 LTS (Noble Numbat), kernel `6.8.0-136-generic`
+(upgraded from 22.04 on 2026-07-26 — see `TODO.md` in the repo root for
+the full rollout). Standard Ubuntu conventions throughout: `systemd` for
+services, `/etc/cron.d/` for scheduled jobs, a normal persistent root
+filesystem — none of the "does this survive a reboot?" caveats that
+`clients/vlda-01/` needs.
+
+## Files, and where they're actually deployed
+
+| File here | Deployed path | Purpose |
+|---|---|---|
+| `sanity-check.sh` | `/usr/local/sbin/sanity-check.sh` | Post-boot/post-change health check across the whole stack (DNS/DNSSEC, mail, MariaDB, fail2ban, Apache vhosts, Docker containers, Gitea). See root `README.md`'s "Post-boot sanity check". |
+| `sanity-check.service` | `/etc/systemd/system/sanity-check.service` | Runs `sanity-check.sh` once per boot (`systemctl enable`d), `After=` the relevant services plus a 10s settle delay. |
+| `attacker-check.py` | `/usr/local/sbin/attacker-check.py` | Weekly fail2ban log analysis — per-IP/per-jail CSVs for trend analysis, plus distributed-scan and regular-timing-evasion detection. See root `README.md`'s "Weekly attacker summary". |
+| `cron.d/attacker-check` | `/etc/cron.d/attacker-check` | Runs `attacker-check.py` weekly, Sunday `06:00`. |
+| `gitea-backup.sh` | `/usr/local/sbin/gitea-backup.sh` | Nightly `gitea dump` + sha256 checksum, ACL'd for the `giteabackup` pull account, pruned to 14 as a safety net. See root `README.md`'s "Gitea backup to `vlda-01`". |
+| `cron.d/gitea-backup` | `/etc/cron.d/gitea-backup` | Runs `gitea-backup.sh` nightly, `02:30`. |
+
+## Permissions as deployed (not preserved by git)
+
+Git doesn't track exact ownership/mode the way these need to run as — set
+these explicitly on (re-)deploy:
+
+```bash
+sudo chown root:root /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh}
+sudo chmod 750 /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh}
+sudo chown root:root /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service
+sudo chmod 644 /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service
+```
+
+`gitea-backup.sh` additionally depends on:
+- The `giteabackup` system account, its chrooted-SFTP `sshd` config
+  (reuses the existing `Match Group sftp` block — no new `sshd_config`
+  needed, just add the user to the `sftp` group), and the `rw` bind mount
+  at `/var/sftp/giteabackup/backups` (see root `README.md` for the full
+  ACL/permission model — this is **not** just "make the script executable
+  and it works", the restricted account is load-bearing for the whole
+  design).
@@ -0,0 +1,257 @@
+#!/usr/bin/env python3
+"""alpha.jayfield.org weekly attacker summary.
+
+Parses fail2ban.log (current + recent rotations, gzipped or not) for the
+past 7 days and produces two append-only CSVs for longitudinal analysis,
+plus a human-readable summary emailed to root@jayfield.org and stored as
+a dated report file on disk.
+
+Run weekly via cron (/etc/cron.d/attacker-check). Safe to also run
+manually at any time: sudo /usr/local/sbin/attacker-check.py
+"""
+import csv
+import gzip
+import re
+import statistics
+import subprocess
+import sys
+from collections import defaultdict
+from datetime import datetime, timedelta
+from pathlib import Path
+
+# current + enough rotations to cover a week even if cron and logrotate's
+# midnight timer land oddly relative to each other
+LOG_FILES = [
+	"/var/log/fail2ban.log",
+	"/var/log/fail2ban.log.1",
+	"/var/log/fail2ban.log.2.gz",
+]
+DATA_DIR = Path("/var/log/attacker-check")
+PER_IP_CSV = DATA_DIR / "per_ip_weekly.csv"
+PER_JAIL_CSV = DATA_DIR / "per_jail_weekly.csv"
+REPORTS_DIR = DATA_DIR / "reports"
+REPORT_EMAIL = "root@jayfield.org"
+
+# distributed-scan detection thresholds: a /24 needs at least this many
+# distinct IPs, each averaging no more than this many events, to be
+# flagged as a likely coordinated single-shot-per-IP scan
+CAMPAIGN_MIN_IPS = 5
+CAMPAIGN_MAX_AVG_EVENTS = 3.0
+
+# "regular timing" flag: repeat hits on the same IP/jail with a
+# coefficient of variation below this are suspiciously scripted rather
+# than organic retry jitter (this is exactly the low-and-slow evasion
+# pattern found twice this engagement - ~22-33min spacing to just miss
+# fail2ban's findtime window)
+REGULAR_TIMING_MAX_CV = 0.15
+REGULAR_TIMING_MIN_EVENTS = 3
+
+LINE_RE = re.compile(
+	r'^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}),\d+\s+\S+\s+\[\d+\]:\s+'
+	r'\S+\s+\[(?P<jail>[\w-]+)\]\s+(?P<action>Found|Ban|Unban)\s+'
+	r'(?P<ip>[0-9a-fA-F:.]+)'
+)
+
+
+def open_maybe_gzip(path: Path):
+	if path.suffix == ".gz":
+		return gzip.open(path, "rt", errors="replace")
+	return path.open("r", errors="replace")
+
+
+def parse_events(since: datetime):
+	"""Returns list of (timestamp, jail, action, ip) for events >= since."""
+	events = []
+	for name in LOG_FILES:
+		path = Path(name)
+		if not path.exists():
+			continue
+		with open_maybe_gzip(path) as f:
+			for line in f:
+				m = LINE_RE.match(line)
+				if not m:
+					continue
+				if "Restore Ban" in line:
+					continue  # persisted state on restart, not a new event
+				ts = datetime.strptime(m.group("ts"), "%Y-%m-%d %H:%M:%S")
+				if ts < since:
+					continue
+				events.append((ts, m.group("jail"), m.group("action"), m.group("ip")))
+	return events
+
+
+def subnet24(ip: str) -> str:
+	parts = ip.split(".")
+	if len(parts) == 4:
+		return ".".join(parts[:3]) + ".0/24"
+	return ip  # IPv6 or malformed - leave as-is, not bucketed
+
+
+def write_csv(path: Path, rows: list, fieldnames: list):
+	if not rows:
+		return
+	write_header = not path.exists()
+	with path.open("a", newline="") as f:
+		writer = csv.DictWriter(f, fieldnames=fieldnames)
+		if write_header:
+			writer.writeheader()
+		writer.writerows(rows)
+
+
+def main():
+	DATA_DIR.mkdir(parents=True, exist_ok=True)
+	REPORTS_DIR.mkdir(parents=True, exist_ok=True)
+	now = datetime.now()
+	since = now - timedelta(days=7)
+	week_ending = now.date().isoformat()
+
+	events = parse_events(since)
+
+	found_by_key = defaultdict(list)  # (jail, ip) -> [timestamps], sorted later
+	ban_counts = defaultdict(int)     # jail -> new-ban count this week
+	for ts, jail, action, ip in events:
+		if action == "Found":
+			found_by_key[(jail, ip)].append(ts)
+		elif action == "Ban":
+			ban_counts[jail] += 1
+
+	per_ip_rows = []
+	subnet_ips = defaultdict(set)  # (jail, subnet) -> {ip, ...}
+	for (jail, ip), timestamps in found_by_key.items():
+		timestamps.sort()
+		intervals = [
+			(timestamps[i + 1] - timestamps[i]).total_seconds()
+			for i in range(len(timestamps) - 1)
+		]
+		mean_iv = statistics.mean(intervals) if intervals else 0.0
+		stdev_iv = statistics.pstdev(intervals) if len(intervals) > 1 else 0.0
+		cv = (stdev_iv / mean_iv) if mean_iv > 0 else None
+		regular_timing = (
+			len(timestamps) >= REGULAR_TIMING_MIN_EVENTS
+			and cv is not None
+			and cv < REGULAR_TIMING_MAX_CV
+		)
+		sn = subnet24(ip)
+		subnet_ips[(jail, sn)].add(ip)
+		per_ip_rows.append({
+			"week_ending": week_ending,
+			"jail": jail,
+			"ip": ip,
+			"subnet_24": sn,
+			"event_count": len(timestamps),
+			"first_seen": timestamps[0].isoformat(),
+			"last_seen": timestamps[-1].isoformat(),
+			"mean_interval_s": round(mean_iv, 1),
+			"stdev_interval_s": round(stdev_iv, 1),
+			"regular_timing_flag": regular_timing,
+		})
+
+	jail_totals = defaultdict(lambda: {"events": 0, "ips": set(), "subnets": set()})
+	for (jail, ip), timestamps in found_by_key.items():
+		jail_totals[jail]["events"] += len(timestamps)
+		jail_totals[jail]["ips"].add(ip)
+		jail_totals[jail]["subnets"].add(subnet24(ip))
+
+	per_jail_rows = []
+	for jail, data in jail_totals.items():
+		per_jail_rows.append({
+			"week_ending": week_ending,
+			"jail": jail,
+			"total_events": data["events"],
+			"distinct_ips": len(data["ips"]),
+			"distinct_24_subnets": len(data["subnets"]),
+			"bans_this_week": ban_counts.get(jail, 0),
+		})
+
+	write_csv(PER_IP_CSV, per_ip_rows, [
+		"week_ending", "jail", "ip", "subnet_24", "event_count",
+		"first_seen", "last_seen", "mean_interval_s", "stdev_interval_s",
+		"regular_timing_flag",
+	])
+	write_csv(PER_JAIL_CSV, per_jail_rows, [
+		"week_ending", "jail", "total_events", "distinct_ips",
+		"distinct_24_subnets", "bans_this_week",
+	])
+
+	# distributed single-shot-per-IP scan detection (same shape as the
+	# apache-noscript campaign found 2026-07-19: many IPs, ~1 hit each)
+	campaigns = []
+	for (jail, sn), ips in subnet_ips.items():
+		if len(ips) < CAMPAIGN_MIN_IPS:
+			continue
+		avg_events = sum(len(found_by_key[(jail, ip)]) for ip in ips) / len(ips)
+		if avg_events <= CAMPAIGN_MAX_AVG_EVENTS:
+			campaigns.append((jail, sn, len(ips), round(avg_events, 1)))
+
+	lines = []
+	lines.append(f"alpha.jayfield.org attacker summary - week ending {week_ending}")
+	lines.append(f"Window: {since.isoformat()} .. {now.isoformat()}")
+	lines.append("")
+
+	lines.append("Per-jail totals:")
+	if per_jail_rows:
+		for row in sorted(per_jail_rows, key=lambda r: -r["total_events"]):
+			lines.append(
+				f"  {row['jail']:<20} events={row['total_events']:<6} "
+				f"ips={row['distinct_ips']:<5} subnets={row['distinct_24_subnets']:<5} "
+				f"new_bans={row['bans_this_week']}"
+			)
+	else:
+		lines.append("  (no fail2ban activity this week)")
+	lines.append("")
+
+	top_ips = sorted(per_ip_rows, key=lambda r: -r["event_count"])[:15]
+	if top_ips:
+		lines.append("Top attacking IPs this week:")
+		for row in top_ips:
+			flag = " [REGULAR TIMING]" if row["regular_timing_flag"] else ""
+			lines.append(
+				f"  {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} "
+				f"mean_interval={row['mean_interval_s']}s{flag}"
+			)
+		lines.append("")
+
+	if campaigns:
+		lines.append(
+			f"Possible distributed scan campaigns (>={CAMPAIGN_MIN_IPS} distinct IPs "
+			f"in one /24, <={CAMPAIGN_MAX_AVG_EVENTS} avg events/IP):"
+		)
+		for jail, sn, n_ips, avg_ev in sorted(campaigns, key=lambda c: -c[2]):
+			lines.append(f"  {jail}: {sn} - {n_ips} distinct IPs, avg {avg_ev} events/IP")
+	else:
+		lines.append("No distributed-scan campaign pattern detected this week.")
+	lines.append("")
+
+	regular_ips = [r for r in per_ip_rows if r["regular_timing_flag"]]
+	if regular_ips:
+		lines.append(
+			"IPs with suspiciously regular timing (possible scripted "
+			"low-and-slow evasion, e.g. spaced just past a jail's findtime):"
+		)
+		for row in sorted(regular_ips, key=lambda r: -r["event_count"]):
+			lines.append(
+				f"  {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} "
+				f"mean_interval={row['mean_interval_s']}s stdev={row['stdev_interval_s']}s"
+			)
+		lines.append("")
+
+	report_path = REPORTS_DIR / f"{week_ending}.txt"
+	lines.append(f"Data appended to {PER_IP_CSV.name} and {PER_JAIL_CSV.name} in {DATA_DIR}/ for trend analysis.")
+	lines.append(f"This report saved to {report_path}.")
+
+	report = "\n".join(lines)
+	print(report)
+
+	report_path.write_text(report + "\n")
+
+	try:
+		subprocess.run(
+			["mail", "-s", f"alpha.jayfield.org attacker summary - {week_ending}", REPORT_EMAIL],
+			input=report, text=True, check=True,
+		)
+	except Exception as e:
+		print(f"WARNING: failed to send email: {e}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+	main()
@@ -0,0 +1,3 @@
+# Weekly attacker/campaign summary from fail2ban logs.
+# Sunday 06:00 - clear of logrotate's daily midnight run.
+0 6 * * 0 root /usr/local/sbin/attacker-check.py >/var/log/attacker-check/last-run.log 2>&1
@@ -0,0 +1,3 @@
+# Nightly Gitea backup (gitea dump), pulled by vlda-01 on its own schedule.
+# 02:30 - clear of logrotate's daily midnight timer.
+30 2 * * * root /usr/local/sbin/gitea-backup.sh >> /var/log/gitea-backup.log 2>&1
@@ -0,0 +1,38 @@
+#!/bin/bash
+# Nightly Gitea backup (alpha.jayfield.org side of the alpha -> vlda-01
+# backup strategy, see SYNC-PLAN.md "Ongoing backup" section).
+#
+# Uses `gitea dump` rather than a raw copy of /srv/gitea/data, since the
+# DB is a live SQLite file - dumping gives one consistent archive instead
+# of risking a torn mid-write copy. vlda-01 pulls the result on its own
+# schedule; this script only produces and prunes local archives.
+
+set -u
+BACKUP_DIR=/srv/gitea/data/backups
+KEEP=14
+FILE="gitea-dump-$(date +%Y%m%d-%H%M%S).zip"
+
+if ! docker exec -u git gitea gitea dump -f "/data/backups/${FILE}" --skip-index -q; then
+	echo "$(date -Is) ERROR: gitea dump failed, see above" >&2
+	exit 1
+fi
+
+# vlda-01 verifies integrity before deleting its remote (alpha-side) copy
+cd "$BACKUP_DIR" || exit 1
+sha256sum "$FILE" > "${FILE}.sha256"
+
+# the giteabackup SFTP account (see sshd Match Group sftp) needs read
+# access to pull these - default ACL inheritance alone isn't enough, the
+# file's own restrictive mask neutralizes the inherited grant, so set it
+# explicitly on every new file (both the dump and its checksum)
+setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}"
+setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}.sha256"
+
+# prune to the last $KEEP archives (safety net only - in normal operation
+# vlda-01 deletes each dump right after a verified pull, well before this
+# would ever trigger; this just bounds disk use if it's offline a while)
+ls -1t gitea-dump-*.zip 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
+	rm -f "$old" "${old}.sha256"
+done
+
+echo "$(date -Is) backup complete: ${FILE} ($(du -h "${BACKUP_DIR}/${FILE}" 2>/dev/null | cut -f1)), $(ls -1 gitea-dump-*.zip 2>/dev/null | wc -l) archive(s) retained"
@@ -0,0 +1,13 @@
+[Unit]
+Description=Post-boot sanity check for alpha.jayfield.org
+After=network-online.target named.service mariadb.service postfix@-.service dovecot.service fail2ban.service apache2.service docker.service rspamd.service
+Wants=network-online.target
+
+[Service]
+Type=oneshot
+ExecStartPre=/bin/sleep 10
+ExecStart=/usr/local/sbin/sanity-check.sh
+RemainAfterExit=yes
+
+[Install]
+WantedBy=multi-user.target
@@ -0,0 +1,214 @@
+#!/bin/bash
+# alpha.jayfield.org post-change sanity check.
+# Run manually after any significant change (apt upgrade, config change);
+# runs automatically once per boot via sanity-check.service (covers OS
+# upgrades, kernel updates, and anything else that ends in a reboot).
+
+set -u
+
+LOGDIR=/var/log/sanity-check
+mkdir -p "$LOGDIR"
+LOG="$LOGDIR/$(date +%Y%m%d-%H%M%S).log"
+exec > >(tee -a "$LOG") 2>&1
+
+PASS=0
+FAIL=0
+WARN=0
+
+ok()   { printf '  OK    %s\n' "$1"; PASS=$((PASS+1)); }
+bad()  { printf '  FAIL  %s\n' "$1"; FAIL=$((FAIL+1)); }
+warn() { printf '  WARN  %s\n' "$1"; WARN=$((WARN+1)); }
+section() { printf '\n== %s ==\n' "$1"; }
+
+echo "alpha.jayfield.org sanity check - $(date -Is)"
+. /etc/os-release
+echo "OS: $PRETTY_NAME, kernel $(uname -r), uptime $(uptime -p)"
+
+# ---------- System package consistency ----------
+section "System package consistency"
+if [ -z "$(sudo dpkg --audit 2>&1)" ]; then
+	ok "dpkg --audit clean"
+else
+	bad "dpkg --audit found issues"
+fi
+
+if sudo apt-get check >/dev/null 2>&1; then
+	ok "apt-get check clean"
+else
+	bad "apt-get check reported broken dependencies"
+fi
+
+UPGRADABLE=$(apt list --upgradable 2>/dev/null | grep -vc '^Listing')
+if [ "$UPGRADABLE" -eq 0 ]; then
+	ok "no pending package upgrades"
+else
+	warn "$UPGRADABLE package(s) still upgradable"
+fi
+
+NEEDRESTART_SVC=$(sudo needrestart -b 2>/dev/null | grep -c '^NEEDRESTART-SVC')
+if [ "$NEEDRESTART_SVC" -le 1 ]; then
+	ok "no outstanding service restarts needed (allowing the harmless per-login user@ unit)"
+else
+	warn "$NEEDRESTART_SVC services still flagged by needrestart"
+fi
+
+# ---------- Disk space ----------
+section "Disk space"
+df -h / /var /boot
+ROOT_PCT=$(df --output=pcent / | tail -1 | tr -dc '0-9')
+if [ "$ROOT_PCT" -lt 90 ]; then
+	ok "root filesystem usage ${ROOT_PCT}%"
+else
+	warn "root filesystem usage ${ROOT_PCT}% - getting full"
+fi
+
+# ---------- SSH ----------
+section "SSH hardening"
+SSHD_T=$(sudo sshd -T 2>/dev/null)
+echo "$SSHD_T" | grep -q '^permitrootlogin no' && ok "PermitRootLogin no" || bad "PermitRootLogin not disabled"
+echo "$SSHD_T" | grep -q '^passwordauthentication no' && ok "PasswordAuthentication no" || bad "PasswordAuthentication not disabled"
+systemctl is-active --quiet ssh && ok "ssh.service active" || bad "ssh.service not active"
+
+# ---------- DNS / BIND ----------
+section "DNS / BIND"
+systemctl is-active --quiet named && ok "named active" || bad "named not active"
+
+if [ "$(dig @127.0.0.1 jayfield.org +short)" = "89.58.8.149" ]; then
+	ok "jayfield.org resolves to the expected IP"
+else
+	bad "jayfield.org did not resolve to the expected IP"
+fi
+
+DNSSEC_STATUS=$(dig @127.0.0.1 github.com +noall +comment 2>/dev/null | grep -oP 'status: \K[A-Z]+')
+if [ "$DNSSEC_STATUS" = "NOERROR" ]; then
+	ok "DNSSEC validation working (github.com -> NOERROR)"
+else
+	bad "DNSSEC validation broken (github.com -> ${DNSSEC_STATUS:-no response})"
+fi
+
+for h in uschi vpn; do
+	IP=$(dig @127.0.0.1 "$h.dyndns.jayfield.org" A +short)
+	if [ -n "$IP" ]; then
+		ok "$h.dyndns.jayfield.org resolves ($IP)"
+	else
+		warn "$h.dyndns.jayfield.org has no A record"
+	fi
+done
+
+# ---------- Mail stack ----------
+section "Mail stack"
+systemctl is-active --quiet postfix@- && ok "postfix active" || bad "postfix not active"
+systemctl is-active --quiet dovecot && ok "dovecot active" || bad "dovecot not active"
+systemctl is-active --quiet rspamd && ok "rspamd active" || bad "rspamd not active"
+
+SMTP_BANNER=$(printf 'QUIT\r\n' | timeout 5 nc 127.0.0.1 25 2>/dev/null | head -1)
+echo "$SMTP_BANNER" | grep -q '220.*Postfix' && ok "SMTP banner OK" || bad "SMTP banner missing/unexpected: $SMTP_BANNER"
+
+IMAP_BANNER=$(printf 'a LOGOUT\r\n' | timeout 5 nc 127.0.0.1 143 2>/dev/null | head -1)
+echo "$IMAP_BANNER" | grep -qi 'dovecot ready\|^\* OK' && ok "IMAP banner OK" || bad "IMAP banner missing/unexpected: $IMAP_BANNER"
+
+for port in 11332 11333 11334; do
+	if sudo ss -ltn | grep -q ":$port "; then
+		ok "rspamd listening on $port"
+	else
+		bad "rspamd NOT listening on $port"
+	fi
+done
+
+RSPAMD_TEST=$(printf 'Subject: sanity test\nFrom: test@example.com\nTo: jens@jayfield.org\n\nbody\n' | rspamc symbols 2>&1)
+if echo "$RSPAMD_TEST" | grep -q 'Action:'; then
+	ok "rspamd genuinely scanning (rspamc symbols responded)"
+else
+	bad "rspamd did not respond to a live scan test - check for a phantom 'active' status"
+fi
+
+if sudo ss -ltn | grep -q ':11332 '; then
+	ok "postfix's configured milter target (11332) is listening"
+else
+	bad "nothing listening on postfix's milter port - mail is flowing unfiltered/unsigned"
+fi
+
+# ---------- MariaDB / vmail ----------
+section "MariaDB / vmail"
+systemctl is-active --quiet mariadb && ok "mariadb active" || bad "mariadb not active"
+
+DOMAINS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.domains;" 2>/dev/null)
+ACCOUNTS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.accounts;" 2>/dev/null)
+[ -n "$DOMAINS" ] && [ "$DOMAINS" -ge 1 ] && ok "vmail.domains has $DOMAINS row(s)" || bad "vmail.domains query failed or returned zero rows"
+[ -n "$ACCOUNTS" ] && [ "$ACCOUNTS" -ge 1 ] && ok "vmail.accounts has $ACCOUNTS row(s)" || bad "vmail.accounts query failed or returned zero rows"
+
+if sudo postmap -q jayfield.org mysql:/etc/postfix/sql/domains.cf 2>/dev/null | grep -q jayfield.org; then
+	ok "Postfix's MySQL domain lookup works"
+else
+	bad "Postfix's MySQL domain lookup failed"
+fi
+
+# ---------- fail2ban ----------
+section "fail2ban"
+systemctl is-active --quiet fail2ban && ok "fail2ban active" || bad "fail2ban not active"
+
+JAIL_COUNT=$(sudo fail2ban-client status 2>/dev/null | grep -oP 'Number of jail:\s*\K[0-9]+')
+if [ -n "$JAIL_COUNT" ] && [ "$JAIL_COUNT" -ge 15 ]; then
+	ok "$JAIL_COUNT jails loaded"
+else
+	bad "only ${JAIL_COUNT:-0} jail(s) loaded (expected ~19)"
+fi
+
+SSHD_FT=$(sudo fail2ban-client get sshd findtime 2>/dev/null)
+DOVECOT_FT=$(sudo fail2ban-client get dovecot findtime 2>/dev/null)
+[ "$SSHD_FT" = "86400" ] && ok "sshd findtime hardening intact (86400s)" || bad "sshd findtime is ${SSHD_FT:-unknown}, expected 86400"
+[ "$DOVECOT_FT" = "86400" ] && ok "dovecot findtime hardening intact (86400s)" || bad "dovecot findtime is ${DOVECOT_FT:-unknown}, expected 86400"
+
+# ---------- Apache / vhosts ----------
+section "Apache / vhosts"
+systemctl is-active --quiet apache2 && ok "apache2 active" || bad "apache2 not active"
+sudo apache2ctl configtest >/dev/null 2>&1 && ok "apache2ctl configtest clean" || bad "apache2ctl configtest failed"
+
+check_url() {
+	local url=$1 expected=$2 code
+	code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$url")
+	if [ "$code" = "$expected" ]; then
+		ok "$url -> $code"
+	else
+		bad "$url -> $code (expected $expected)"
+	fi
+}
+check_url "https://jayfield.org/" 200
+check_url "https://www.jayfield.org/" 200
+check_url "https://dyndns.jayfield.org/" 401
+check_url "https://mail.jayfield.org/" 403
+check_url "https://cloud.jayfield.org/" 302
+check_url "https://portainer.jayfield.org/" 200
+check_url "https://git.jayfield.org/" 200
+
+# ---------- Docker ----------
+section "Docker"
+systemctl is-active --quiet docker && ok "docker active" || bad "docker not active"
+
+for c in portainer nextcloud nextcloud-redis nextcloud-db web gitea; do
+	STATUS=$(sudo docker inspect -f '{{.State.Status}}' "$c" 2>/dev/null)
+	if [ "$STATUS" = "running" ]; then
+		ok "container $c running"
+	else
+		bad "container $c is '${STATUS:-missing}' (expected running)"
+	fi
+done
+
+GITEA_REPO_COUNT=$(curl -s --max-time 5 'https://git.jayfield.org/api/v1/repos/search?limit=1' | python3 -c 'import json,sys; print(len(json.load(sys.stdin).get("data",[])))' 2>/dev/null)
+if [ "$GITEA_REPO_COUNT" = "1" ]; then
+	ok "Gitea API returns real repo data (not just a static page)"
+else
+	bad "Gitea API did not return expected repo data"
+fi
+
+# ---------- Summary ----------
+section "Summary"
+echo "PASS: $PASS   WARN: $WARN   FAIL: $FAIL"
+echo "Full log: $LOG"
+
+if command -v mail >/dev/null 2>&1; then
+	SUBJECT="alpha.jayfield.org sanity check: $([ "$FAIL" -gt 0 ] && echo "$FAIL FAILURE(S)" || echo "OK")"
+	mail -s "$SUBJECT" jens@jayfield.org < "$LOG"
+fi
+
+[ "$FAIL" -eq 0 ]
@@ -0,0 +1,14 @@
+# Client hosts
+
+Scripts and configs that run on other hosts this project administers —
+not on `alpha.jayfield.org` itself — one subdirectory per host, named
+after it. Each subdirectory's own `README.md` states that host's exact
+OS/version, since a script written for one host's conventions generally
+will not just work unchanged on another (see `../README.md` for why this
+split exists at all).
+
+## Hosts
+
+- `vlda-01/` — the home intranet server referenced throughout
+  `SYNC-PLAN.md` (Unraid). Currently holds the pull side of the
+  `alpha` → `vlda-01` Gitea backup.
@@ -0,0 +1,61 @@
+# Scripts running on `vlda-01`
+
+**OS**: Unraid `7.3.2`, kernel `6.18.38-Unraid`. **Very different
+persistence model from Ubuntu/`alpha`** — this tripped up the first version
+of every script here, so it's worth stating plainly before anything else:
+
+- `/` and `/usr` are RAM-backed (`rootfs` / an `overlay` on top of the
+  read-only squashfs image loaded from the boot USB stick). **Anything
+  written there — a plain crontab, a script dropped in `/usr/local/sbin`
+  — is silently gone on the next reboot.** Only `/boot` (the actual USB
+  stick, mounted read/write as a normal filesystem) survives.
+- `/boot` is **VFAT**, which has no concept of a Unix execute bit at all.
+  A script stored there can never be `chmod +x`'d in any way that sticks —
+  don't try to `exec` it directly from another script; invoke it via
+  `bash <path>` instead. (The **User Scripts** plugin itself handles this
+  correctly when it runs a script *through the plugin* — it copies the
+  content to a RAM-backed tmp location and `chmod +x`s that copy before
+  running it — but a script invoking *another* script directly, as the
+  `-onboot` wrapper here does, has to work around it manually.)
+- Real, persistent storage lives on the array (`/mnt/user/...`) or on
+  **Unassigned Devices** — extra disks attached but not part of the
+  protected array, mounted at `/mnt/disks/<label>` — which is where these
+  backups actually land. An Unassigned Device is **not guaranteed to
+  already be mounted** when a script runs; see `gitea-backup-pull`'s own
+  handling of this below.
+- Scheduled tasks persist via the **User Scripts** plugin (already
+  installed, already used for this box's other maintenance), which stores
+  scripts under `/boot/config/plugins/user.scripts/scripts/<name>/` and
+  scheduling in `/boot/config/plugins/user.scripts/schedule.json` — both
+  on the persistent USB stick. `update_cron` folds the JSON's cron-style
+  entries into the live `/etc/cron.d/root` at every boot; `"frequency":
+  "boot"` entries instead run once via the plugin's own `disks_mounted`
+  array-start event hook, **not** through cron at all.
+- SSH keys under `/root/.ssh` needed no special persistence handling —
+  Unraid already auto-mirrors that whole directory to
+  `/boot/config/ssh/root/` on its own.
+
+## Files, and where they're actually deployed
+
+| File here | Deployed path | Purpose |
+|---|---|---|
+| `user-scripts/gitea-backup-pull/script` | `/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script` | Pulls new `gitea-dump` archives from `alpha` via the restricted `giteabackup` SFTP account, verifies each one's sha256 checksum, deletes both files on `alpha` only after a confirmed match, prunes its own local retention (30). Resolves and mounts the `WSD2L840` Unassigned Device itself first — see the script's own header comment for the full reasoning. |
+| `user-scripts/gitea-backup-pull/description` | `.../gitea-backup-pull/description` | Shown in the User Scripts UI. |
+| `user-scripts/gitea-backup-pull-onboot/script` | `.../gitea-backup-pull-onboot/script` | One-line wrapper: `bash`-invokes the script above. Exists only because `schedule.json` supports one schedule per script *path* — this is how the "also run once at every restart" requirement was added alongside the existing daily `03:30` schedule without disturbing it. |
+| `user-scripts/gitea-backup-pull-onboot/description` | `.../gitea-backup-pull-onboot/description` | Shown in the User Scripts UI. |
+| `user-scripts/schedule.json` | `/boot/config/plugins/user.scripts/schedule.json` | **Whole file, as deployed** — includes entries for this box's *other*, pre-existing scheduled maintenance (`delete_dangling_images`, `rc.fancontrol`, etc.) that predate this project and aren't described elsewhere in this repo. Only the `gitea-backup-pull` and `gitea-backup-pull-onboot` entries near the bottom belong to this project. |
+
+## Redeploying from this checked-in copy
+
+```bash
+# on vlda-01:
+mkdir -p /boot/config/plugins/user.scripts/scripts/gitea-backup-pull
+mkdir -p /boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot
+# copy script/description files into place from this repo, then:
+cp /boot/config/plugins/user.scripts/schedule.json /tmp/user.scripts/schedule.json
+/usr/local/sbin/update_cron
+```
+The dedicated ed25519 keypair this depends on (`/root/.ssh/gitea-backup/`,
+authorized on `alpha`'s `giteabackup` account) is **not** in this repo —
+regenerate it and re-authorize the new public key on `alpha` rather than
+ever committing a private key here.
@@ -0,0 +1 @@
+Runs the gitea-backup-pull sync once at every array startup, in addition to its daily 03:30 schedule - so a restart after being offline catches up right away instead of waiting for the next scheduled time.
@@ -0,0 +1,2 @@
+#!/bin/bash
+bash /boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script
@@ -0,0 +1 @@
+Pulls new gitea-dump backup archives from alpha.jayfield.org (git.jayfield.org) via the restricted giteabackup SFTP account. Part of the alpha -> vlda-01 backup strategy, see docs-alpha.jayfield.org/SYNC-PLAN.md.
@@ -0,0 +1,130 @@
+#!/bin/bash
+# vlda-01 side of the alpha -> vlda-01 Gitea backup strategy
+# (see SYNC-PLAN.md "Ongoing backup" section). Pulls whatever nightly
+# gitea-dump archives exist on alpha that aren't already here yet - so a
+# run after several days offline just catches up on all of them, not
+# just the latest. Verifies each dump's sha256 checksum after pulling
+# it, and only then deletes the alpha-side copy - a failed/incomplete
+# verification leaves the remote copy alone for a retry next run.
+#
+# Uses SFTP (not real rsync) because the alpha-side account is
+# restricted to `ForceCommand internal-sftp` - that blocks a real
+# `rsync --server` invocation, so plain SFTP get/rm is the compatible
+# transport here.
+#
+# Stored on the WSD2L840 Unassigned Device (not the array), which isn't
+# guaranteed to already be mounted - this script mounts it itself first
+# and refuses to proceed if that didn't actually result in a real
+# mountpoint, rather than risk silently writing backups onto the array.
+
+set -u
+KEY=/root/.ssh/gitea-backup/id_ed25519
+KNOWN_HOSTS=/root/.ssh/gitea-backup/known_hosts
+REMOTE=giteabackup@alpha.jayfield.org
+REMOTE_PORT=10022
+UD_LABEL=WSD2L840
+UD_MOUNTPOINT=/mnt/disks/WSD2L840
+LOCAL_DIR="$UD_MOUNTPOINT/git"
+KEEP=30
+
+# WSD2L840 is an Unassigned Device, not part of the array - not
+# guaranteed to already be mounted when this runs. Resolve by label
+# (not a hardcoded /dev/sdX - USB/UD device paths aren't stable across
+# reboots) and mount it via the Unassigned Devices plugin's own tool,
+# which is safe to call even if it's already mounted.
+UD_DEV=$(findfs LABEL="$UD_LABEL" 2>/dev/null)
+if [ -z "$UD_DEV" ]; then
+	echo "$(date -Is) ERROR: no device labeled $UD_LABEL found - is it plugged in?" >&2
+	exit 1
+fi
+
+/usr/local/sbin/rc.unassigned mount "$UD_DEV" >/dev/null 2>&1
+
+if ! mountpoint -q "$UD_MOUNTPOINT"; then
+	echo "$(date -Is) ERROR: $UD_MOUNTPOINT did not come up as a real mountpoint after mount attempt - refusing to write backups onto the array by mistake" >&2
+	exit 1
+fi
+
+mkdir -p "$LOCAL_DIR"
+
+SFTP="sftp -P $REMOTE_PORT -i $KEY -o UserKnownHostsFile=$KNOWN_HOSTS -o BatchMode=yes"
+
+REMOTE_LISTING=$($SFTP "$REMOTE" <<< 'ls -1 backups/' 2>/dev/null)
+if [ -z "$REMOTE_LISTING" ]; then
+	echo "$(date -Is) ERROR: could not list remote backups (host down or auth failed)" >&2
+	exit 1
+fi
+REMOTE_ZIPS=$(echo "$REMOTE_LISTING" | grep '\.zip$')
+
+GET_BATCH=$(mktemp)
+DEL_BATCH=$(mktemp)
+trap 'rm -f "$GET_BATCH" "$DEL_BATCH"' EXIT
+
+TO_FETCH=0
+while read -r fname; do
+	[ -z "$fname" ] && continue
+	base=$(basename "$fname")
+	if [ ! -f "$LOCAL_DIR/$base" ]; then
+		echo "get $fname $LOCAL_DIR/" >> "$GET_BATCH"
+		TO_FETCH=$((TO_FETCH + 1))
+		# grab the checksum alongside it if alpha actually has one
+		# (older dumps from before checksums existed won't)
+		if echo "$REMOTE_LISTING" | grep -qxF "${fname}.sha256"; then
+			echo "get ${fname}.sha256 $LOCAL_DIR/" >> "$GET_BATCH"
+		fi
+	fi
+done <<< "$REMOTE_ZIPS"
+
+if [ "$TO_FETCH" -eq 0 ]; then
+	echo "$(date -Is) up to date, nothing new to pull"
+	exit 0
+fi
+
+if ! $SFTP -b "$GET_BATCH" "$REMOTE" 2>&1; then
+	echo "$(date -Is) ERROR: sftp batch fetch failed partway through" >&2
+	exit 1
+fi
+
+# verify each freshly-fetched dump before trusting it enough to delete
+# the alpha-side copy
+VERIFIED=0
+UNVERIFIED=0
+while read -r fname; do
+	[ -z "$fname" ] && continue
+	base=$(basename "$fname")
+	local_zip="$LOCAL_DIR/$base"
+	local_sum="$LOCAL_DIR/$base.sha256"
+	[ -f "$local_zip" ] || continue   # wasn't newly fetched this run
+
+	if [ ! -f "$local_sum" ]; then
+		echo "$(date -Is) WARNING: no checksum available for $base (older dump, predates checksums?) - keeping local copy, leaving alpha's copy alone" >&2
+		UNVERIFIED=$((UNVERIFIED + 1))
+		continue
+	fi
+
+	if (cd "$LOCAL_DIR" && sha256sum -c "$(basename "$local_sum")") >/dev/null 2>&1; then
+		echo "rm $fname" >> "$DEL_BATCH"
+		echo "rm ${fname}.sha256" >> "$DEL_BATCH"
+		VERIFIED=$((VERIFIED + 1))
+	else
+		echo "$(date -Is) ERROR: checksum mismatch for $base - deleting the bad local copy, leaving alpha's copy for a retry" >&2
+		rm -f "$local_zip" "$local_sum"
+		UNVERIFIED=$((UNVERIFIED + 1))
+	fi
+done <<< "$REMOTE_ZIPS"
+
+if [ -s "$DEL_BATCH" ]; then
+	if ! $SFTP -b "$DEL_BATCH" "$REMOTE" 2>&1; then
+		echo "$(date -Is) WARNING: some verified files could not be deleted on alpha - harmless, alpha's own retention prune will clean them up eventually" >&2
+	fi
+fi
+
+# prune to the last $KEEP archives (vlda-01 has far more headroom than
+# alpha, so this is the real long-term history now that alpha deletes
+# its own copies right after a verified pull)
+cd "$LOCAL_DIR" || exit 1
+ls -1t gitea-dump-*.zip 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
+	rm -f "$old" "${old}.sha256"
+done
+
+echo "$(date -Is) pulled $TO_FETCH new archive(s), $VERIFIED verified+deleted from alpha, $UNVERIFIED kept unverified, $(ls -1 gitea-dump-*.zip 2>/dev/null | wc -l) retained locally"
@@ -0,0 +1,38 @@
+{
+    "/boot/config/plugins/user.scripts/scripts/delete_dangling_images/script": {
+        "script": "/boot/config/plugins/user.scripts/scripts/delete_dangling_images/script",
+        "frequency": "disabled",
+        "id": "scheduledelete_dangling_images",
+        "custom": ""
+    },
+    "/boot/config/plugins/user.scripts/scripts/delete.ds_store/script": {
+        "script": "/boot/config/plugins/user.scripts/scripts/delete.ds_store/script",
+        "frequency": "disabled",
+        "id": "scheduledelete-ds_store",
+        "custom": ""
+    },
+    "/boot/config/plugins/user.scripts/scripts/rc.fancontrol/script": {
+        "script": "/boot/config/plugins/user.scripts/scripts/rc.fancontrol/script",
+        "frequency": "boot",
+        "id": "schedulerc-fancontrol",
+        "custom": ""
+    },
+    "/boot/config/plugins/user.scripts/scripts/viewDockerLogSize/script": {
+        "script": "/boot/config/plugins/user.scripts/scripts/viewDockerLogSize/script",
+        "frequency": "disabled",
+        "id": "scheduleviewDockerLogSize",
+        "custom": ""
+    },
+    "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script": {
+        "script": "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script",
+        "frequency": "custom",
+        "id": "schedulegitea-backup-pull",
+        "custom": "30 3 * * *"
+    },
+    "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot/script": {
+        "script": "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot/script",
+        "frequency": "boot",
+        "id": "schedulegitea-backup-pull-onboot",
+        "custom": ""
+    }
+}
				`@@ -0,0 +1 @@`
				`Runs the gitea-backup-pull sync once at every array startup, in addition to its daily 03:30 schedule - so a restart after being offline catches up right away instead of waiting for the next scheduled time.`
				`@@ -0,0 +1 @@`
				`Pulls new gitea-dump backup archives from alpha.jayfield.org (git.jayfield.org) via the restricted giteabackup SFTP account. Part of the alpha -> vlda-01 backup strategy, see docs-alpha.jayfield.org/SYNC-PLAN.md.`