From e9b51dbffe2fac183cc645b8552a21f665d3fe68 Mon Sep 17 00:00:00 2001 From: Jens Ahrensfeld Date: Sun, 26 Jul 2026 17:21:05 +0200 Subject: [PATCH] Check in scripts/configs, organized and documented by host OS Every script/config built this session (sanity-check, attacker-check, the Gitea backup pipeline on both ends) now lives in this repo as reference copies, not just described in prose - previously they only existed on the live servers. Split into scripts/alpha/ (Ubuntu 24.04.4 LTS) and scripts/clients/vlda-01/ (Unraid 7.3.2), each with its own README stating the exact OS/kernel and a file-by-file map to deployed paths, since a script written for one host's conventions doesn't just work unchanged on the other - the vlda-01 README in particular documents the persistence gotchas that actually broke earlier attempts (RAM- backed root filesystem, VFAT /boot with no execute bit, Unassigned Device auto-mount). Root README.md now links directly to these files from each relevant section instead of only describing them. Explicitly not included: the dedicated SSH private key vlda-01 uses to authenticate to alpha - noted in clients/vlda-01/README.md to regenerate rather than ever commit one. Co-Authored-By: Claude Sonnet 5 --- README.md | 18 ++ scripts/README.md | 36 +++ scripts/alpha/README.md | 40 +++ scripts/alpha/attacker-check.py | 257 ++++++++++++++++++ scripts/alpha/cron.d/attacker-check | 3 + scripts/alpha/cron.d/gitea-backup | 3 + scripts/alpha/gitea-backup.sh | 38 +++ scripts/alpha/sanity-check.service | 13 + scripts/alpha/sanity-check.sh | 214 +++++++++++++++ scripts/clients/README.md | 14 + scripts/clients/vlda-01/README.md | 61 +++++ .../gitea-backup-pull-onboot/description | 1 + .../gitea-backup-pull-onboot/script | 2 + .../gitea-backup-pull/description | 1 + .../user-scripts/gitea-backup-pull/script | 130 +++++++++ .../vlda-01/user-scripts/schedule.json | 38 +++ 16 files changed, 869 insertions(+) create mode 100644 scripts/README.md create mode 100644 scripts/alpha/README.md create mode 100644 scripts/alpha/attacker-check.py create mode 100644 scripts/alpha/cron.d/attacker-check create mode 100644 scripts/alpha/cron.d/gitea-backup create mode 100644 scripts/alpha/gitea-backup.sh create mode 100644 scripts/alpha/sanity-check.service create mode 100644 scripts/alpha/sanity-check.sh create mode 100644 scripts/clients/README.md create mode 100644 scripts/clients/vlda-01/README.md create mode 100644 scripts/clients/vlda-01/user-scripts/gitea-backup-pull-onboot/description create mode 100644 scripts/clients/vlda-01/user-scripts/gitea-backup-pull-onboot/script create mode 100644 scripts/clients/vlda-01/user-scripts/gitea-backup-pull/description create mode 100644 scripts/clients/vlda-01/user-scripts/gitea-backup-pull/script create mode 100644 scripts/clients/vlda-01/user-scripts/schedule.json diff --git a/README.md b/README.md index 141d5eb..05fafe4 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,10 @@ now fully resolved, see `TODO.md`. ## Post-boot sanity check +Checked into this repo: [`scripts/alpha/sanity-check.sh`](scripts/alpha/sanity-check.sh) / +[`scripts/alpha/sanity-check.service`](scripts/alpha/sanity-check.service) +(Ubuntu 24.04 — see `scripts/alpha/README.md`). + `/usr/local/sbin/sanity-check.sh` (added 2026-07-26, after the 24.04 upgrade turned up two real post-upgrade breakages — see `TODO.md`) checks that the whole stack is actually healthy, not just "systemd says active": @@ -73,6 +77,10 @@ routine pending package upgrades, don't fail the run). ## Weekly attacker summary +Checked into this repo: [`scripts/alpha/attacker-check.py`](scripts/alpha/attacker-check.py) / +[`scripts/alpha/cron.d/attacker-check`](scripts/alpha/cron.d/attacker-check) +(Ubuntu 24.04 — see `scripts/alpha/README.md`). + `/usr/local/sbin/attacker-check.py` (added 2026-07-26) parses `fail2ban.log` (current + recent rotations, gzipped or not — reads enough history to cover a full week regardless of exactly when logrotate's daily @@ -354,6 +362,16 @@ Internet → Apache :80/:443 (git.jayfield.org vhost) ### Gitea backup to `vlda-01` (`SYNC-PLAN.md` item 2b) +Checked into this repo, split by which host each half runs on (they're on +different OSes with different conventions — see each `README.md`): +- Alpha side (Ubuntu 24.04): [`scripts/alpha/gitea-backup.sh`](scripts/alpha/gitea-backup.sh) / + [`scripts/alpha/cron.d/gitea-backup`](scripts/alpha/cron.d/gitea-backup) + — see `scripts/alpha/README.md`. +- `vlda-01` side (Unraid 7.3.2): [`scripts/clients/vlda-01/`](scripts/clients/vlda-01/) + — see `scripts/clients/vlda-01/README.md`, which also covers the + Unraid-specific persistence gotchas (RAM-backed root, VFAT `/boot`, + Unassigned Devices) that shaped how this had to be built. + Built 2026-07-26. A point-in-time backup, not a hot standby/live mirror — now that alpha is the canonical live Gitea, this keeps `vlda-01`'s copy from just going stale after the migration. Same network direction as diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..81318d3 --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,36 @@ +# Scripts and configs — checked-in reference copies + +This directory holds the actual scripts and config files running on +`alpha.jayfield.org` and its backup client(s), checked into git so they're +versioned and reviewable alongside the prose docs that explain *why* they +exist (`README.md`, `TODO.md`, `SYNC-PLAN.md` in the repo root). + +**These are reference copies, not a deployment source.** Editing a file +here does not change anything on the live server — there is no CI/CD or +sync mechanism pulling from this repo onto either host (matching this +whole repo's existing convention: "changes are made directly on the +server and recorded here afterward", see the root `README.md`). After +changing something on a live host, re-fetch it into this directory so the +checked-in copy doesn't drift from reality. + +## Layout + +- `alpha/` — scripts and configs that run **on `alpha.jayfield.org` + itself** (Ubuntu 24.04 LTS — see `alpha/README.md` for the exact + version/kernel). +- `clients/` — scripts and configs that run on other hosts this project + administers, each in its own subdirectory named after the host. Currently + just `clients/vlda-01/` (Unraid — see its own `README.md`), but structured + to hold more if that ever grows (see `SYNC-PLAN.md` for what else lives + on `vlda-01`). + +**Why this split matters**: `alpha` and its clients run different, often +very different, operating systems (Ubuntu vs. Unraid so far) — a script +written for one's conventions (systemd, `/etc/cron.d`, standard persistent +filesystem) will not just drop into the other unchanged (Unraid's `/` and +`/usr` are RAM-backed and don't survive a reboot, its own scheduling is +via the User Scripts plugin, not raw crontab — see `clients/vlda-01/README.md` +for the specifics that tripped this up while building it). Each +subdirectory's `README.md` states the exact OS/version a script was +written for, so anyone reusing these later — including future-me — knows +whether a given script's assumptions still hold. diff --git a/scripts/alpha/README.md b/scripts/alpha/README.md new file mode 100644 index 0000000..9b6d9c1 --- /dev/null +++ b/scripts/alpha/README.md @@ -0,0 +1,40 @@ +# Scripts running on `alpha.jayfield.org` + +**OS**: Ubuntu 24.04.4 LTS (Noble Numbat), kernel `6.8.0-136-generic` +(upgraded from 22.04 on 2026-07-26 — see `TODO.md` in the repo root for +the full rollout). Standard Ubuntu conventions throughout: `systemd` for +services, `/etc/cron.d/` for scheduled jobs, a normal persistent root +filesystem — none of the "does this survive a reboot?" caveats that +`clients/vlda-01/` needs. + +## Files, and where they're actually deployed + +| File here | Deployed path | Purpose | +|---|---|---| +| `sanity-check.sh` | `/usr/local/sbin/sanity-check.sh` | Post-boot/post-change health check across the whole stack (DNS/DNSSEC, mail, MariaDB, fail2ban, Apache vhosts, Docker containers, Gitea). See root `README.md`'s "Post-boot sanity check". | +| `sanity-check.service` | `/etc/systemd/system/sanity-check.service` | Runs `sanity-check.sh` once per boot (`systemctl enable`d), `After=` the relevant services plus a 10s settle delay. | +| `attacker-check.py` | `/usr/local/sbin/attacker-check.py` | Weekly fail2ban log analysis — per-IP/per-jail CSVs for trend analysis, plus distributed-scan and regular-timing-evasion detection. See root `README.md`'s "Weekly attacker summary". | +| `cron.d/attacker-check` | `/etc/cron.d/attacker-check` | Runs `attacker-check.py` weekly, Sunday `06:00`. | +| `gitea-backup.sh` | `/usr/local/sbin/gitea-backup.sh` | Nightly `gitea dump` + sha256 checksum, ACL'd for the `giteabackup` pull account, pruned to 14 as a safety net. See root `README.md`'s "Gitea backup to `vlda-01`". | +| `cron.d/gitea-backup` | `/etc/cron.d/gitea-backup` | Runs `gitea-backup.sh` nightly, `02:30`. | + +## Permissions as deployed (not preserved by git) + +Git doesn't track exact ownership/mode the way these need to run as — set +these explicitly on (re-)deploy: + +```bash +sudo chown root:root /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh} +sudo chmod 750 /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh} +sudo chown root:root /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service +sudo chmod 644 /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service +``` + +`gitea-backup.sh` additionally depends on: +- The `giteabackup` system account, its chrooted-SFTP `sshd` config + (reuses the existing `Match Group sftp` block — no new `sshd_config` + needed, just add the user to the `sftp` group), and the `rw` bind mount + at `/var/sftp/giteabackup/backups` (see root `README.md` for the full + ACL/permission model — this is **not** just "make the script executable + and it works", the restricted account is load-bearing for the whole + design). diff --git a/scripts/alpha/attacker-check.py b/scripts/alpha/attacker-check.py new file mode 100644 index 0000000..eae1759 --- /dev/null +++ b/scripts/alpha/attacker-check.py @@ -0,0 +1,257 @@ +#!/usr/bin/env python3 +"""alpha.jayfield.org weekly attacker summary. + +Parses fail2ban.log (current + recent rotations, gzipped or not) for the +past 7 days and produces two append-only CSVs for longitudinal analysis, +plus a human-readable summary emailed to root@jayfield.org and stored as +a dated report file on disk. + +Run weekly via cron (/etc/cron.d/attacker-check). Safe to also run +manually at any time: sudo /usr/local/sbin/attacker-check.py +""" +import csv +import gzip +import re +import statistics +import subprocess +import sys +from collections import defaultdict +from datetime import datetime, timedelta +from pathlib import Path + +# current + enough rotations to cover a week even if cron and logrotate's +# midnight timer land oddly relative to each other +LOG_FILES = [ + "/var/log/fail2ban.log", + "/var/log/fail2ban.log.1", + "/var/log/fail2ban.log.2.gz", +] +DATA_DIR = Path("/var/log/attacker-check") +PER_IP_CSV = DATA_DIR / "per_ip_weekly.csv" +PER_JAIL_CSV = DATA_DIR / "per_jail_weekly.csv" +REPORTS_DIR = DATA_DIR / "reports" +REPORT_EMAIL = "root@jayfield.org" + +# distributed-scan detection thresholds: a /24 needs at least this many +# distinct IPs, each averaging no more than this many events, to be +# flagged as a likely coordinated single-shot-per-IP scan +CAMPAIGN_MIN_IPS = 5 +CAMPAIGN_MAX_AVG_EVENTS = 3.0 + +# "regular timing" flag: repeat hits on the same IP/jail with a +# coefficient of variation below this are suspiciously scripted rather +# than organic retry jitter (this is exactly the low-and-slow evasion +# pattern found twice this engagement - ~22-33min spacing to just miss +# fail2ban's findtime window) +REGULAR_TIMING_MAX_CV = 0.15 +REGULAR_TIMING_MIN_EVENTS = 3 + +LINE_RE = re.compile( + r'^(?P\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}),\d+\s+\S+\s+\[\d+\]:\s+' + r'\S+\s+\[(?P[\w-]+)\]\s+(?PFound|Ban|Unban)\s+' + r'(?P[0-9a-fA-F:.]+)' +) + + +def open_maybe_gzip(path: Path): + if path.suffix == ".gz": + return gzip.open(path, "rt", errors="replace") + return path.open("r", errors="replace") + + +def parse_events(since: datetime): + """Returns list of (timestamp, jail, action, ip) for events >= since.""" + events = [] + for name in LOG_FILES: + path = Path(name) + if not path.exists(): + continue + with open_maybe_gzip(path) as f: + for line in f: + m = LINE_RE.match(line) + if not m: + continue + if "Restore Ban" in line: + continue # persisted state on restart, not a new event + ts = datetime.strptime(m.group("ts"), "%Y-%m-%d %H:%M:%S") + if ts < since: + continue + events.append((ts, m.group("jail"), m.group("action"), m.group("ip"))) + return events + + +def subnet24(ip: str) -> str: + parts = ip.split(".") + if len(parts) == 4: + return ".".join(parts[:3]) + ".0/24" + return ip # IPv6 or malformed - leave as-is, not bucketed + + +def write_csv(path: Path, rows: list, fieldnames: list): + if not rows: + return + write_header = not path.exists() + with path.open("a", newline="") as f: + writer = csv.DictWriter(f, fieldnames=fieldnames) + if write_header: + writer.writeheader() + writer.writerows(rows) + + +def main(): + DATA_DIR.mkdir(parents=True, exist_ok=True) + REPORTS_DIR.mkdir(parents=True, exist_ok=True) + now = datetime.now() + since = now - timedelta(days=7) + week_ending = now.date().isoformat() + + events = parse_events(since) + + found_by_key = defaultdict(list) # (jail, ip) -> [timestamps], sorted later + ban_counts = defaultdict(int) # jail -> new-ban count this week + for ts, jail, action, ip in events: + if action == "Found": + found_by_key[(jail, ip)].append(ts) + elif action == "Ban": + ban_counts[jail] += 1 + + per_ip_rows = [] + subnet_ips = defaultdict(set) # (jail, subnet) -> {ip, ...} + for (jail, ip), timestamps in found_by_key.items(): + timestamps.sort() + intervals = [ + (timestamps[i + 1] - timestamps[i]).total_seconds() + for i in range(len(timestamps) - 1) + ] + mean_iv = statistics.mean(intervals) if intervals else 0.0 + stdev_iv = statistics.pstdev(intervals) if len(intervals) > 1 else 0.0 + cv = (stdev_iv / mean_iv) if mean_iv > 0 else None + regular_timing = ( + len(timestamps) >= REGULAR_TIMING_MIN_EVENTS + and cv is not None + and cv < REGULAR_TIMING_MAX_CV + ) + sn = subnet24(ip) + subnet_ips[(jail, sn)].add(ip) + per_ip_rows.append({ + "week_ending": week_ending, + "jail": jail, + "ip": ip, + "subnet_24": sn, + "event_count": len(timestamps), + "first_seen": timestamps[0].isoformat(), + "last_seen": timestamps[-1].isoformat(), + "mean_interval_s": round(mean_iv, 1), + "stdev_interval_s": round(stdev_iv, 1), + "regular_timing_flag": regular_timing, + }) + + jail_totals = defaultdict(lambda: {"events": 0, "ips": set(), "subnets": set()}) + for (jail, ip), timestamps in found_by_key.items(): + jail_totals[jail]["events"] += len(timestamps) + jail_totals[jail]["ips"].add(ip) + jail_totals[jail]["subnets"].add(subnet24(ip)) + + per_jail_rows = [] + for jail, data in jail_totals.items(): + per_jail_rows.append({ + "week_ending": week_ending, + "jail": jail, + "total_events": data["events"], + "distinct_ips": len(data["ips"]), + "distinct_24_subnets": len(data["subnets"]), + "bans_this_week": ban_counts.get(jail, 0), + }) + + write_csv(PER_IP_CSV, per_ip_rows, [ + "week_ending", "jail", "ip", "subnet_24", "event_count", + "first_seen", "last_seen", "mean_interval_s", "stdev_interval_s", + "regular_timing_flag", + ]) + write_csv(PER_JAIL_CSV, per_jail_rows, [ + "week_ending", "jail", "total_events", "distinct_ips", + "distinct_24_subnets", "bans_this_week", + ]) + + # distributed single-shot-per-IP scan detection (same shape as the + # apache-noscript campaign found 2026-07-19: many IPs, ~1 hit each) + campaigns = [] + for (jail, sn), ips in subnet_ips.items(): + if len(ips) < CAMPAIGN_MIN_IPS: + continue + avg_events = sum(len(found_by_key[(jail, ip)]) for ip in ips) / len(ips) + if avg_events <= CAMPAIGN_MAX_AVG_EVENTS: + campaigns.append((jail, sn, len(ips), round(avg_events, 1))) + + lines = [] + lines.append(f"alpha.jayfield.org attacker summary - week ending {week_ending}") + lines.append(f"Window: {since.isoformat()} .. {now.isoformat()}") + lines.append("") + + lines.append("Per-jail totals:") + if per_jail_rows: + for row in sorted(per_jail_rows, key=lambda r: -r["total_events"]): + lines.append( + f" {row['jail']:<20} events={row['total_events']:<6} " + f"ips={row['distinct_ips']:<5} subnets={row['distinct_24_subnets']:<5} " + f"new_bans={row['bans_this_week']}" + ) + else: + lines.append(" (no fail2ban activity this week)") + lines.append("") + + top_ips = sorted(per_ip_rows, key=lambda r: -r["event_count"])[:15] + if top_ips: + lines.append("Top attacking IPs this week:") + for row in top_ips: + flag = " [REGULAR TIMING]" if row["regular_timing_flag"] else "" + lines.append( + f" {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} " + f"mean_interval={row['mean_interval_s']}s{flag}" + ) + lines.append("") + + if campaigns: + lines.append( + f"Possible distributed scan campaigns (>={CAMPAIGN_MIN_IPS} distinct IPs " + f"in one /24, <={CAMPAIGN_MAX_AVG_EVENTS} avg events/IP):" + ) + for jail, sn, n_ips, avg_ev in sorted(campaigns, key=lambda c: -c[2]): + lines.append(f" {jail}: {sn} - {n_ips} distinct IPs, avg {avg_ev} events/IP") + else: + lines.append("No distributed-scan campaign pattern detected this week.") + lines.append("") + + regular_ips = [r for r in per_ip_rows if r["regular_timing_flag"]] + if regular_ips: + lines.append( + "IPs with suspiciously regular timing (possible scripted " + "low-and-slow evasion, e.g. spaced just past a jail's findtime):" + ) + for row in sorted(regular_ips, key=lambda r: -r["event_count"]): + lines.append( + f" {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} " + f"mean_interval={row['mean_interval_s']}s stdev={row['stdev_interval_s']}s" + ) + lines.append("") + + report_path = REPORTS_DIR / f"{week_ending}.txt" + lines.append(f"Data appended to {PER_IP_CSV.name} and {PER_JAIL_CSV.name} in {DATA_DIR}/ for trend analysis.") + lines.append(f"This report saved to {report_path}.") + + report = "\n".join(lines) + print(report) + + report_path.write_text(report + "\n") + + try: + subprocess.run( + ["mail", "-s", f"alpha.jayfield.org attacker summary - {week_ending}", REPORT_EMAIL], + input=report, text=True, check=True, + ) + except Exception as e: + print(f"WARNING: failed to send email: {e}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/scripts/alpha/cron.d/attacker-check b/scripts/alpha/cron.d/attacker-check new file mode 100644 index 0000000..32b2792 --- /dev/null +++ b/scripts/alpha/cron.d/attacker-check @@ -0,0 +1,3 @@ +# Weekly attacker/campaign summary from fail2ban logs. +# Sunday 06:00 - clear of logrotate's daily midnight run. +0 6 * * 0 root /usr/local/sbin/attacker-check.py >/var/log/attacker-check/last-run.log 2>&1 diff --git a/scripts/alpha/cron.d/gitea-backup b/scripts/alpha/cron.d/gitea-backup new file mode 100644 index 0000000..404d650 --- /dev/null +++ b/scripts/alpha/cron.d/gitea-backup @@ -0,0 +1,3 @@ +# Nightly Gitea backup (gitea dump), pulled by vlda-01 on its own schedule. +# 02:30 - clear of logrotate's daily midnight timer. +30 2 * * * root /usr/local/sbin/gitea-backup.sh >> /var/log/gitea-backup.log 2>&1 diff --git a/scripts/alpha/gitea-backup.sh b/scripts/alpha/gitea-backup.sh new file mode 100644 index 0000000..6715ff6 --- /dev/null +++ b/scripts/alpha/gitea-backup.sh @@ -0,0 +1,38 @@ +#!/bin/bash +# Nightly Gitea backup (alpha.jayfield.org side of the alpha -> vlda-01 +# backup strategy, see SYNC-PLAN.md "Ongoing backup" section). +# +# Uses `gitea dump` rather than a raw copy of /srv/gitea/data, since the +# DB is a live SQLite file - dumping gives one consistent archive instead +# of risking a torn mid-write copy. vlda-01 pulls the result on its own +# schedule; this script only produces and prunes local archives. + +set -u +BACKUP_DIR=/srv/gitea/data/backups +KEEP=14 +FILE="gitea-dump-$(date +%Y%m%d-%H%M%S).zip" + +if ! docker exec -u git gitea gitea dump -f "/data/backups/${FILE}" --skip-index -q; then + echo "$(date -Is) ERROR: gitea dump failed, see above" >&2 + exit 1 +fi + +# vlda-01 verifies integrity before deleting its remote (alpha-side) copy +cd "$BACKUP_DIR" || exit 1 +sha256sum "$FILE" > "${FILE}.sha256" + +# the giteabackup SFTP account (see sshd Match Group sftp) needs read +# access to pull these - default ACL inheritance alone isn't enough, the +# file's own restrictive mask neutralizes the inherited grant, so set it +# explicitly on every new file (both the dump and its checksum) +setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}" +setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}.sha256" + +# prune to the last $KEEP archives (safety net only - in normal operation +# vlda-01 deletes each dump right after a verified pull, well before this +# would ever trigger; this just bounds disk use if it's offline a while) +ls -1t gitea-dump-*.zip 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do + rm -f "$old" "${old}.sha256" +done + +echo "$(date -Is) backup complete: ${FILE} ($(du -h "${BACKUP_DIR}/${FILE}" 2>/dev/null | cut -f1)), $(ls -1 gitea-dump-*.zip 2>/dev/null | wc -l) archive(s) retained" diff --git a/scripts/alpha/sanity-check.service b/scripts/alpha/sanity-check.service new file mode 100644 index 0000000..84f4789 --- /dev/null +++ b/scripts/alpha/sanity-check.service @@ -0,0 +1,13 @@ +[Unit] +Description=Post-boot sanity check for alpha.jayfield.org +After=network-online.target named.service mariadb.service postfix@-.service dovecot.service fail2ban.service apache2.service docker.service rspamd.service +Wants=network-online.target + +[Service] +Type=oneshot +ExecStartPre=/bin/sleep 10 +ExecStart=/usr/local/sbin/sanity-check.sh +RemainAfterExit=yes + +[Install] +WantedBy=multi-user.target diff --git a/scripts/alpha/sanity-check.sh b/scripts/alpha/sanity-check.sh new file mode 100644 index 0000000..f367306 --- /dev/null +++ b/scripts/alpha/sanity-check.sh @@ -0,0 +1,214 @@ +#!/bin/bash +# alpha.jayfield.org post-change sanity check. +# Run manually after any significant change (apt upgrade, config change); +# runs automatically once per boot via sanity-check.service (covers OS +# upgrades, kernel updates, and anything else that ends in a reboot). + +set -u + +LOGDIR=/var/log/sanity-check +mkdir -p "$LOGDIR" +LOG="$LOGDIR/$(date +%Y%m%d-%H%M%S).log" +exec > >(tee -a "$LOG") 2>&1 + +PASS=0 +FAIL=0 +WARN=0 + +ok() { printf ' OK %s\n' "$1"; PASS=$((PASS+1)); } +bad() { printf ' FAIL %s\n' "$1"; FAIL=$((FAIL+1)); } +warn() { printf ' WARN %s\n' "$1"; WARN=$((WARN+1)); } +section() { printf '\n== %s ==\n' "$1"; } + +echo "alpha.jayfield.org sanity check - $(date -Is)" +. /etc/os-release +echo "OS: $PRETTY_NAME, kernel $(uname -r), uptime $(uptime -p)" + +# ---------- System package consistency ---------- +section "System package consistency" +if [ -z "$(sudo dpkg --audit 2>&1)" ]; then + ok "dpkg --audit clean" +else + bad "dpkg --audit found issues" +fi + +if sudo apt-get check >/dev/null 2>&1; then + ok "apt-get check clean" +else + bad "apt-get check reported broken dependencies" +fi + +UPGRADABLE=$(apt list --upgradable 2>/dev/null | grep -vc '^Listing') +if [ "$UPGRADABLE" -eq 0 ]; then + ok "no pending package upgrades" +else + warn "$UPGRADABLE package(s) still upgradable" +fi + +NEEDRESTART_SVC=$(sudo needrestart -b 2>/dev/null | grep -c '^NEEDRESTART-SVC') +if [ "$NEEDRESTART_SVC" -le 1 ]; then + ok "no outstanding service restarts needed (allowing the harmless per-login user@ unit)" +else + warn "$NEEDRESTART_SVC services still flagged by needrestart" +fi + +# ---------- Disk space ---------- +section "Disk space" +df -h / /var /boot +ROOT_PCT=$(df --output=pcent / | tail -1 | tr -dc '0-9') +if [ "$ROOT_PCT" -lt 90 ]; then + ok "root filesystem usage ${ROOT_PCT}%" +else + warn "root filesystem usage ${ROOT_PCT}% - getting full" +fi + +# ---------- SSH ---------- +section "SSH hardening" +SSHD_T=$(sudo sshd -T 2>/dev/null) +echo "$SSHD_T" | grep -q '^permitrootlogin no' && ok "PermitRootLogin no" || bad "PermitRootLogin not disabled" +echo "$SSHD_T" | grep -q '^passwordauthentication no' && ok "PasswordAuthentication no" || bad "PasswordAuthentication not disabled" +systemctl is-active --quiet ssh && ok "ssh.service active" || bad "ssh.service not active" + +# ---------- DNS / BIND ---------- +section "DNS / BIND" +systemctl is-active --quiet named && ok "named active" || bad "named not active" + +if [ "$(dig @127.0.0.1 jayfield.org +short)" = "89.58.8.149" ]; then + ok "jayfield.org resolves to the expected IP" +else + bad "jayfield.org did not resolve to the expected IP" +fi + +DNSSEC_STATUS=$(dig @127.0.0.1 github.com +noall +comment 2>/dev/null | grep -oP 'status: \K[A-Z]+') +if [ "$DNSSEC_STATUS" = "NOERROR" ]; then + ok "DNSSEC validation working (github.com -> NOERROR)" +else + bad "DNSSEC validation broken (github.com -> ${DNSSEC_STATUS:-no response})" +fi + +for h in uschi vpn; do + IP=$(dig @127.0.0.1 "$h.dyndns.jayfield.org" A +short) + if [ -n "$IP" ]; then + ok "$h.dyndns.jayfield.org resolves ($IP)" + else + warn "$h.dyndns.jayfield.org has no A record" + fi +done + +# ---------- Mail stack ---------- +section "Mail stack" +systemctl is-active --quiet postfix@- && ok "postfix active" || bad "postfix not active" +systemctl is-active --quiet dovecot && ok "dovecot active" || bad "dovecot not active" +systemctl is-active --quiet rspamd && ok "rspamd active" || bad "rspamd not active" + +SMTP_BANNER=$(printf 'QUIT\r\n' | timeout 5 nc 127.0.0.1 25 2>/dev/null | head -1) +echo "$SMTP_BANNER" | grep -q '220.*Postfix' && ok "SMTP banner OK" || bad "SMTP banner missing/unexpected: $SMTP_BANNER" + +IMAP_BANNER=$(printf 'a LOGOUT\r\n' | timeout 5 nc 127.0.0.1 143 2>/dev/null | head -1) +echo "$IMAP_BANNER" | grep -qi 'dovecot ready\|^\* OK' && ok "IMAP banner OK" || bad "IMAP banner missing/unexpected: $IMAP_BANNER" + +for port in 11332 11333 11334; do + if sudo ss -ltn | grep -q ":$port "; then + ok "rspamd listening on $port" + else + bad "rspamd NOT listening on $port" + fi +done + +RSPAMD_TEST=$(printf 'Subject: sanity test\nFrom: test@example.com\nTo: jens@jayfield.org\n\nbody\n' | rspamc symbols 2>&1) +if echo "$RSPAMD_TEST" | grep -q 'Action:'; then + ok "rspamd genuinely scanning (rspamc symbols responded)" +else + bad "rspamd did not respond to a live scan test - check for a phantom 'active' status" +fi + +if sudo ss -ltn | grep -q ':11332 '; then + ok "postfix's configured milter target (11332) is listening" +else + bad "nothing listening on postfix's milter port - mail is flowing unfiltered/unsigned" +fi + +# ---------- MariaDB / vmail ---------- +section "MariaDB / vmail" +systemctl is-active --quiet mariadb && ok "mariadb active" || bad "mariadb not active" + +DOMAINS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.domains;" 2>/dev/null) +ACCOUNTS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.accounts;" 2>/dev/null) +[ -n "$DOMAINS" ] && [ "$DOMAINS" -ge 1 ] && ok "vmail.domains has $DOMAINS row(s)" || bad "vmail.domains query failed or returned zero rows" +[ -n "$ACCOUNTS" ] && [ "$ACCOUNTS" -ge 1 ] && ok "vmail.accounts has $ACCOUNTS row(s)" || bad "vmail.accounts query failed or returned zero rows" + +if sudo postmap -q jayfield.org mysql:/etc/postfix/sql/domains.cf 2>/dev/null | grep -q jayfield.org; then + ok "Postfix's MySQL domain lookup works" +else + bad "Postfix's MySQL domain lookup failed" +fi + +# ---------- fail2ban ---------- +section "fail2ban" +systemctl is-active --quiet fail2ban && ok "fail2ban active" || bad "fail2ban not active" + +JAIL_COUNT=$(sudo fail2ban-client status 2>/dev/null | grep -oP 'Number of jail:\s*\K[0-9]+') +if [ -n "$JAIL_COUNT" ] && [ "$JAIL_COUNT" -ge 15 ]; then + ok "$JAIL_COUNT jails loaded" +else + bad "only ${JAIL_COUNT:-0} jail(s) loaded (expected ~19)" +fi + +SSHD_FT=$(sudo fail2ban-client get sshd findtime 2>/dev/null) +DOVECOT_FT=$(sudo fail2ban-client get dovecot findtime 2>/dev/null) +[ "$SSHD_FT" = "86400" ] && ok "sshd findtime hardening intact (86400s)" || bad "sshd findtime is ${SSHD_FT:-unknown}, expected 86400" +[ "$DOVECOT_FT" = "86400" ] && ok "dovecot findtime hardening intact (86400s)" || bad "dovecot findtime is ${DOVECOT_FT:-unknown}, expected 86400" + +# ---------- Apache / vhosts ---------- +section "Apache / vhosts" +systemctl is-active --quiet apache2 && ok "apache2 active" || bad "apache2 not active" +sudo apache2ctl configtest >/dev/null 2>&1 && ok "apache2ctl configtest clean" || bad "apache2ctl configtest failed" + +check_url() { + local url=$1 expected=$2 code + code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$url") + if [ "$code" = "$expected" ]; then + ok "$url -> $code" + else + bad "$url -> $code (expected $expected)" + fi +} +check_url "https://jayfield.org/" 200 +check_url "https://www.jayfield.org/" 200 +check_url "https://dyndns.jayfield.org/" 401 +check_url "https://mail.jayfield.org/" 403 +check_url "https://cloud.jayfield.org/" 302 +check_url "https://portainer.jayfield.org/" 200 +check_url "https://git.jayfield.org/" 200 + +# ---------- Docker ---------- +section "Docker" +systemctl is-active --quiet docker && ok "docker active" || bad "docker not active" + +for c in portainer nextcloud nextcloud-redis nextcloud-db web gitea; do + STATUS=$(sudo docker inspect -f '{{.State.Status}}' "$c" 2>/dev/null) + if [ "$STATUS" = "running" ]; then + ok "container $c running" + else + bad "container $c is '${STATUS:-missing}' (expected running)" + fi +done + +GITEA_REPO_COUNT=$(curl -s --max-time 5 'https://git.jayfield.org/api/v1/repos/search?limit=1' | python3 -c 'import json,sys; print(len(json.load(sys.stdin).get("data",[])))' 2>/dev/null) +if [ "$GITEA_REPO_COUNT" = "1" ]; then + ok "Gitea API returns real repo data (not just a static page)" +else + bad "Gitea API did not return expected repo data" +fi + +# ---------- Summary ---------- +section "Summary" +echo "PASS: $PASS WARN: $WARN FAIL: $FAIL" +echo "Full log: $LOG" + +if command -v mail >/dev/null 2>&1; then + SUBJECT="alpha.jayfield.org sanity check: $([ "$FAIL" -gt 0 ] && echo "$FAIL FAILURE(S)" || echo "OK")" + mail -s "$SUBJECT" jens@jayfield.org < "$LOG" +fi + +[ "$FAIL" -eq 0 ] diff --git a/scripts/clients/README.md b/scripts/clients/README.md new file mode 100644 index 0000000..3fdbc81 --- /dev/null +++ b/scripts/clients/README.md @@ -0,0 +1,14 @@ +# Client hosts + +Scripts and configs that run on other hosts this project administers — +not on `alpha.jayfield.org` itself — one subdirectory per host, named +after it. Each subdirectory's own `README.md` states that host's exact +OS/version, since a script written for one host's conventions generally +will not just work unchanged on another (see `../README.md` for why this +split exists at all). + +## Hosts + +- `vlda-01/` — the home intranet server referenced throughout + `SYNC-PLAN.md` (Unraid). Currently holds the pull side of the + `alpha` → `vlda-01` Gitea backup. diff --git a/scripts/clients/vlda-01/README.md b/scripts/clients/vlda-01/README.md new file mode 100644 index 0000000..bfa7035 --- /dev/null +++ b/scripts/clients/vlda-01/README.md @@ -0,0 +1,61 @@ +# Scripts running on `vlda-01` + +**OS**: Unraid `7.3.2`, kernel `6.18.38-Unraid`. **Very different +persistence model from Ubuntu/`alpha`** — this tripped up the first version +of every script here, so it's worth stating plainly before anything else: + +- `/` and `/usr` are RAM-backed (`rootfs` / an `overlay` on top of the + read-only squashfs image loaded from the boot USB stick). **Anything + written there — a plain crontab, a script dropped in `/usr/local/sbin` + — is silently gone on the next reboot.** Only `/boot` (the actual USB + stick, mounted read/write as a normal filesystem) survives. +- `/boot` is **VFAT**, which has no concept of a Unix execute bit at all. + A script stored there can never be `chmod +x`'d in any way that sticks — + don't try to `exec` it directly from another script; invoke it via + `bash ` instead. (The **User Scripts** plugin itself handles this + correctly when it runs a script *through the plugin* — it copies the + content to a RAM-backed tmp location and `chmod +x`s that copy before + running it — but a script invoking *another* script directly, as the + `-onboot` wrapper here does, has to work around it manually.) +- Real, persistent storage lives on the array (`/mnt/user/...`) or on + **Unassigned Devices** — extra disks attached but not part of the + protected array, mounted at `/mnt/disks/