jens/docs-alpha.jayfield.org
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Check in scripts/configs, organized and documented by host OS

Browse Source 
Every script/config built this session (sanity-check, attacker-check,
the Gitea backup pipeline on both ends) now lives in this repo as
reference copies, not just described in prose - previously they only
existed on the live servers.

Split into scripts/alpha/ (Ubuntu 24.04.4 LTS) and
scripts/clients/vlda-01/ (Unraid 7.3.2), each with its own README
stating the exact OS/kernel and a file-by-file map to deployed paths,
since a script written for one host's conventions doesn't just work
unchanged on the other - the vlda-01 README in particular documents
the persistence gotchas that actually broke earlier attempts (RAM-
backed root filesystem, VFAT /boot with no execute bit, Unassigned
Device auto-mount). Root README.md now links directly to these files
from each relevant section instead of only describing them.

Explicitly not included: the dedicated SSH private key vlda-01 uses to
authenticate to alpha - noted in clients/vlda-01/README.md to
regenerate rather than ever commit one.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
jens
2026-07-26 17:21:05 +02:00
co-authored by Claude Sonnet 5
parent e88b815775
commit e9b51dbffe
16 changed files with 869 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+18
View File
@@ -46,6 +46,10 @@ now fully resolved, see `TODO.md`.
## Post-boot sanity check ## Post-boot sanity check
Checked into this repo: [`scripts/alpha/sanity-check.sh`](scripts/alpha/sanity-check.sh) /
[`scripts/alpha/sanity-check.service`](scripts/alpha/sanity-check.service)
(Ubuntu 24.04 — see `scripts/alpha/README.md`).
`/usr/local/sbin/sanity-check.sh` (added 2026-07-26, after the 24.04 `/usr/local/sbin/sanity-check.sh` (added 2026-07-26, after the 24.04
upgrade turned up two real post-upgrade breakages — see `TODO.md`) checks upgrade turned up two real post-upgrade breakages — see `TODO.md`) checks
that the whole stack is actually healthy, not just "systemd says active": that the whole stack is actually healthy, not just "systemd says active":
@@ -73,6 +77,10 @@ routine pending package upgrades, don't fail the run).
## Weekly attacker summary ## Weekly attacker summary
Checked into this repo: [`scripts/alpha/attacker-check.py`](scripts/alpha/attacker-check.py) /
[`scripts/alpha/cron.d/attacker-check`](scripts/alpha/cron.d/attacker-check)
(Ubuntu 24.04 — see `scripts/alpha/README.md`).
`/usr/local/sbin/attacker-check.py` (added 2026-07-26) parses `/usr/local/sbin/attacker-check.py` (added 2026-07-26) parses
`fail2ban.log` (current + recent rotations, gzipped or not — reads enough `fail2ban.log` (current + recent rotations, gzipped or not — reads enough
history to cover a full week regardless of exactly when logrotate's daily history to cover a full week regardless of exactly when logrotate's daily
@@ -354,6 +362,16 @@ Internet → Apache :80/:443 (git.jayfield.org vhost)
### Gitea backup to `vlda-01` (`SYNC-PLAN.md` item 2b) ### Gitea backup to `vlda-01` (`SYNC-PLAN.md` item 2b)
Checked into this repo, split by which host each half runs on (they're on
different OSes with different conventions — see each `README.md`):
- Alpha side (Ubuntu 24.04): [`scripts/alpha/gitea-backup.sh`](scripts/alpha/gitea-backup.sh) /
[`scripts/alpha/cron.d/gitea-backup`](scripts/alpha/cron.d/gitea-backup)
— see `scripts/alpha/README.md`.
- `vlda-01` side (Unraid 7.3.2): [`scripts/clients/vlda-01/`](scripts/clients/vlda-01/)
— see `scripts/clients/vlda-01/README.md`, which also covers the
Unraid-specific persistence gotchas (RAM-backed root, VFAT `/boot`,
Unassigned Devices) that shaped how this had to be built.
Built 2026-07-26. A point-in-time backup, not a hot standby/live mirror — Built 2026-07-26. A point-in-time backup, not a hot standby/live mirror —
now that alpha is the canonical live Gitea, this keeps `vlda-01`'s copy now that alpha is the canonical live Gitea, this keeps `vlda-01`'s copy
from just going stale after the migration. Same network direction as from just going stale after the migration. Same network direction as
scripts/README.md
+36
View File
@@ -0,0 +1,36 @@
# Scripts and configs — checked-in reference copies
This directory holds the actual scripts and config files running on
`alpha.jayfield.org` and its backup client(s), checked into git so they're
versioned and reviewable alongside the prose docs that explain *why* they
exist (`README.md`, `TODO.md`, `SYNC-PLAN.md` in the repo root).
**These are reference copies, not a deployment source.** Editing a file
here does not change anything on the live server — there is no CI/CD or
sync mechanism pulling from this repo onto either host (matching this
whole repo's existing convention: "changes are made directly on the
server and recorded here afterward", see the root `README.md`). After
changing something on a live host, re-fetch it into this directory so the
checked-in copy doesn't drift from reality.
## Layout
- `alpha/` — scripts and configs that run **on `alpha.jayfield.org`
itself** (Ubuntu 24.04 LTS — see `alpha/README.md` for the exact
version/kernel).
- `clients/` — scripts and configs that run on other hosts this project
administers, each in its own subdirectory named after the host. Currently
just `clients/vlda-01/` (Unraid — see its own `README.md`), but structured
to hold more if that ever grows (see `SYNC-PLAN.md` for what else lives
on `vlda-01`).
**Why this split matters**: `alpha` and its clients run different, often
very different, operating systems (Ubuntu vs. Unraid so far) — a script
written for one's conventions (systemd, `/etc/cron.d`, standard persistent
filesystem) will not just drop into the other unchanged (Unraid's `/` and
`/usr` are RAM-backed and don't survive a reboot, its own scheduling is
via the User Scripts plugin, not raw crontab — see `clients/vlda-01/README.md`
for the specifics that tripped this up while building it). Each
subdirectory's `README.md` states the exact OS/version a script was
written for, so anyone reusing these later — including future-me — knows
whether a given script's assumptions still hold.
scripts/alpha/README.md
+40
View File
@@ -0,0 +1,40 @@
# Scripts running on `alpha.jayfield.org`
**OS**: Ubuntu 24.04.4 LTS (Noble Numbat), kernel `6.8.0-136-generic`
(upgraded from 22.04 on 2026-07-26 — see `TODO.md` in the repo root for
the full rollout). Standard Ubuntu conventions throughout: `systemd` for
services, `/etc/cron.d/` for scheduled jobs, a normal persistent root
filesystem — none of the "does this survive a reboot?" caveats that
`clients/vlda-01/` needs.
## Files, and where they're actually deployed
| File here | Deployed path | Purpose |
|---|---|---|
| `sanity-check.sh` | `/usr/local/sbin/sanity-check.sh` | Post-boot/post-change health check across the whole stack (DNS/DNSSEC, mail, MariaDB, fail2ban, Apache vhosts, Docker containers, Gitea). See root `README.md`'s "Post-boot sanity check". |
| `sanity-check.service` | `/etc/systemd/system/sanity-check.service` | Runs `sanity-check.sh` once per boot (`systemctl enable`d), `After=` the relevant services plus a 10s settle delay. |
| `attacker-check.py` | `/usr/local/sbin/attacker-check.py` | Weekly fail2ban log analysis — per-IP/per-jail CSVs for trend analysis, plus distributed-scan and regular-timing-evasion detection. See root `README.md`'s "Weekly attacker summary". |
| `cron.d/attacker-check` | `/etc/cron.d/attacker-check` | Runs `attacker-check.py` weekly, Sunday `06:00`. |
| `gitea-backup.sh` | `/usr/local/sbin/gitea-backup.sh` | Nightly `gitea dump` + sha256 checksum, ACL'd for the `giteabackup` pull account, pruned to 14 as a safety net. See root `README.md`'s "Gitea backup to `vlda-01`". |
| `cron.d/gitea-backup` | `/etc/cron.d/gitea-backup` | Runs `gitea-backup.sh` nightly, `02:30`. |
## Permissions as deployed (not preserved by git)
Git doesn't track exact ownership/mode the way these need to run as — set
these explicitly on (re-)deploy:
```bash
sudo chown root:root /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh}
sudo chmod 750 /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh}
sudo chown root:root /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service
sudo chmod 644 /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service
```
`gitea-backup.sh` additionally depends on:
- The `giteabackup` system account, its chrooted-SFTP `sshd` config
(reuses the existing `Match Group sftp` block — no new `sshd_config`
needed, just add the user to the `sftp` group), and the `rw` bind mount
at `/var/sftp/giteabackup/backups` (see root `README.md` for the full
ACL/permission model — this is **not** just "make the script executable
and it works", the restricted account is load-bearing for the whole
design).
scripts/alpha/attacker-check.py
+257
View File
@@ -0,0 +1,257 @@
#!/usr/bin/env python3
"""alpha.jayfield.org weekly attacker summary.
Parses fail2ban.log (current + recent rotations, gzipped or not) for the
past 7 days and produces two append-only CSVs for longitudinal analysis,
plus a human-readable summary emailed to root@jayfield.org and stored as
a dated report file on disk.
Run weekly via cron (/etc/cron.d/attacker-check). Safe to also run
manually at any time: sudo /usr/local/sbin/attacker-check.py
"""
import csv
import gzip
import re
import statistics
import subprocess
import sys
from collections import defaultdict
from datetime import datetime, timedelta
from pathlib import Path
# current + enough rotations to cover a week even if cron and logrotate's
# midnight timer land oddly relative to each other
LOG_FILES = [
"/var/log/fail2ban.log",
"/var/log/fail2ban.log.1",
"/var/log/fail2ban.log.2.gz",
]
DATA_DIR = Path("/var/log/attacker-check")
PER_IP_CSV = DATA_DIR / "per_ip_weekly.csv"
PER_JAIL_CSV = DATA_DIR / "per_jail_weekly.csv"
REPORTS_DIR = DATA_DIR / "reports"
REPORT_EMAIL = "root@jayfield.org"
# distributed-scan detection thresholds: a /24 needs at least this many
# distinct IPs, each averaging no more than this many events, to be
# flagged as a likely coordinated single-shot-per-IP scan
CAMPAIGN_MIN_IPS = 5
CAMPAIGN_MAX_AVG_EVENTS = 3.0
# "regular timing" flag: repeat hits on the same IP/jail with a
# coefficient of variation below this are suspiciously scripted rather
# than organic retry jitter (this is exactly the low-and-slow evasion
# pattern found twice this engagement - ~22-33min spacing to just miss
# fail2ban's findtime window)
REGULAR_TIMING_MAX_CV = 0.15
REGULAR_TIMING_MIN_EVENTS = 3
LINE_RE = re.compile(
r'^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}),\d+\s+\S+\s+\[\d+\]:\s+'
r'\S+\s+\[(?P<jail>[\w-]+)\]\s+(?P<action>Found|Ban|Unban)\s+'
r'(?P<ip>[0-9a-fA-F:.]+)'
)
def open_maybe_gzip(path: Path):
if path.suffix == ".gz":
return gzip.open(path, "rt", errors="replace")
return path.open("r", errors="replace")
def parse_events(since: datetime):
"""Returns list of (timestamp, jail, action, ip) for events >= since."""
events = []
for name in LOG_FILES:
path = Path(name)
if not path.exists():
continue
with open_maybe_gzip(path) as f:
for line in f:
m = LINE_RE.match(line)
if not m:
continue
if "Restore Ban" in line:
continue # persisted state on restart, not a new event
ts = datetime.strptime(m.group("ts"), "%Y-%m-%d %H:%M:%S")
if ts < since:
continue
events.append((ts, m.group("jail"), m.group("action"), m.group("ip")))
return events
def subnet24(ip: str) -> str:
parts = ip.split(".")
if len(parts) == 4:
return ".".join(parts[:3]) + ".0/24"
return ip # IPv6 or malformed - leave as-is, not bucketed
def write_csv(path: Path, rows: list, fieldnames: list):
if not rows:
return
write_header = not path.exists()
with path.open("a", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
if write_header:
writer.writeheader()
writer.writerows(rows)
def main():
DATA_DIR.mkdir(parents=True, exist_ok=True)
REPORTS_DIR.mkdir(parents=True, exist_ok=True)
now = datetime.now()
since = now - timedelta(days=7)
week_ending = now.date().isoformat()
events = parse_events(since)
found_by_key = defaultdict(list) # (jail, ip) -> [timestamps], sorted later
ban_counts = defaultdict(int) # jail -> new-ban count this week
for ts, jail, action, ip in events:
if action == "Found":
found_by_key[(jail, ip)].append(ts)
elif action == "Ban":
ban_counts[jail] += 1
per_ip_rows = []
subnet_ips = defaultdict(set) # (jail, subnet) -> {ip, ...}
for (jail, ip), timestamps in found_by_key.items():
timestamps.sort()
intervals = [
(timestamps[i + 1] - timestamps[i]).total_seconds()
for i in range(len(timestamps) - 1)
]
mean_iv = statistics.mean(intervals) if intervals else 0.0
stdev_iv = statistics.pstdev(intervals) if len(intervals) > 1 else 0.0
cv = (stdev_iv / mean_iv) if mean_iv > 0 else None
regular_timing = (
len(timestamps) >= REGULAR_TIMING_MIN_EVENTS
and cv is not None
and cv < REGULAR_TIMING_MAX_CV
)
sn = subnet24(ip)
subnet_ips[(jail, sn)].add(ip)
per_ip_rows.append({
"week_ending": week_ending,
"jail": jail,
"ip": ip,
"subnet_24": sn,
"event_count": len(timestamps),
"first_seen": timestamps[0].isoformat(),
"last_seen": timestamps[-1].isoformat(),
"mean_interval_s": round(mean_iv, 1),
"stdev_interval_s": round(stdev_iv, 1),
"regular_timing_flag": regular_timing,
})
jail_totals = defaultdict(lambda: {"events": 0, "ips": set(), "subnets": set()})
for (jail, ip), timestamps in found_by_key.items():
jail_totals[jail]["events"] += len(timestamps)
jail_totals[jail]["ips"].add(ip)
jail_totals[jail]["subnets"].add(subnet24(ip))
per_jail_rows = []
for jail, data in jail_totals.items():
per_jail_rows.append({
"week_ending": week_ending,
"jail": jail,
"total_events": data["events"],
"distinct_ips": len(data["ips"]),
"distinct_24_subnets": len(data["subnets"]),
"bans_this_week": ban_counts.get(jail, 0),
})
write_csv(PER_IP_CSV, per_ip_rows, [
"week_ending", "jail", "ip", "subnet_24", "event_count",
"first_seen", "last_seen", "mean_interval_s", "stdev_interval_s",
"regular_timing_flag",
])
write_csv(PER_JAIL_CSV, per_jail_rows, [
"week_ending", "jail", "total_events", "distinct_ips",
"distinct_24_subnets", "bans_this_week",
])
# distributed single-shot-per-IP scan detection (same shape as the
# apache-noscript campaign found 2026-07-19: many IPs, ~1 hit each)
campaigns = []
for (jail, sn), ips in subnet_ips.items():
if len(ips) < CAMPAIGN_MIN_IPS:
continue
avg_events = sum(len(found_by_key[(jail, ip)]) for ip in ips) / len(ips)
if avg_events <= CAMPAIGN_MAX_AVG_EVENTS:
campaigns.append((jail, sn, len(ips), round(avg_events, 1)))
lines = []
lines.append(f"alpha.jayfield.org attacker summary - week ending {week_ending}")
lines.append(f"Window: {since.isoformat()} .. {now.isoformat()}")
lines.append("")
lines.append("Per-jail totals:")
if per_jail_rows:
for row in sorted(per_jail_rows, key=lambda r: -r["total_events"]):
lines.append(
f" {row['jail']:<20} events={row['total_events']:<6} "
f"ips={row['distinct_ips']:<5} subnets={row['distinct_24_subnets']:<5} "
f"new_bans={row['bans_this_week']}"
)
else:
lines.append(" (no fail2ban activity this week)")
lines.append("")
top_ips = sorted(per_ip_rows, key=lambda r: -r["event_count"])[:15]
if top_ips:
lines.append("Top attacking IPs this week:")
for row in top_ips:
flag = " [REGULAR TIMING]" if row["regular_timing_flag"] else ""
lines.append(
f" {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} "
f"mean_interval={row['mean_interval_s']}s{flag}"
)
lines.append("")
if campaigns:
lines.append(
f"Possible distributed scan campaigns (>={CAMPAIGN_MIN_IPS} distinct IPs "
f"in one /24, <={CAMPAIGN_MAX_AVG_EVENTS} avg events/IP):"
)
for jail, sn, n_ips, avg_ev in sorted(campaigns, key=lambda c: -c[2]):
lines.append(f" {jail}: {sn} - {n_ips} distinct IPs, avg {avg_ev} events/IP")
else:
lines.append("No distributed-scan campaign pattern detected this week.")
lines.append("")
regular_ips = [r for r in per_ip_rows if r["regular_timing_flag"]]
if regular_ips:
lines.append(
"IPs with suspiciously regular timing (possible scripted "
"low-and-slow evasion, e.g. spaced just past a jail's findtime):"
)
for row in sorted(regular_ips, key=lambda r: -r["event_count"]):
lines.append(
f" {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} "
f"mean_interval={row['mean_interval_s']}s stdev={row['stdev_interval_s']}s"
)
lines.append("")
report_path = REPORTS_DIR / f"{week_ending}.txt"
lines.append(f"Data appended to {PER_IP_CSV.name} and {PER_JAIL_CSV.name} in {DATA_DIR}/ for trend analysis.")
lines.append(f"This report saved to {report_path}.")
report = "\n".join(lines)
print(report)
report_path.write_text(report + "\n")
try:
subprocess.run(
["mail", "-s", f"alpha.jayfield.org attacker summary - {week_ending}", REPORT_EMAIL],
input=report, text=True, check=True,
)
except Exception as e:
print(f"WARNING: failed to send email: {e}", file=sys.stderr)
if __name__ == "__main__":
main()
scripts/alpha/cron.d/attacker-check
+3
View File
@@ -0,0 +1,3 @@
# Weekly attacker/campaign summary from fail2ban logs.
# Sunday 06:00 - clear of logrotate's daily midnight run.
0 6 * * 0 root /usr/local/sbin/attacker-check.py >/var/log/attacker-check/last-run.log 2>&1
scripts/alpha/cron.d/gitea-backup
+3
View File
@@ -0,0 +1,3 @@
# Nightly Gitea backup (gitea dump), pulled by vlda-01 on its own schedule.
# 02:30 - clear of logrotate's daily midnight timer.
30 2 * * * root /usr/local/sbin/gitea-backup.sh >> /var/log/gitea-backup.log 2>&1
scripts/alpha/gitea-backup.sh
+38
View File
@@ -0,0 +1,38 @@
#!/bin/bash
# Nightly Gitea backup (alpha.jayfield.org side of the alpha -> vlda-01
# backup strategy, see SYNC-PLAN.md "Ongoing backup" section).
#
# Uses `gitea dump` rather than a raw copy of /srv/gitea/data, since the
# DB is a live SQLite file - dumping gives one consistent archive instead
# of risking a torn mid-write copy. vlda-01 pulls the result on its own
# schedule; this script only produces and prunes local archives.
set -u
BACKUP_DIR=/srv/gitea/data/backups
KEEP=14
FILE="gitea-dump-$(date +%Y%m%d-%H%M%S).zip"
if ! docker exec -u git gitea gitea dump -f "/data/backups/${FILE}" --skip-index -q; then
echo "$(date -Is) ERROR: gitea dump failed, see above" >&2
exit 1
fi
# vlda-01 verifies integrity before deleting its remote (alpha-side) copy
cd "$BACKUP_DIR" || exit 1
sha256sum "$FILE" > "${FILE}.sha256"
# the giteabackup SFTP account (see sshd Match Group sftp) needs read
# access to pull these - default ACL inheritance alone isn't enough, the
# file's own restrictive mask neutralizes the inherited grant, so set it
# explicitly on every new file (both the dump and its checksum)
setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}"
setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}.sha256"
# prune to the last $KEEP archives (safety net only - in normal operation
# vlda-01 deletes each dump right after a verified pull, well before this
# would ever trigger; this just bounds disk use if it's offline a while)
ls -1t gitea-dump-*.zip 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
rm -f "$old" "${old}.sha256"
done
echo "$(date -Is) backup complete: ${FILE} ($(du -h "${BACKUP_DIR}/${FILE}" 2>/dev/null | cut -f1)), $(ls -1 gitea-dump-*.zip 2>/dev/null | wc -l) archive(s) retained"
scripts/alpha/sanity-check.service
+13
View File
@@ -0,0 +1,13 @@
[Unit]
Description=Post-boot sanity check for alpha.jayfield.org
After=network-online.target named.service mariadb.service postfix@-.service dovecot.service fail2ban.service apache2.service docker.service rspamd.service
Wants=network-online.target
[Service]
Type=oneshot
ExecStartPre=/bin/sleep 10
ExecStart=/usr/local/sbin/sanity-check.sh
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
scripts/alpha/sanity-check.sh
+214
View File
@@ -0,0 +1,214 @@
#!/bin/bash
# alpha.jayfield.org post-change sanity check.
# Run manually after any significant change (apt upgrade, config change);
# runs automatically once per boot via sanity-check.service (covers OS
# upgrades, kernel updates, and anything else that ends in a reboot).
set -u
LOGDIR=/var/log/sanity-check
mkdir -p "$LOGDIR"
LOG="$LOGDIR/$(date +%Y%m%d-%H%M%S).log"
exec > >(tee -a "$LOG") 2>&1
PASS=0
FAIL=0
WARN=0
ok() { printf ' OK %s\n' "$1"; PASS=$((PASS+1)); }
bad() { printf ' FAIL %s\n' "$1"; FAIL=$((FAIL+1)); }
warn() { printf ' WARN %s\n' "$1"; WARN=$((WARN+1)); }
section() { printf '\n== %s ==\n' "$1"; }
echo "alpha.jayfield.org sanity check - $(date -Is)"
. /etc/os-release
echo "OS: $PRETTY_NAME, kernel $(uname -r), uptime $(uptime -p)"
# ---------- System package consistency ----------
section "System package consistency"
if [ -z "$(sudo dpkg --audit 2>&1)" ]; then
ok "dpkg --audit clean"
else
bad "dpkg --audit found issues"
fi
if sudo apt-get check >/dev/null 2>&1; then
ok "apt-get check clean"
else
bad "apt-get check reported broken dependencies"
fi
UPGRADABLE=$(apt list --upgradable 2>/dev/null | grep -vc '^Listing')
if [ "$UPGRADABLE" -eq 0 ]; then
ok "no pending package upgrades"
else
warn "$UPGRADABLE package(s) still upgradable"
fi
NEEDRESTART_SVC=$(sudo needrestart -b 2>/dev/null | grep -c '^NEEDRESTART-SVC')
if [ "$NEEDRESTART_SVC" -le 1 ]; then
ok "no outstanding service restarts needed (allowing the harmless per-login user@ unit)"
else
warn "$NEEDRESTART_SVC services still flagged by needrestart"
fi
# ---------- Disk space ----------
section "Disk space"
df -h / /var /boot
ROOT_PCT=$(df --output=pcent / | tail -1 | tr -dc '0-9')
if [ "$ROOT_PCT" -lt 90 ]; then
ok "root filesystem usage ${ROOT_PCT}%"
else
warn "root filesystem usage ${ROOT_PCT}% - getting full"
fi
# ---------- SSH ----------
section "SSH hardening"
SSHD_T=$(sudo sshd -T 2>/dev/null)
echo "$SSHD_T" | grep -q '^permitrootlogin no' && ok "PermitRootLogin no" || bad "PermitRootLogin not disabled"
echo "$SSHD_T" | grep -q '^passwordauthentication no' && ok "PasswordAuthentication no" || bad "PasswordAuthentication not disabled"
systemctl is-active --quiet ssh && ok "ssh.service active" || bad "ssh.service not active"
# ---------- DNS / BIND ----------
section "DNS / BIND"
systemctl is-active --quiet named && ok "named active" || bad "named not active"
if [ "$(dig @127.0.0.1 jayfield.org +short)" = "89.58.8.149" ]; then
ok "jayfield.org resolves to the expected IP"
else
bad "jayfield.org did not resolve to the expected IP"
fi
DNSSEC_STATUS=$(dig @127.0.0.1 github.com +noall +comment 2>/dev/null | grep -oP 'status: \K[A-Z]+')
if [ "$DNSSEC_STATUS" = "NOERROR" ]; then
ok "DNSSEC validation working (github.com -> NOERROR)"
else
bad "DNSSEC validation broken (github.com -> ${DNSSEC_STATUS:-no response})"
fi
for h in uschi vpn; do
IP=$(dig @127.0.0.1 "$h.dyndns.jayfield.org" A +short)
if [ -n "$IP" ]; then
ok "$h.dyndns.jayfield.org resolves ($IP)"
else
warn "$h.dyndns.jayfield.org has no A record"
fi
done
# ---------- Mail stack ----------
section "Mail stack"
systemctl is-active --quiet postfix@- && ok "postfix active" || bad "postfix not active"
systemctl is-active --quiet dovecot && ok "dovecot active" || bad "dovecot not active"
systemctl is-active --quiet rspamd && ok "rspamd active" || bad "rspamd not active"
SMTP_BANNER=$(printf 'QUIT\r\n' | timeout 5 nc 127.0.0.1 25 2>/dev/null | head -1)
echo "$SMTP_BANNER" | grep -q '220.*Postfix' && ok "SMTP banner OK" || bad "SMTP banner missing/unexpected: $SMTP_BANNER"
IMAP_BANNER=$(printf 'a LOGOUT\r\n' | timeout 5 nc 127.0.0.1 143 2>/dev/null | head -1)
echo "$IMAP_BANNER" | grep -qi 'dovecot ready\|^\* OK' && ok "IMAP banner OK" || bad "IMAP banner missing/unexpected: $IMAP_BANNER"
for port in 11332 11333 11334; do
if sudo ss -ltn | grep -q ":$port "; then
ok "rspamd listening on $port"
else
bad "rspamd NOT listening on $port"
fi
done
RSPAMD_TEST=$(printf 'Subject: sanity test\nFrom: test@example.com\nTo: jens@jayfield.org\n\nbody\n' | rspamc symbols 2>&1)
if echo "$RSPAMD_TEST" | grep -q 'Action:'; then
ok "rspamd genuinely scanning (rspamc symbols responded)"
else
bad "rspamd did not respond to a live scan test - check for a phantom 'active' status"
fi
if sudo ss -ltn | grep -q ':11332 '; then
ok "postfix's configured milter target (11332) is listening"
else
bad "nothing listening on postfix's milter port - mail is flowing unfiltered/unsigned"
fi
# ---------- MariaDB / vmail ----------
section "MariaDB / vmail"
systemctl is-active --quiet mariadb && ok "mariadb active" || bad "mariadb not active"
DOMAINS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.domains;" 2>/dev/null)
ACCOUNTS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.accounts;" 2>/dev/null)
[ -n "$DOMAINS" ] && [ "$DOMAINS" -ge 1 ] && ok "vmail.domains has $DOMAINS row(s)" || bad "vmail.domains query failed or returned zero rows"
[ -n "$ACCOUNTS" ] && [ "$ACCOUNTS" -ge 1 ] && ok "vmail.accounts has $ACCOUNTS row(s)" || bad "vmail.accounts query failed or returned zero rows"
if sudo postmap -q jayfield.org mysql:/etc/postfix/sql/domains.cf 2>/dev/null | grep -q jayfield.org; then
ok "Postfix's MySQL domain lookup works"
else
bad "Postfix's MySQL domain lookup failed"
fi
# ---------- fail2ban ----------
section "fail2ban"
systemctl is-active --quiet fail2ban && ok "fail2ban active" || bad "fail2ban not active"
JAIL_COUNT=$(sudo fail2ban-client status 2>/dev/null | grep -oP 'Number of jail:\s*\K[0-9]+')
if [ -n "$JAIL_COUNT" ] && [ "$JAIL_COUNT" -ge 15 ]; then
ok "$JAIL_COUNT jails loaded"
else
bad "only ${JAIL_COUNT:-0} jail(s) loaded (expected ~19)"
fi
SSHD_FT=$(sudo fail2ban-client get sshd findtime 2>/dev/null)
DOVECOT_FT=$(sudo fail2ban-client get dovecot findtime 2>/dev/null)
[ "$SSHD_FT" = "86400" ] && ok "sshd findtime hardening intact (86400s)" || bad "sshd findtime is ${SSHD_FT:-unknown}, expected 86400"
[ "$DOVECOT_FT" = "86400" ] && ok "dovecot findtime hardening intact (86400s)" || bad "dovecot findtime is ${DOVECOT_FT:-unknown}, expected 86400"
# ---------- Apache / vhosts ----------
section "Apache / vhosts"
systemctl is-active --quiet apache2 && ok "apache2 active" || bad "apache2 not active"
sudo apache2ctl configtest >/dev/null 2>&1 && ok "apache2ctl configtest clean" || bad "apache2ctl configtest failed"
check_url() {
local url=$1 expected=$2 code
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$url")
if [ "$code" = "$expected" ]; then
ok "$url -> $code"
else
bad "$url -> $code (expected $expected)"
fi
}
check_url "https://jayfield.org/" 200
check_url "https://www.jayfield.org/" 200
check_url "https://dyndns.jayfield.org/" 401
check_url "https://mail.jayfield.org/" 403
check_url "https://cloud.jayfield.org/" 302
check_url "https://portainer.jayfield.org/" 200
check_url "https://git.jayfield.org/" 200
# ---------- Docker ----------
section "Docker"
systemctl is-active --quiet docker && ok "docker active" || bad "docker not active"
for c in portainer nextcloud nextcloud-redis nextcloud-db web gitea; do
STATUS=$(sudo docker inspect -f '{{.State.Status}}' "$c" 2>/dev/null)
if [ "$STATUS" = "running" ]; then
ok "container $c running"
else
bad "container $c is '${STATUS:-missing}' (expected running)"
fi
done
GITEA_REPO_COUNT=$(curl -s --max-time 5 'https://git.jayfield.org/api/v1/repos/search?limit=1' | python3 -c 'import json,sys; print(len(json.load(sys.stdin).get("data",[])))' 2>/dev/null)
if [ "$GITEA_REPO_COUNT" = "1" ]; then
ok "Gitea API returns real repo data (not just a static page)"
else
bad "Gitea API did not return expected repo data"
fi
# ---------- Summary ----------
section "Summary"
echo "PASS: $PASS WARN: $WARN FAIL: $FAIL"
echo "Full log: $LOG"
if command -v mail >/dev/null 2>&1; then
SUBJECT="alpha.jayfield.org sanity check: $([ "$FAIL" -gt 0 ] && echo "$FAIL FAILURE(S)" || echo "OK")"
mail -s "$SUBJECT" jens@jayfield.org < "$LOG"
fi
[ "$FAIL" -eq 0 ]
scripts/clients/README.md
+14
View File
@@ -0,0 +1,14 @@
# Client hosts
Scripts and configs that run on other hosts this project administers —
not on `alpha.jayfield.org` itself — one subdirectory per host, named
after it. Each subdirectory's own `README.md` states that host's exact
OS/version, since a script written for one host's conventions generally
will not just work unchanged on another (see `../README.md` for why this
split exists at all).
## Hosts
- `vlda-01/` — the home intranet server referenced throughout
`SYNC-PLAN.md` (Unraid). Currently holds the pull side of the
`alpha` → `vlda-01` Gitea backup.
scripts/clients/vlda-01/README.md
+61
View File
@@ -0,0 +1,61 @@
# Scripts running on `vlda-01`
**OS**: Unraid `7.3.2`, kernel `6.18.38-Unraid`. **Very different
persistence model from Ubuntu/`alpha`** — this tripped up the first version
of every script here, so it's worth stating plainly before anything else:
- `/` and `/usr` are RAM-backed (`rootfs` / an `overlay` on top of the
read-only squashfs image loaded from the boot USB stick). **Anything
written there — a plain crontab, a script dropped in `/usr/local/sbin`
— is silently gone on the next reboot.** Only `/boot` (the actual USB
stick, mounted read/write as a normal filesystem) survives.
- `/boot` is **VFAT**, which has no concept of a Unix execute bit at all.
A script stored there can never be `chmod +x`'d in any way that sticks —
don't try to `exec` it directly from another script; invoke it via
`bash <path>` instead. (The **User Scripts** plugin itself handles this
correctly when it runs a script *through the plugin* — it copies the
content to a RAM-backed tmp location and `chmod +x`s that copy before
running it — but a script invoking *another* script directly, as the
`-onboot` wrapper here does, has to work around it manually.)
- Real, persistent storage lives on the array (`/mnt/user/...`) or on
**Unassigned Devices** — extra disks attached but not part of the
protected array, mounted at `/mnt/disks/<label>` — which is where these
backups actually land. An Unassigned Device is **not guaranteed to
already be mounted** when a script runs; see `gitea-backup-pull`'s own
handling of this below.
- Scheduled tasks persist via the **User Scripts** plugin (already
installed, already used for this box's other maintenance), which stores
scripts under `/boot/config/plugins/user.scripts/scripts/<name>/` and
scheduling in `/boot/config/plugins/user.scripts/schedule.json` — both
on the persistent USB stick. `update_cron` folds the JSON's cron-style
entries into the live `/etc/cron.d/root` at every boot; `"frequency":
"boot"` entries instead run once via the plugin's own `disks_mounted`
array-start event hook, **not** through cron at all.
- SSH keys under `/root/.ssh` needed no special persistence handling —
Unraid already auto-mirrors that whole directory to
`/boot/config/ssh/root/` on its own.
## Files, and where they're actually deployed
| File here | Deployed path | Purpose |
|---|---|---|
| `user-scripts/gitea-backup-pull/script` | `/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script` | Pulls new `gitea-dump` archives from `alpha` via the restricted `giteabackup` SFTP account, verifies each one's sha256 checksum, deletes both files on `alpha` only after a confirmed match, prunes its own local retention (30). Resolves and mounts the `WSD2L840` Unassigned Device itself first — see the script's own header comment for the full reasoning. |
| `user-scripts/gitea-backup-pull/description` | `.../gitea-backup-pull/description` | Shown in the User Scripts UI. |
| `user-scripts/gitea-backup-pull-onboot/script` | `.../gitea-backup-pull-onboot/script` | One-line wrapper: `bash`-invokes the script above. Exists only because `schedule.json` supports one schedule per script *path* — this is how the "also run once at every restart" requirement was added alongside the existing daily `03:30` schedule without disturbing it. |
| `user-scripts/gitea-backup-pull-onboot/description` | `.../gitea-backup-pull-onboot/description` | Shown in the User Scripts UI. |
| `user-scripts/schedule.json` | `/boot/config/plugins/user.scripts/schedule.json` | **Whole file, as deployed** — includes entries for this box's *other*, pre-existing scheduled maintenance (`delete_dangling_images`, `rc.fancontrol`, etc.) that predate this project and aren't described elsewhere in this repo. Only the `gitea-backup-pull` and `gitea-backup-pull-onboot` entries near the bottom belong to this project. |
## Redeploying from this checked-in copy
```bash
# on vlda-01:
mkdir -p /boot/config/plugins/user.scripts/scripts/gitea-backup-pull
mkdir -p /boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot
# copy script/description files into place from this repo, then:
cp /boot/config/plugins/user.scripts/schedule.json /tmp/user.scripts/schedule.json
/usr/local/sbin/update_cron
```
The dedicated ed25519 keypair this depends on (`/root/.ssh/gitea-backup/`,
authorized on `alpha`'s `giteabackup` account) is **not** in this repo —
regenerate it and re-authorize the new public key on `alpha` rather than
ever committing a private key here.
scripts/clients/vlda-01/user-scripts/gitea-backup-pull-onboot/description
+1
View File
@@ -0,0 +1 @@
Runs the gitea-backup-pull sync once at every array startup, in addition to its daily 03:30 schedule - so a restart after being offline catches up right away instead of waiting for the next scheduled time.
scripts/clients/vlda-01/user-scripts/gitea-backup-pull-onboot/script
+2
View File
@@ -0,0 +1,2 @@
#!/bin/bash
bash /boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script
scripts/clients/vlda-01/user-scripts/gitea-backup-pull/description
+1
View File
@@ -0,0 +1 @@
Pulls new gitea-dump backup archives from alpha.jayfield.org (git.jayfield.org) via the restricted giteabackup SFTP account. Part of the alpha -> vlda-01 backup strategy, see docs-alpha.jayfield.org/SYNC-PLAN.md.
scripts/clients/vlda-01/user-scripts/gitea-backup-pull/script
+130
View File
@@ -0,0 +1,130 @@
#!/bin/bash
# vlda-01 side of the alpha -> vlda-01 Gitea backup strategy
# (see SYNC-PLAN.md "Ongoing backup" section). Pulls whatever nightly
# gitea-dump archives exist on alpha that aren't already here yet - so a
# run after several days offline just catches up on all of them, not
# just the latest. Verifies each dump's sha256 checksum after pulling
# it, and only then deletes the alpha-side copy - a failed/incomplete
# verification leaves the remote copy alone for a retry next run.
#
# Uses SFTP (not real rsync) because the alpha-side account is
# restricted to `ForceCommand internal-sftp` - that blocks a real
# `rsync --server` invocation, so plain SFTP get/rm is the compatible
# transport here.
#
# Stored on the WSD2L840 Unassigned Device (not the array), which isn't
# guaranteed to already be mounted - this script mounts it itself first
# and refuses to proceed if that didn't actually result in a real
# mountpoint, rather than risk silently writing backups onto the array.
set -u
KEY=/root/.ssh/gitea-backup/id_ed25519
KNOWN_HOSTS=/root/.ssh/gitea-backup/known_hosts
REMOTE=giteabackup@alpha.jayfield.org
REMOTE_PORT=10022
UD_LABEL=WSD2L840
UD_MOUNTPOINT=/mnt/disks/WSD2L840
LOCAL_DIR="$UD_MOUNTPOINT/git"
KEEP=30
# WSD2L840 is an Unassigned Device, not part of the array - not
# guaranteed to already be mounted when this runs. Resolve by label
# (not a hardcoded /dev/sdX - USB/UD device paths aren't stable across
# reboots) and mount it via the Unassigned Devices plugin's own tool,
# which is safe to call even if it's already mounted.
UD_DEV=$(findfs LABEL="$UD_LABEL" 2>/dev/null)
if [ -z "$UD_DEV" ]; then
echo "$(date -Is) ERROR: no device labeled $UD_LABEL found - is it plugged in?" >&2
exit 1
fi
/usr/local/sbin/rc.unassigned mount "$UD_DEV" >/dev/null 2>&1
if ! mountpoint -q "$UD_MOUNTPOINT"; then
echo "$(date -Is) ERROR: $UD_MOUNTPOINT did not come up as a real mountpoint after mount attempt - refusing to write backups onto the array by mistake" >&2
exit 1
fi
mkdir -p "$LOCAL_DIR"
SFTP="sftp -P $REMOTE_PORT -i $KEY -o UserKnownHostsFile=$KNOWN_HOSTS -o BatchMode=yes"
REMOTE_LISTING=$($SFTP "$REMOTE" <<< 'ls -1 backups/' 2>/dev/null)
if [ -z "$REMOTE_LISTING" ]; then
echo "$(date -Is) ERROR: could not list remote backups (host down or auth failed)" >&2
exit 1
fi
REMOTE_ZIPS=$(echo "$REMOTE_LISTING" | grep '\.zip$')
GET_BATCH=$(mktemp)
DEL_BATCH=$(mktemp)
trap 'rm -f "$GET_BATCH" "$DEL_BATCH"' EXIT
TO_FETCH=0
while read -r fname; do
[ -z "$fname" ] && continue
base=$(basename "$fname")
if [ ! -f "$LOCAL_DIR/$base" ]; then
echo "get $fname $LOCAL_DIR/" >> "$GET_BATCH"
TO_FETCH=$((TO_FETCH + 1))
# grab the checksum alongside it if alpha actually has one
# (older dumps from before checksums existed won't)
if echo "$REMOTE_LISTING" | grep -qxF "${fname}.sha256"; then
echo "get ${fname}.sha256 $LOCAL_DIR/" >> "$GET_BATCH"
fi
fi
done <<< "$REMOTE_ZIPS"
if [ "$TO_FETCH" -eq 0 ]; then
echo "$(date -Is) up to date, nothing new to pull"
exit 0
fi
if ! $SFTP -b "$GET_BATCH" "$REMOTE" 2>&1; then
echo "$(date -Is) ERROR: sftp batch fetch failed partway through" >&2
exit 1
fi
# verify each freshly-fetched dump before trusting it enough to delete
# the alpha-side copy
VERIFIED=0
UNVERIFIED=0
while read -r fname; do
[ -z "$fname" ] && continue
base=$(basename "$fname")
local_zip="$LOCAL_DIR/$base"
local_sum="$LOCAL_DIR/$base.sha256"
[ -f "$local_zip" ] || continue # wasn't newly fetched this run
if [ ! -f "$local_sum" ]; then
echo "$(date -Is) WARNING: no checksum available for $base (older dump, predates checksums?) - keeping local copy, leaving alpha's copy alone" >&2
UNVERIFIED=$((UNVERIFIED + 1))
continue
fi
if (cd "$LOCAL_DIR" && sha256sum -c "$(basename "$local_sum")") >/dev/null 2>&1; then
echo "rm $fname" >> "$DEL_BATCH"
echo "rm ${fname}.sha256" >> "$DEL_BATCH"
VERIFIED=$((VERIFIED + 1))
else
echo "$(date -Is) ERROR: checksum mismatch for $base - deleting the bad local copy, leaving alpha's copy for a retry" >&2
rm -f "$local_zip" "$local_sum"
UNVERIFIED=$((UNVERIFIED + 1))
fi
done <<< "$REMOTE_ZIPS"
if [ -s "$DEL_BATCH" ]; then
if ! $SFTP -b "$DEL_BATCH" "$REMOTE" 2>&1; then
echo "$(date -Is) WARNING: some verified files could not be deleted on alpha - harmless, alpha's own retention prune will clean them up eventually" >&2
fi
fi
# prune to the last $KEEP archives (vlda-01 has far more headroom than
# alpha, so this is the real long-term history now that alpha deletes
# its own copies right after a verified pull)
cd "$LOCAL_DIR" || exit 1
ls -1t gitea-dump-*.zip 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
rm -f "$old" "${old}.sha256"
done
echo "$(date -Is) pulled $TO_FETCH new archive(s), $VERIFIED verified+deleted from alpha, $UNVERIFIED kept unverified, $(ls -1 gitea-dump-*.zip 2>/dev/null | wc -l) retained locally"
scripts/clients/vlda-01/user-scripts/schedule.json
+38
View File
@@ -0,0 +1,38 @@
{
"/boot/config/plugins/user.scripts/scripts/delete_dangling_images/script": {
"script": "/boot/config/plugins/user.scripts/scripts/delete_dangling_images/script",
"frequency": "disabled",
"id": "scheduledelete_dangling_images",
"custom": ""
},
"/boot/config/plugins/user.scripts/scripts/delete.ds_store/script": {
"script": "/boot/config/plugins/user.scripts/scripts/delete.ds_store/script",
"frequency": "disabled",
"id": "scheduledelete-ds_store",
"custom": ""
},
"/boot/config/plugins/user.scripts/scripts/rc.fancontrol/script": {
"script": "/boot/config/plugins/user.scripts/scripts/rc.fancontrol/script",
"frequency": "boot",
"id": "schedulerc-fancontrol",
"custom": ""
},
"/boot/config/plugins/user.scripts/scripts/viewDockerLogSize/script": {
"script": "/boot/config/plugins/user.scripts/scripts/viewDockerLogSize/script",
"frequency": "disabled",
"id": "scheduleviewDockerLogSize",
"custom": ""
},
"/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script": {
"script": "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script",
"frequency": "custom",
"id": "schedulegitea-backup-pull",
"custom": "30 3 * * *"
},
"/boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot/script": {
"script": "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot/script",
"frequency": "boot",
"id": "schedulegitea-backup-pull-onboot",
"custom": ""
}
}