Check in scripts/configs, organized and documented by host OS
Every script/config built this session (sanity-check, attacker-check, the Gitea backup pipeline on both ends) now lives in this repo as reference copies, not just described in prose - previously they only existed on the live servers. Split into scripts/alpha/ (Ubuntu 24.04.4 LTS) and scripts/clients/vlda-01/ (Unraid 7.3.2), each with its own README stating the exact OS/kernel and a file-by-file map to deployed paths, since a script written for one host's conventions doesn't just work unchanged on the other - the vlda-01 README in particular documents the persistence gotchas that actually broke earlier attempts (RAM- backed root filesystem, VFAT /boot with no execute bit, Unassigned Device auto-mount). Root README.md now links directly to these files from each relevant section instead of only describing them. Explicitly not included: the dedicated SSH private key vlda-01 uses to authenticate to alpha - noted in clients/vlda-01/README.md to regenerate rather than ever commit one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,40 @@
|
||||
# Scripts running on `alpha.jayfield.org`
|
||||
|
||||
**OS**: Ubuntu 24.04.4 LTS (Noble Numbat), kernel `6.8.0-136-generic`
|
||||
(upgraded from 22.04 on 2026-07-26 — see `TODO.md` in the repo root for
|
||||
the full rollout). Standard Ubuntu conventions throughout: `systemd` for
|
||||
services, `/etc/cron.d/` for scheduled jobs, a normal persistent root
|
||||
filesystem — none of the "does this survive a reboot?" caveats that
|
||||
`clients/vlda-01/` needs.
|
||||
|
||||
## Files, and where they're actually deployed
|
||||
|
||||
| File here | Deployed path | Purpose |
|
||||
|---|---|---|
|
||||
| `sanity-check.sh` | `/usr/local/sbin/sanity-check.sh` | Post-boot/post-change health check across the whole stack (DNS/DNSSEC, mail, MariaDB, fail2ban, Apache vhosts, Docker containers, Gitea). See root `README.md`'s "Post-boot sanity check". |
|
||||
| `sanity-check.service` | `/etc/systemd/system/sanity-check.service` | Runs `sanity-check.sh` once per boot (`systemctl enable`d), `After=` the relevant services plus a 10s settle delay. |
|
||||
| `attacker-check.py` | `/usr/local/sbin/attacker-check.py` | Weekly fail2ban log analysis — per-IP/per-jail CSVs for trend analysis, plus distributed-scan and regular-timing-evasion detection. See root `README.md`'s "Weekly attacker summary". |
|
||||
| `cron.d/attacker-check` | `/etc/cron.d/attacker-check` | Runs `attacker-check.py` weekly, Sunday `06:00`. |
|
||||
| `gitea-backup.sh` | `/usr/local/sbin/gitea-backup.sh` | Nightly `gitea dump` + sha256 checksum, ACL'd for the `giteabackup` pull account, pruned to 14 as a safety net. See root `README.md`'s "Gitea backup to `vlda-01`". |
|
||||
| `cron.d/gitea-backup` | `/etc/cron.d/gitea-backup` | Runs `gitea-backup.sh` nightly, `02:30`. |
|
||||
|
||||
## Permissions as deployed (not preserved by git)
|
||||
|
||||
Git doesn't track exact ownership/mode the way these need to run as — set
|
||||
these explicitly on (re-)deploy:
|
||||
|
||||
```bash
|
||||
sudo chown root:root /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh}
|
||||
sudo chmod 750 /usr/local/sbin/{sanity-check.sh,attacker-check.py,gitea-backup.sh}
|
||||
sudo chown root:root /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service
|
||||
sudo chmod 644 /etc/cron.d/{attacker-check,gitea-backup} /etc/systemd/system/sanity-check.service
|
||||
```
|
||||
|
||||
`gitea-backup.sh` additionally depends on:
|
||||
- The `giteabackup` system account, its chrooted-SFTP `sshd` config
|
||||
(reuses the existing `Match Group sftp` block — no new `sshd_config`
|
||||
needed, just add the user to the `sftp` group), and the `rw` bind mount
|
||||
at `/var/sftp/giteabackup/backups` (see root `README.md` for the full
|
||||
ACL/permission model — this is **not** just "make the script executable
|
||||
and it works", the restricted account is load-bearing for the whole
|
||||
design).
|
||||
@@ -0,0 +1,257 @@
|
||||
#!/usr/bin/env python3
|
||||
"""alpha.jayfield.org weekly attacker summary.
|
||||
|
||||
Parses fail2ban.log (current + recent rotations, gzipped or not) for the
|
||||
past 7 days and produces two append-only CSVs for longitudinal analysis,
|
||||
plus a human-readable summary emailed to root@jayfield.org and stored as
|
||||
a dated report file on disk.
|
||||
|
||||
Run weekly via cron (/etc/cron.d/attacker-check). Safe to also run
|
||||
manually at any time: sudo /usr/local/sbin/attacker-check.py
|
||||
"""
|
||||
import csv
|
||||
import gzip
|
||||
import re
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
# current + enough rotations to cover a week even if cron and logrotate's
|
||||
# midnight timer land oddly relative to each other
|
||||
LOG_FILES = [
|
||||
"/var/log/fail2ban.log",
|
||||
"/var/log/fail2ban.log.1",
|
||||
"/var/log/fail2ban.log.2.gz",
|
||||
]
|
||||
DATA_DIR = Path("/var/log/attacker-check")
|
||||
PER_IP_CSV = DATA_DIR / "per_ip_weekly.csv"
|
||||
PER_JAIL_CSV = DATA_DIR / "per_jail_weekly.csv"
|
||||
REPORTS_DIR = DATA_DIR / "reports"
|
||||
REPORT_EMAIL = "root@jayfield.org"
|
||||
|
||||
# distributed-scan detection thresholds: a /24 needs at least this many
|
||||
# distinct IPs, each averaging no more than this many events, to be
|
||||
# flagged as a likely coordinated single-shot-per-IP scan
|
||||
CAMPAIGN_MIN_IPS = 5
|
||||
CAMPAIGN_MAX_AVG_EVENTS = 3.0
|
||||
|
||||
# "regular timing" flag: repeat hits on the same IP/jail with a
|
||||
# coefficient of variation below this are suspiciously scripted rather
|
||||
# than organic retry jitter (this is exactly the low-and-slow evasion
|
||||
# pattern found twice this engagement - ~22-33min spacing to just miss
|
||||
# fail2ban's findtime window)
|
||||
REGULAR_TIMING_MAX_CV = 0.15
|
||||
REGULAR_TIMING_MIN_EVENTS = 3
|
||||
|
||||
LINE_RE = re.compile(
|
||||
r'^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}),\d+\s+\S+\s+\[\d+\]:\s+'
|
||||
r'\S+\s+\[(?P<jail>[\w-]+)\]\s+(?P<action>Found|Ban|Unban)\s+'
|
||||
r'(?P<ip>[0-9a-fA-F:.]+)'
|
||||
)
|
||||
|
||||
|
||||
def open_maybe_gzip(path: Path):
|
||||
if path.suffix == ".gz":
|
||||
return gzip.open(path, "rt", errors="replace")
|
||||
return path.open("r", errors="replace")
|
||||
|
||||
|
||||
def parse_events(since: datetime):
|
||||
"""Returns list of (timestamp, jail, action, ip) for events >= since."""
|
||||
events = []
|
||||
for name in LOG_FILES:
|
||||
path = Path(name)
|
||||
if not path.exists():
|
||||
continue
|
||||
with open_maybe_gzip(path) as f:
|
||||
for line in f:
|
||||
m = LINE_RE.match(line)
|
||||
if not m:
|
||||
continue
|
||||
if "Restore Ban" in line:
|
||||
continue # persisted state on restart, not a new event
|
||||
ts = datetime.strptime(m.group("ts"), "%Y-%m-%d %H:%M:%S")
|
||||
if ts < since:
|
||||
continue
|
||||
events.append((ts, m.group("jail"), m.group("action"), m.group("ip")))
|
||||
return events
|
||||
|
||||
|
||||
def subnet24(ip: str) -> str:
|
||||
parts = ip.split(".")
|
||||
if len(parts) == 4:
|
||||
return ".".join(parts[:3]) + ".0/24"
|
||||
return ip # IPv6 or malformed - leave as-is, not bucketed
|
||||
|
||||
|
||||
def write_csv(path: Path, rows: list, fieldnames: list):
|
||||
if not rows:
|
||||
return
|
||||
write_header = not path.exists()
|
||||
with path.open("a", newline="") as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
||||
if write_header:
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
|
||||
def main():
|
||||
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||
REPORTS_DIR.mkdir(parents=True, exist_ok=True)
|
||||
now = datetime.now()
|
||||
since = now - timedelta(days=7)
|
||||
week_ending = now.date().isoformat()
|
||||
|
||||
events = parse_events(since)
|
||||
|
||||
found_by_key = defaultdict(list) # (jail, ip) -> [timestamps], sorted later
|
||||
ban_counts = defaultdict(int) # jail -> new-ban count this week
|
||||
for ts, jail, action, ip in events:
|
||||
if action == "Found":
|
||||
found_by_key[(jail, ip)].append(ts)
|
||||
elif action == "Ban":
|
||||
ban_counts[jail] += 1
|
||||
|
||||
per_ip_rows = []
|
||||
subnet_ips = defaultdict(set) # (jail, subnet) -> {ip, ...}
|
||||
for (jail, ip), timestamps in found_by_key.items():
|
||||
timestamps.sort()
|
||||
intervals = [
|
||||
(timestamps[i + 1] - timestamps[i]).total_seconds()
|
||||
for i in range(len(timestamps) - 1)
|
||||
]
|
||||
mean_iv = statistics.mean(intervals) if intervals else 0.0
|
||||
stdev_iv = statistics.pstdev(intervals) if len(intervals) > 1 else 0.0
|
||||
cv = (stdev_iv / mean_iv) if mean_iv > 0 else None
|
||||
regular_timing = (
|
||||
len(timestamps) >= REGULAR_TIMING_MIN_EVENTS
|
||||
and cv is not None
|
||||
and cv < REGULAR_TIMING_MAX_CV
|
||||
)
|
||||
sn = subnet24(ip)
|
||||
subnet_ips[(jail, sn)].add(ip)
|
||||
per_ip_rows.append({
|
||||
"week_ending": week_ending,
|
||||
"jail": jail,
|
||||
"ip": ip,
|
||||
"subnet_24": sn,
|
||||
"event_count": len(timestamps),
|
||||
"first_seen": timestamps[0].isoformat(),
|
||||
"last_seen": timestamps[-1].isoformat(),
|
||||
"mean_interval_s": round(mean_iv, 1),
|
||||
"stdev_interval_s": round(stdev_iv, 1),
|
||||
"regular_timing_flag": regular_timing,
|
||||
})
|
||||
|
||||
jail_totals = defaultdict(lambda: {"events": 0, "ips": set(), "subnets": set()})
|
||||
for (jail, ip), timestamps in found_by_key.items():
|
||||
jail_totals[jail]["events"] += len(timestamps)
|
||||
jail_totals[jail]["ips"].add(ip)
|
||||
jail_totals[jail]["subnets"].add(subnet24(ip))
|
||||
|
||||
per_jail_rows = []
|
||||
for jail, data in jail_totals.items():
|
||||
per_jail_rows.append({
|
||||
"week_ending": week_ending,
|
||||
"jail": jail,
|
||||
"total_events": data["events"],
|
||||
"distinct_ips": len(data["ips"]),
|
||||
"distinct_24_subnets": len(data["subnets"]),
|
||||
"bans_this_week": ban_counts.get(jail, 0),
|
||||
})
|
||||
|
||||
write_csv(PER_IP_CSV, per_ip_rows, [
|
||||
"week_ending", "jail", "ip", "subnet_24", "event_count",
|
||||
"first_seen", "last_seen", "mean_interval_s", "stdev_interval_s",
|
||||
"regular_timing_flag",
|
||||
])
|
||||
write_csv(PER_JAIL_CSV, per_jail_rows, [
|
||||
"week_ending", "jail", "total_events", "distinct_ips",
|
||||
"distinct_24_subnets", "bans_this_week",
|
||||
])
|
||||
|
||||
# distributed single-shot-per-IP scan detection (same shape as the
|
||||
# apache-noscript campaign found 2026-07-19: many IPs, ~1 hit each)
|
||||
campaigns = []
|
||||
for (jail, sn), ips in subnet_ips.items():
|
||||
if len(ips) < CAMPAIGN_MIN_IPS:
|
||||
continue
|
||||
avg_events = sum(len(found_by_key[(jail, ip)]) for ip in ips) / len(ips)
|
||||
if avg_events <= CAMPAIGN_MAX_AVG_EVENTS:
|
||||
campaigns.append((jail, sn, len(ips), round(avg_events, 1)))
|
||||
|
||||
lines = []
|
||||
lines.append(f"alpha.jayfield.org attacker summary - week ending {week_ending}")
|
||||
lines.append(f"Window: {since.isoformat()} .. {now.isoformat()}")
|
||||
lines.append("")
|
||||
|
||||
lines.append("Per-jail totals:")
|
||||
if per_jail_rows:
|
||||
for row in sorted(per_jail_rows, key=lambda r: -r["total_events"]):
|
||||
lines.append(
|
||||
f" {row['jail']:<20} events={row['total_events']:<6} "
|
||||
f"ips={row['distinct_ips']:<5} subnets={row['distinct_24_subnets']:<5} "
|
||||
f"new_bans={row['bans_this_week']}"
|
||||
)
|
||||
else:
|
||||
lines.append(" (no fail2ban activity this week)")
|
||||
lines.append("")
|
||||
|
||||
top_ips = sorted(per_ip_rows, key=lambda r: -r["event_count"])[:15]
|
||||
if top_ips:
|
||||
lines.append("Top attacking IPs this week:")
|
||||
for row in top_ips:
|
||||
flag = " [REGULAR TIMING]" if row["regular_timing_flag"] else ""
|
||||
lines.append(
|
||||
f" {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} "
|
||||
f"mean_interval={row['mean_interval_s']}s{flag}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
if campaigns:
|
||||
lines.append(
|
||||
f"Possible distributed scan campaigns (>={CAMPAIGN_MIN_IPS} distinct IPs "
|
||||
f"in one /24, <={CAMPAIGN_MAX_AVG_EVENTS} avg events/IP):"
|
||||
)
|
||||
for jail, sn, n_ips, avg_ev in sorted(campaigns, key=lambda c: -c[2]):
|
||||
lines.append(f" {jail}: {sn} - {n_ips} distinct IPs, avg {avg_ev} events/IP")
|
||||
else:
|
||||
lines.append("No distributed-scan campaign pattern detected this week.")
|
||||
lines.append("")
|
||||
|
||||
regular_ips = [r for r in per_ip_rows if r["regular_timing_flag"]]
|
||||
if regular_ips:
|
||||
lines.append(
|
||||
"IPs with suspiciously regular timing (possible scripted "
|
||||
"low-and-slow evasion, e.g. spaced just past a jail's findtime):"
|
||||
)
|
||||
for row in sorted(regular_ips, key=lambda r: -r["event_count"]):
|
||||
lines.append(
|
||||
f" {row['ip']:<16} jail={row['jail']:<18} events={row['event_count']:<5} "
|
||||
f"mean_interval={row['mean_interval_s']}s stdev={row['stdev_interval_s']}s"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
report_path = REPORTS_DIR / f"{week_ending}.txt"
|
||||
lines.append(f"Data appended to {PER_IP_CSV.name} and {PER_JAIL_CSV.name} in {DATA_DIR}/ for trend analysis.")
|
||||
lines.append(f"This report saved to {report_path}.")
|
||||
|
||||
report = "\n".join(lines)
|
||||
print(report)
|
||||
|
||||
report_path.write_text(report + "\n")
|
||||
|
||||
try:
|
||||
subprocess.run(
|
||||
["mail", "-s", f"alpha.jayfield.org attacker summary - {week_ending}", REPORT_EMAIL],
|
||||
input=report, text=True, check=True,
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"WARNING: failed to send email: {e}", file=sys.stderr)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,3 @@
|
||||
# Weekly attacker/campaign summary from fail2ban logs.
|
||||
# Sunday 06:00 - clear of logrotate's daily midnight run.
|
||||
0 6 * * 0 root /usr/local/sbin/attacker-check.py >/var/log/attacker-check/last-run.log 2>&1
|
||||
@@ -0,0 +1,3 @@
|
||||
# Nightly Gitea backup (gitea dump), pulled by vlda-01 on its own schedule.
|
||||
# 02:30 - clear of logrotate's daily midnight timer.
|
||||
30 2 * * * root /usr/local/sbin/gitea-backup.sh >> /var/log/gitea-backup.log 2>&1
|
||||
@@ -0,0 +1,38 @@
|
||||
#!/bin/bash
|
||||
# Nightly Gitea backup (alpha.jayfield.org side of the alpha -> vlda-01
|
||||
# backup strategy, see SYNC-PLAN.md "Ongoing backup" section).
|
||||
#
|
||||
# Uses `gitea dump` rather than a raw copy of /srv/gitea/data, since the
|
||||
# DB is a live SQLite file - dumping gives one consistent archive instead
|
||||
# of risking a torn mid-write copy. vlda-01 pulls the result on its own
|
||||
# schedule; this script only produces and prunes local archives.
|
||||
|
||||
set -u
|
||||
BACKUP_DIR=/srv/gitea/data/backups
|
||||
KEEP=14
|
||||
FILE="gitea-dump-$(date +%Y%m%d-%H%M%S).zip"
|
||||
|
||||
if ! docker exec -u git gitea gitea dump -f "/data/backups/${FILE}" --skip-index -q; then
|
||||
echo "$(date -Is) ERROR: gitea dump failed, see above" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# vlda-01 verifies integrity before deleting its remote (alpha-side) copy
|
||||
cd "$BACKUP_DIR" || exit 1
|
||||
sha256sum "$FILE" > "${FILE}.sha256"
|
||||
|
||||
# the giteabackup SFTP account (see sshd Match Group sftp) needs read
|
||||
# access to pull these - default ACL inheritance alone isn't enough, the
|
||||
# file's own restrictive mask neutralizes the inherited grant, so set it
|
||||
# explicitly on every new file (both the dump and its checksum)
|
||||
setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}"
|
||||
setfacl -m u:giteabackup:r "${BACKUP_DIR}/${FILE}.sha256"
|
||||
|
||||
# prune to the last $KEEP archives (safety net only - in normal operation
|
||||
# vlda-01 deletes each dump right after a verified pull, well before this
|
||||
# would ever trigger; this just bounds disk use if it's offline a while)
|
||||
ls -1t gitea-dump-*.zip 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
|
||||
rm -f "$old" "${old}.sha256"
|
||||
done
|
||||
|
||||
echo "$(date -Is) backup complete: ${FILE} ($(du -h "${BACKUP_DIR}/${FILE}" 2>/dev/null | cut -f1)), $(ls -1 gitea-dump-*.zip 2>/dev/null | wc -l) archive(s) retained"
|
||||
@@ -0,0 +1,13 @@
|
||||
[Unit]
|
||||
Description=Post-boot sanity check for alpha.jayfield.org
|
||||
After=network-online.target named.service mariadb.service postfix@-.service dovecot.service fail2ban.service apache2.service docker.service rspamd.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStartPre=/bin/sleep 10
|
||||
ExecStart=/usr/local/sbin/sanity-check.sh
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,214 @@
|
||||
#!/bin/bash
|
||||
# alpha.jayfield.org post-change sanity check.
|
||||
# Run manually after any significant change (apt upgrade, config change);
|
||||
# runs automatically once per boot via sanity-check.service (covers OS
|
||||
# upgrades, kernel updates, and anything else that ends in a reboot).
|
||||
|
||||
set -u
|
||||
|
||||
LOGDIR=/var/log/sanity-check
|
||||
mkdir -p "$LOGDIR"
|
||||
LOG="$LOGDIR/$(date +%Y%m%d-%H%M%S).log"
|
||||
exec > >(tee -a "$LOG") 2>&1
|
||||
|
||||
PASS=0
|
||||
FAIL=0
|
||||
WARN=0
|
||||
|
||||
ok() { printf ' OK %s\n' "$1"; PASS=$((PASS+1)); }
|
||||
bad() { printf ' FAIL %s\n' "$1"; FAIL=$((FAIL+1)); }
|
||||
warn() { printf ' WARN %s\n' "$1"; WARN=$((WARN+1)); }
|
||||
section() { printf '\n== %s ==\n' "$1"; }
|
||||
|
||||
echo "alpha.jayfield.org sanity check - $(date -Is)"
|
||||
. /etc/os-release
|
||||
echo "OS: $PRETTY_NAME, kernel $(uname -r), uptime $(uptime -p)"
|
||||
|
||||
# ---------- System package consistency ----------
|
||||
section "System package consistency"
|
||||
if [ -z "$(sudo dpkg --audit 2>&1)" ]; then
|
||||
ok "dpkg --audit clean"
|
||||
else
|
||||
bad "dpkg --audit found issues"
|
||||
fi
|
||||
|
||||
if sudo apt-get check >/dev/null 2>&1; then
|
||||
ok "apt-get check clean"
|
||||
else
|
||||
bad "apt-get check reported broken dependencies"
|
||||
fi
|
||||
|
||||
UPGRADABLE=$(apt list --upgradable 2>/dev/null | grep -vc '^Listing')
|
||||
if [ "$UPGRADABLE" -eq 0 ]; then
|
||||
ok "no pending package upgrades"
|
||||
else
|
||||
warn "$UPGRADABLE package(s) still upgradable"
|
||||
fi
|
||||
|
||||
NEEDRESTART_SVC=$(sudo needrestart -b 2>/dev/null | grep -c '^NEEDRESTART-SVC')
|
||||
if [ "$NEEDRESTART_SVC" -le 1 ]; then
|
||||
ok "no outstanding service restarts needed (allowing the harmless per-login user@ unit)"
|
||||
else
|
||||
warn "$NEEDRESTART_SVC services still flagged by needrestart"
|
||||
fi
|
||||
|
||||
# ---------- Disk space ----------
|
||||
section "Disk space"
|
||||
df -h / /var /boot
|
||||
ROOT_PCT=$(df --output=pcent / | tail -1 | tr -dc '0-9')
|
||||
if [ "$ROOT_PCT" -lt 90 ]; then
|
||||
ok "root filesystem usage ${ROOT_PCT}%"
|
||||
else
|
||||
warn "root filesystem usage ${ROOT_PCT}% - getting full"
|
||||
fi
|
||||
|
||||
# ---------- SSH ----------
|
||||
section "SSH hardening"
|
||||
SSHD_T=$(sudo sshd -T 2>/dev/null)
|
||||
echo "$SSHD_T" | grep -q '^permitrootlogin no' && ok "PermitRootLogin no" || bad "PermitRootLogin not disabled"
|
||||
echo "$SSHD_T" | grep -q '^passwordauthentication no' && ok "PasswordAuthentication no" || bad "PasswordAuthentication not disabled"
|
||||
systemctl is-active --quiet ssh && ok "ssh.service active" || bad "ssh.service not active"
|
||||
|
||||
# ---------- DNS / BIND ----------
|
||||
section "DNS / BIND"
|
||||
systemctl is-active --quiet named && ok "named active" || bad "named not active"
|
||||
|
||||
if [ "$(dig @127.0.0.1 jayfield.org +short)" = "89.58.8.149" ]; then
|
||||
ok "jayfield.org resolves to the expected IP"
|
||||
else
|
||||
bad "jayfield.org did not resolve to the expected IP"
|
||||
fi
|
||||
|
||||
DNSSEC_STATUS=$(dig @127.0.0.1 github.com +noall +comment 2>/dev/null | grep -oP 'status: \K[A-Z]+')
|
||||
if [ "$DNSSEC_STATUS" = "NOERROR" ]; then
|
||||
ok "DNSSEC validation working (github.com -> NOERROR)"
|
||||
else
|
||||
bad "DNSSEC validation broken (github.com -> ${DNSSEC_STATUS:-no response})"
|
||||
fi
|
||||
|
||||
for h in uschi vpn; do
|
||||
IP=$(dig @127.0.0.1 "$h.dyndns.jayfield.org" A +short)
|
||||
if [ -n "$IP" ]; then
|
||||
ok "$h.dyndns.jayfield.org resolves ($IP)"
|
||||
else
|
||||
warn "$h.dyndns.jayfield.org has no A record"
|
||||
fi
|
||||
done
|
||||
|
||||
# ---------- Mail stack ----------
|
||||
section "Mail stack"
|
||||
systemctl is-active --quiet postfix@- && ok "postfix active" || bad "postfix not active"
|
||||
systemctl is-active --quiet dovecot && ok "dovecot active" || bad "dovecot not active"
|
||||
systemctl is-active --quiet rspamd && ok "rspamd active" || bad "rspamd not active"
|
||||
|
||||
SMTP_BANNER=$(printf 'QUIT\r\n' | timeout 5 nc 127.0.0.1 25 2>/dev/null | head -1)
|
||||
echo "$SMTP_BANNER" | grep -q '220.*Postfix' && ok "SMTP banner OK" || bad "SMTP banner missing/unexpected: $SMTP_BANNER"
|
||||
|
||||
IMAP_BANNER=$(printf 'a LOGOUT\r\n' | timeout 5 nc 127.0.0.1 143 2>/dev/null | head -1)
|
||||
echo "$IMAP_BANNER" | grep -qi 'dovecot ready\|^\* OK' && ok "IMAP banner OK" || bad "IMAP banner missing/unexpected: $IMAP_BANNER"
|
||||
|
||||
for port in 11332 11333 11334; do
|
||||
if sudo ss -ltn | grep -q ":$port "; then
|
||||
ok "rspamd listening on $port"
|
||||
else
|
||||
bad "rspamd NOT listening on $port"
|
||||
fi
|
||||
done
|
||||
|
||||
RSPAMD_TEST=$(printf 'Subject: sanity test\nFrom: test@example.com\nTo: jens@jayfield.org\n\nbody\n' | rspamc symbols 2>&1)
|
||||
if echo "$RSPAMD_TEST" | grep -q 'Action:'; then
|
||||
ok "rspamd genuinely scanning (rspamc symbols responded)"
|
||||
else
|
||||
bad "rspamd did not respond to a live scan test - check for a phantom 'active' status"
|
||||
fi
|
||||
|
||||
if sudo ss -ltn | grep -q ':11332 '; then
|
||||
ok "postfix's configured milter target (11332) is listening"
|
||||
else
|
||||
bad "nothing listening on postfix's milter port - mail is flowing unfiltered/unsigned"
|
||||
fi
|
||||
|
||||
# ---------- MariaDB / vmail ----------
|
||||
section "MariaDB / vmail"
|
||||
systemctl is-active --quiet mariadb && ok "mariadb active" || bad "mariadb not active"
|
||||
|
||||
DOMAINS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.domains;" 2>/dev/null)
|
||||
ACCOUNTS=$(sudo mysql -N -e "SELECT COUNT(*) FROM vmail.accounts;" 2>/dev/null)
|
||||
[ -n "$DOMAINS" ] && [ "$DOMAINS" -ge 1 ] && ok "vmail.domains has $DOMAINS row(s)" || bad "vmail.domains query failed or returned zero rows"
|
||||
[ -n "$ACCOUNTS" ] && [ "$ACCOUNTS" -ge 1 ] && ok "vmail.accounts has $ACCOUNTS row(s)" || bad "vmail.accounts query failed or returned zero rows"
|
||||
|
||||
if sudo postmap -q jayfield.org mysql:/etc/postfix/sql/domains.cf 2>/dev/null | grep -q jayfield.org; then
|
||||
ok "Postfix's MySQL domain lookup works"
|
||||
else
|
||||
bad "Postfix's MySQL domain lookup failed"
|
||||
fi
|
||||
|
||||
# ---------- fail2ban ----------
|
||||
section "fail2ban"
|
||||
systemctl is-active --quiet fail2ban && ok "fail2ban active" || bad "fail2ban not active"
|
||||
|
||||
JAIL_COUNT=$(sudo fail2ban-client status 2>/dev/null | grep -oP 'Number of jail:\s*\K[0-9]+')
|
||||
if [ -n "$JAIL_COUNT" ] && [ "$JAIL_COUNT" -ge 15 ]; then
|
||||
ok "$JAIL_COUNT jails loaded"
|
||||
else
|
||||
bad "only ${JAIL_COUNT:-0} jail(s) loaded (expected ~19)"
|
||||
fi
|
||||
|
||||
SSHD_FT=$(sudo fail2ban-client get sshd findtime 2>/dev/null)
|
||||
DOVECOT_FT=$(sudo fail2ban-client get dovecot findtime 2>/dev/null)
|
||||
[ "$SSHD_FT" = "86400" ] && ok "sshd findtime hardening intact (86400s)" || bad "sshd findtime is ${SSHD_FT:-unknown}, expected 86400"
|
||||
[ "$DOVECOT_FT" = "86400" ] && ok "dovecot findtime hardening intact (86400s)" || bad "dovecot findtime is ${DOVECOT_FT:-unknown}, expected 86400"
|
||||
|
||||
# ---------- Apache / vhosts ----------
|
||||
section "Apache / vhosts"
|
||||
systemctl is-active --quiet apache2 && ok "apache2 active" || bad "apache2 not active"
|
||||
sudo apache2ctl configtest >/dev/null 2>&1 && ok "apache2ctl configtest clean" || bad "apache2ctl configtest failed"
|
||||
|
||||
check_url() {
|
||||
local url=$1 expected=$2 code
|
||||
code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$url")
|
||||
if [ "$code" = "$expected" ]; then
|
||||
ok "$url -> $code"
|
||||
else
|
||||
bad "$url -> $code (expected $expected)"
|
||||
fi
|
||||
}
|
||||
check_url "https://jayfield.org/" 200
|
||||
check_url "https://www.jayfield.org/" 200
|
||||
check_url "https://dyndns.jayfield.org/" 401
|
||||
check_url "https://mail.jayfield.org/" 403
|
||||
check_url "https://cloud.jayfield.org/" 302
|
||||
check_url "https://portainer.jayfield.org/" 200
|
||||
check_url "https://git.jayfield.org/" 200
|
||||
|
||||
# ---------- Docker ----------
|
||||
section "Docker"
|
||||
systemctl is-active --quiet docker && ok "docker active" || bad "docker not active"
|
||||
|
||||
for c in portainer nextcloud nextcloud-redis nextcloud-db web gitea; do
|
||||
STATUS=$(sudo docker inspect -f '{{.State.Status}}' "$c" 2>/dev/null)
|
||||
if [ "$STATUS" = "running" ]; then
|
||||
ok "container $c running"
|
||||
else
|
||||
bad "container $c is '${STATUS:-missing}' (expected running)"
|
||||
fi
|
||||
done
|
||||
|
||||
GITEA_REPO_COUNT=$(curl -s --max-time 5 'https://git.jayfield.org/api/v1/repos/search?limit=1' | python3 -c 'import json,sys; print(len(json.load(sys.stdin).get("data",[])))' 2>/dev/null)
|
||||
if [ "$GITEA_REPO_COUNT" = "1" ]; then
|
||||
ok "Gitea API returns real repo data (not just a static page)"
|
||||
else
|
||||
bad "Gitea API did not return expected repo data"
|
||||
fi
|
||||
|
||||
# ---------- Summary ----------
|
||||
section "Summary"
|
||||
echo "PASS: $PASS WARN: $WARN FAIL: $FAIL"
|
||||
echo "Full log: $LOG"
|
||||
|
||||
if command -v mail >/dev/null 2>&1; then
|
||||
SUBJECT="alpha.jayfield.org sanity check: $([ "$FAIL" -gt 0 ] && echo "$FAIL FAILURE(S)" || echo "OK")"
|
||||
mail -s "$SUBJECT" jens@jayfield.org < "$LOG"
|
||||
fi
|
||||
|
||||
[ "$FAIL" -eq 0 ]
|
||||
Reference in New Issue
Block a user