Previously the pull script mounted the Unassigned Device and left it
mounted indefinitely. Now tracks whether it was already mounted before
this run and only unmounts at the end if it wasn't - never disturbs a
mount already in place for some other reason. Handled via an EXIT trap
so it fires on any exit path, not just success.
Found and fixed a real bug testing this: the script cd's into the
mounted directory and never leaves, so the unmount was failing with
"target is busy" (the shell's own cwd still being on the device) until
the cleanup function explicitly cd's out first. Verified both
directions live: already-mounted stays mounted after; not-mounted gets
mounted, used, and cleanly unmounted again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every script/config built this session (sanity-check, attacker-check,
the Gitea backup pipeline on both ends) now lives in this repo as
reference copies, not just described in prose - previously they only
existed on the live servers.
Split into scripts/alpha/ (Ubuntu 24.04.4 LTS) and
scripts/clients/vlda-01/ (Unraid 7.3.2), each with its own README
stating the exact OS/kernel and a file-by-file map to deployed paths,
since a script written for one host's conventions doesn't just work
unchanged on the other - the vlda-01 README in particular documents
the persistence gotchas that actually broke earlier attempts (RAM-
backed root filesystem, VFAT /boot with no execute bit, Unassigned
Device auto-mount). Root README.md now links directly to these files
from each relevant section instead of only describing them.
Explicitly not included: the dedicated SSH private key vlda-01 uses to
authenticate to alpha - noted in clients/vlda-01/README.md to
regenerate rather than ever commit one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three changes to the alpha -> vlda-01 backup pipeline:
- alpha now writes a sha256sum-format checksum alongside every dump,
ACL'd for the pull account same as the dump itself.
- vlda-01 verifies each freshly-pulled dump's checksum before trusting
it enough to delete the alpha-side copy - a mismatch removes the bad
local copy instead and leaves alpha's for a retry, so neither end can
end up trusting a corrupt file or losing the only good copy. This
required loosening the giteabackup account from strictly read-only:
the bind mount is now rw and the directory ACL grants rwx (needed for
delete, a directory-level operation in POSIX), but per-file ACLs stay
read-only - verified the boundary holds (rm succeeds, put/overwrite
still fails with Permission denied).
- vlda-01 now also runs the sync once at every array startup via a
second User Scripts entry (schedule.json only supports one schedule
per script path), not just the daily 03:30 - hit the same VFAT
execute-bit issue as before along the way (a wrapper can't directly
exec a /boot-resident file, has to invoke it via bash).
Verified the full pipeline end-to-end with a genuinely fresh dump:
checksum generated, fetched, verified locally, both files deleted from
alpha only after a confirmed match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Was landing in /mnt/user/appdata/ (array storage); moved to a dedicated
external disk per request. That disk isn't guaranteed to already be
mounted when the pull script runs, so it now resolves the device by
filesystem label (not a hardcoded /dev/sdX - unassigned device paths
aren't stable across reboots) and mounts it itself via the Unassigned
Devices plugin's own tool, refusing to proceed unless a real mountpoint
actually comes up - avoids the failure mode of silently writing backups
onto the array if the disk is unplugged or fails to mount. Migrated the
two already-pulled archives to the new location and verified the
updated script end-to-end through the real scheduled-invocation path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements the sketch from SYNC-PLAN.md item 2b. gitea dump (not raw
rsync, to avoid grabbing the live SQLite DB mid-write) runs nightly on
alpha via cron, into a dedicated read-only-chrooted SFTP account
(giteabackup) reusing the existing ftp_jens/ftp_alex sshd pattern.
vlda-01 pulls whatever's new via a User Scripts plugin entry (not a raw
crontab - Unraid's / and /usr are RAM-backed and don't survive reboot,
only /boot does), fetching only files it doesn't already have so a
run after days offline catches up on everything missed.
Found and fixed two real issues along the way: a default ACL on the
backup directory doesn't actually grant read access to new dump files,
since gitea dump creates them 600 and the resulting ACL mask neuters
the inherited grant - fixed by applying the ACL explicitly per file.
And the restricted account's ForceCommand internal-sftp blocks a real
rsync invocation, so the transport is SFTP get, not rsync.
Verified end-to-end through the actual scheduled-invocation path on
both ends, including a second run correctly skipping an already-pulled
file instead of re-fetching it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jayfield.org and its public subdomains all point at this host
(89.58.8.149) and are what this whole doc set is about. home.jayfield.org
is a separate, deliberately private namespace for the home intranet
(vlda-01 etc.) - confirmed it's genuinely absent from this host's
authoritative zone, consistent with being kept off public DNS by
design. Also fixed SYNC-PLAN.md's stale top-of-file status line, which
still said "nothing implemented yet" after the Gitea migration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Renamed via Gitea's web admin panel (the code path that correctly
moves on-disk repo storage and sets up the old-path redirect together)
rather than a direct DB/API edit. Password rotated at the same time.
Updated this repo's local origin remote and stored git credentials to
match; verified new credentials authenticate and both the new and
redirected old repo paths return 200.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds git.jayfield.org to the vhost checks, gitea to the container
list, and a genuine API check (not just an HTTP 200) confirming Gitea
is actually serving repo data. Verified with a live run: 47 pass, 0
fail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First item off SYNC-PLAN.md's build order. Full migration, not sync:
stopped vlda-01's Gitea container, copied its entire data volume
(SQLite DB, repos, LFS, SSH host keys - ~4GB) directly via a piped ssh
tar stream, brought it up on alpha as a permanent container on the
same web/cloud/portainer pattern (127.0.0.1-only, Apache reverse
proxy, certbot). Remapped file ownership from the source's uid 1000
(which collides with vmail on alpha) to a dedicated 1010:1010. Fixed
Gitea generating http:// clone URLs by adding X-Forwarded-Proto,
same fix cloud.jayfield.org already needed. Verified end-to-end: all
repos present via the API, a real git clone reproduces identical
history, and this repo's own origin now points at git.jayfield.org
with working push access. vlda-01's copy left stopped as a cold
backup, not decommissioned.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
root@jayfield.org didn't resolve to anything (missing from both
/etc/aliases and vmail.aliases) - added it as a new alias to
jens@jayfield.org, matching the existing postmaster/hostmaster/
webmaster/wlan pattern, so the weekly report doesn't just bounce.
Also added per-run dated report files under
/var/log/attacker-check/reports/ alongside the existing CSVs, for a
plain-text historical archive browsable by date.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Parses fail2ban.log for the past 7 days into two append-only CSVs
(per-IP and per-jail rollups) for longitudinal analysis, plus a
human-readable summary emailed weekly. Two detection heuristics run on
top of the raw data: distributed single-shot-per-IP scans across a /24
(shape of the apache-noscript campaign), and regular-timing evasion on
repeat hits (shape of the sshd/dovecot low-and-slow campaigns) - both
modeled on real campaigns found earlier this engagement. A live test
run against real logs correctly re-identified the same dovecot
campaign IPs found manually earlier, with matching interval timing.
Runs weekly via cron, Sunday 06:00, clear of logrotate's daily timer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codifies every manual verification done throughout today's 22.04->24.04
upgrade work into a single script: package consistency, SSH hardening,
DNS/DNSSEC, full mail stack (including a live rspamd scan test - would
have caught today's phantom-active rspamd immediately), MariaDB/vmail,
fail2ban jails and custom hardening, all public vhosts, and Docker
containers. Runs automatically once per boot via a systemd oneshot
unit, since every significant change so far has ended in a reboot
anyway; also safe to run manually. Emails a pass/warn/fail summary to
jens@jayfield.org via the local mail stack. Tested live: manual run
and an actual reboot both produced identical clean results (44 pass,
1 warn - pending routine package updates, 0 fail).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran do-release-upgrade detached and non-interactive after the specific
failure mode that killed the 2026-07-18 attempt (an unattended
interactive prompt), validated with a full test reboot first. Caught
and fixed 12 more leftover packages from the original incomplete
upgrade that an earlier cleanup pass had missed. Completed successfully
into Ubuntu 24.04.4 / kernel 6.8.0-136.
Two real post-upgrade problems found and fixed same day: fail2ban
failed to start due to a genuine gap in the noble package (a
[DEFAULT] value referenced but never defined), fixed with an owned
jail.d drop-in; rspamd was fully removed (not just held back like
PHP), its service masked as a safety measure, its apt source migrated
to the new deb822 format and disabled - reinstalled from a re-enabled
noble-pointed repo and verified it's genuinely scanning again, not
just a phantom "active" service.
Full verification pass confirms mail, DNS/DNSSEC, dyndns, Docker,
and SSH hardening all intact. MariaDB auto-upgraded again to noble's
native 10.11.14 as part of this run; vmail data verified unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README.md's credential-audit table and command-injection writeup still
described the per-user auth binding gap as open; it's been closed since
2026-07-26. SETUP.md's TSIG section still taught the old single-shared-
key pattern for a fresh install - replaced with the per-host key +
update-policy pattern actually running in production now, plus a
pointer to the phased-rollout approach documented in TODO.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were left as "revisit if it escalates" notes from earlier audits.
Checked live logs on alpha via SSH (port 10022): postfix-sasl remains
scattered low-volume noise, not a coordinated low-and-slow campaign like
the dovecot spray; the original three /24s behind the apache-noscript
distributed scan are no longer active, current activity is normal
per-IP jail behavior. No fixes needed for either; also reconfirmed the
dyndns per-user auth binding gap is still present and unfixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real root cause turned out to differ from the original TODO: there was
no broken global CustomLog, mod_vhost_combined was already logging
every vhost without its own CustomLog into other_vhosts_access.log.
Added a dedicated CustomLog per real site anyway for easier incident
review, verified live with curl against all 7 domains after reload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README's public-access table and "previously resolved" summary now
reflect the dyndns injection fix and the TSIG-permission regression it
uncovered. Adds two open TODO items found during that audit: the dyndns
app's lack of real per-user auth binding, and the fact that access
logging for every real vhost on this host is currently a no-op.
A stray /var/www/html/.htaccess (dated 2025-02-18, predating everything
else in TODO.md) was silently requiring HTTP Basic Auth for jayfield.org
and www.jayfield.org's shared DocumentRoot, 401ing every real visitor to
the intentionally-public static site. Removed (backed up, not deleted).
Also adds a README table auditing which public hostnames do/don't
require credentials, prompted by checking this.
Moved Portainer's HTTPS UI behind an Apache reverse proxy on its own
subdomain (portainer.jayfield.org) instead of publishing it directly on
:9443 with a self-signed cert, which HSTS's includeSubDomains policy made
unreachable in Firefox with no click-through exception. Same pattern as
the existing web.jayfield.org/cloud.jayfield.org proxied subdomains: DNS
A record, Apache vhost proxying to 127.0.0.1:9443 over HTTPS, real
Let's Encrypt cert via certbot, standard HSTS header. Portainer container
recreated to bind 9443 to loopback only; verified end-to-end (cert chain,
HSTS header, redirect, and that :9443 is no longer reachable externally).
Firefox refuses https://alpha.jayfield.org:9443 with
MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT and no click-through: the
includeSubDomains HSTS policy on the real alpha.jayfield.org vhost pins
the hostname to trusted-cert-only HTTPS on every port, but Portainer
still serves its stock self-signed cert directly on 9443. Records the
root cause and the planned reverse-proxy fix; no server changes made yet.
Records the host's service inventory, setup runbook, mail account list,
sync plan, package diff vs. clean install, and hardening TODOs, including
today's SSH hardening (key-only, no root login) and the fail2ban sshd
findtime fix for a low-and-slow brute-force evasion pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdTyvEJkfNLDVAWt8WQ6X9
jensandClaude Sonnet 5