Three changes to the alpha -> vlda-01 backup pipeline:
- alpha now writes a sha256sum-format checksum alongside every dump,
ACL'd for the pull account same as the dump itself.
- vlda-01 verifies each freshly-pulled dump's checksum before trusting
it enough to delete the alpha-side copy - a mismatch removes the bad
local copy instead and leaves alpha's for a retry, so neither end can
end up trusting a corrupt file or losing the only good copy. This
required loosening the giteabackup account from strictly read-only:
the bind mount is now rw and the directory ACL grants rwx (needed for
delete, a directory-level operation in POSIX), but per-file ACLs stay
read-only - verified the boundary holds (rm succeeds, put/overwrite
still fails with Permission denied).
- vlda-01 now also runs the sync once at every array startup via a
second User Scripts entry (schedule.json only supports one schedule
per script path), not just the daily 03:30 - hit the same VFAT
execute-bit issue as before along the way (a wrapper can't directly
exec a /boot-resident file, has to invoke it via bash).
Verified the full pipeline end-to-end with a genuinely fresh dump:
checksum generated, fetched, verified locally, both files deleted from
alpha only after a confirmed match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Was landing in /mnt/user/appdata/ (array storage); moved to a dedicated
external disk per request. That disk isn't guaranteed to already be
mounted when the pull script runs, so it now resolves the device by
filesystem label (not a hardcoded /dev/sdX - unassigned device paths
aren't stable across reboots) and mounts it itself via the Unassigned
Devices plugin's own tool, refusing to proceed unless a real mountpoint
actually comes up - avoids the failure mode of silently writing backups
onto the array if the disk is unplugged or fails to mount. Migrated the
two already-pulled archives to the new location and verified the
updated script end-to-end through the real scheduled-invocation path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements the sketch from SYNC-PLAN.md item 2b. gitea dump (not raw
rsync, to avoid grabbing the live SQLite DB mid-write) runs nightly on
alpha via cron, into a dedicated read-only-chrooted SFTP account
(giteabackup) reusing the existing ftp_jens/ftp_alex sshd pattern.
vlda-01 pulls whatever's new via a User Scripts plugin entry (not a raw
crontab - Unraid's / and /usr are RAM-backed and don't survive reboot,
only /boot does), fetching only files it doesn't already have so a
run after days offline catches up on everything missed.
Found and fixed two real issues along the way: a default ACL on the
backup directory doesn't actually grant read access to new dump files,
since gitea dump creates them 600 and the resulting ACL mask neuters
the inherited grant - fixed by applying the ACL explicitly per file.
And the restricted account's ForceCommand internal-sftp blocks a real
rsync invocation, so the transport is SFTP get, not rsync.
Verified end-to-end through the actual scheduled-invocation path on
both ends, including a second run correctly skipping an already-pulled
file instead of re-fetching it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jayfield.org and its public subdomains all point at this host
(89.58.8.149) and are what this whole doc set is about. home.jayfield.org
is a separate, deliberately private namespace for the home intranet
(vlda-01 etc.) - confirmed it's genuinely absent from this host's
authoritative zone, consistent with being kept off public DNS by
design. Also fixed SYNC-PLAN.md's stale top-of-file status line, which
still said "nothing implemented yet" after the Gitea migration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Renamed via Gitea's web admin panel (the code path that correctly
moves on-disk repo storage and sets up the old-path redirect together)
rather than a direct DB/API edit. Password rotated at the same time.
Updated this repo's local origin remote and stored git credentials to
match; verified new credentials authenticate and both the new and
redirected old repo paths return 200.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds git.jayfield.org to the vhost checks, gitea to the container
list, and a genuine API check (not just an HTTP 200) confirming Gitea
is actually serving repo data. Verified with a live run: 47 pass, 0
fail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First item off SYNC-PLAN.md's build order. Full migration, not sync:
stopped vlda-01's Gitea container, copied its entire data volume
(SQLite DB, repos, LFS, SSH host keys - ~4GB) directly via a piped ssh
tar stream, brought it up on alpha as a permanent container on the
same web/cloud/portainer pattern (127.0.0.1-only, Apache reverse
proxy, certbot). Remapped file ownership from the source's uid 1000
(which collides with vmail on alpha) to a dedicated 1010:1010. Fixed
Gitea generating http:// clone URLs by adding X-Forwarded-Proto,
same fix cloud.jayfield.org already needed. Verified end-to-end: all
repos present via the API, a real git clone reproduces identical
history, and this repo's own origin now points at git.jayfield.org
with working push access. vlda-01's copy left stopped as a cold
backup, not decommissioned.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
root@jayfield.org didn't resolve to anything (missing from both
/etc/aliases and vmail.aliases) - added it as a new alias to
jens@jayfield.org, matching the existing postmaster/hostmaster/
webmaster/wlan pattern, so the weekly report doesn't just bounce.
Also added per-run dated report files under
/var/log/attacker-check/reports/ alongside the existing CSVs, for a
plain-text historical archive browsable by date.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Parses fail2ban.log for the past 7 days into two append-only CSVs
(per-IP and per-jail rollups) for longitudinal analysis, plus a
human-readable summary emailed weekly. Two detection heuristics run on
top of the raw data: distributed single-shot-per-IP scans across a /24
(shape of the apache-noscript campaign), and regular-timing evasion on
repeat hits (shape of the sshd/dovecot low-and-slow campaigns) - both
modeled on real campaigns found earlier this engagement. A live test
run against real logs correctly re-identified the same dovecot
campaign IPs found manually earlier, with matching interval timing.
Runs weekly via cron, Sunday 06:00, clear of logrotate's daily timer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codifies every manual verification done throughout today's 22.04->24.04
upgrade work into a single script: package consistency, SSH hardening,
DNS/DNSSEC, full mail stack (including a live rspamd scan test - would
have caught today's phantom-active rspamd immediately), MariaDB/vmail,
fail2ban jails and custom hardening, all public vhosts, and Docker
containers. Runs automatically once per boot via a systemd oneshot
unit, since every significant change so far has ended in a reboot
anyway; also safe to run manually. Emails a pass/warn/fail summary to
jens@jayfield.org via the local mail stack. Tested live: manual run
and an actual reboot both produced identical clean results (44 pass,
1 warn - pending routine package updates, 0 fail).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran do-release-upgrade detached and non-interactive after the specific
failure mode that killed the 2026-07-18 attempt (an unattended
interactive prompt), validated with a full test reboot first. Caught
and fixed 12 more leftover packages from the original incomplete
upgrade that an earlier cleanup pass had missed. Completed successfully
into Ubuntu 24.04.4 / kernel 6.8.0-136.
Two real post-upgrade problems found and fixed same day: fail2ban
failed to start due to a genuine gap in the noble package (a
[DEFAULT] value referenced but never defined), fixed with an owned
jail.d drop-in; rspamd was fully removed (not just held back like
PHP), its service masked as a safety measure, its apt source migrated
to the new deb822 format and disabled - reinstalled from a re-enabled
noble-pointed repo and verified it's genuinely scanning again, not
just a phantom "active" service.
Full verification pass confirms mail, DNS/DNSSEC, dyndns, Docker,
and SSH hardening all intact. MariaDB auto-upgraded again to noble's
native 10.11.14 as part of this run; vmail data verified unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README.md's credential-audit table and command-injection writeup still
described the per-user auth binding gap as open; it's been closed since
2026-07-26. SETUP.md's TSIG section still taught the old single-shared-
key pattern for a fresh install - replaced with the per-host key +
update-policy pattern actually running in production now, plus a
pointer to the phased-rollout approach documented in TODO.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were left as "revisit if it escalates" notes from earlier audits.
Checked live logs on alpha via SSH (port 10022): postfix-sasl remains
scattered low-volume noise, not a coordinated low-and-slow campaign like
the dovecot spray; the original three /24s behind the apache-noscript
distributed scan are no longer active, current activity is normal
per-IP jail behavior. No fixes needed for either; also reconfirmed the
dyndns per-user auth binding gap is still present and unfixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real root cause turned out to differ from the original TODO: there was
no broken global CustomLog, mod_vhost_combined was already logging
every vhost without its own CustomLog into other_vhosts_access.log.
Added a dedicated CustomLog per real site anyway for easier incident
review, verified live with curl against all 7 domains after reload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README's public-access table and "previously resolved" summary now
reflect the dyndns injection fix and the TSIG-permission regression it
uncovered. Adds two open TODO items found during that audit: the dyndns
app's lack of real per-user auth binding, and the fact that access
logging for every real vhost on this host is currently a no-op.
A stray /var/www/html/.htaccess (dated 2025-02-18, predating everything
else in TODO.md) was silently requiring HTTP Basic Auth for jayfield.org
and www.jayfield.org's shared DocumentRoot, 401ing every real visitor to
the intentionally-public static site. Removed (backed up, not deleted).
Also adds a README table auditing which public hostnames do/don't
require credentials, prompted by checking this.
Moved Portainer's HTTPS UI behind an Apache reverse proxy on its own
subdomain (portainer.jayfield.org) instead of publishing it directly on
:9443 with a self-signed cert, which HSTS's includeSubDomains policy made
unreachable in Firefox with no click-through exception. Same pattern as
the existing web.jayfield.org/cloud.jayfield.org proxied subdomains: DNS
A record, Apache vhost proxying to 127.0.0.1:9443 over HTTPS, real
Let's Encrypt cert via certbot, standard HSTS header. Portainer container
recreated to bind 9443 to loopback only; verified end-to-end (cert chain,
HSTS header, redirect, and that :9443 is no longer reachable externally).
Firefox refuses https://alpha.jayfield.org:9443 with
MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT and no click-through: the
includeSubDomains HSTS policy on the real alpha.jayfield.org vhost pins
the hostname to trusted-cert-only HTTPS on every port, but Portainer
still serves its stock self-signed cert directly on 9443. Records the
root cause and the planned reverse-proxy fix; no server changes made yet.
Records the host's service inventory, setup runbook, mail account list,
sync plan, package diff vs. clean install, and hardening TODOs, including
today's SSH hardening (key-only, no root login) and the fail2ban sshd
findtime fix for a low-and-slow brute-force evasion pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdTyvEJkfNLDVAWt8WQ6X9
jensandClaude Sonnet 5