Previously the pull script mounted the Unassigned Device and left it
mounted indefinitely. Now tracks whether it was already mounted before
this run and only unmounts at the end if it wasn't - never disturbs a
mount already in place for some other reason. Handled via an EXIT trap
so it fires on any exit path, not just success.
Found and fixed a real bug testing this: the script cd's into the
mounted directory and never leaves, so the unmount was failing with
"target is busy" (the shell's own cwd still being on the device) until
the cleanup function explicitly cd's out first. Verified both
directions live: already-mounted stays mounted after; not-mounted gets
mounted, used, and cleanly unmounted again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Every script/config built this session (sanity-check, attacker-check,
the Gitea backup pipeline on both ends) now lives in this repo as
reference copies, not just described in prose - previously they only
existed on the live servers.
Split into scripts/alpha/ (Ubuntu 24.04.4 LTS) and
scripts/clients/vlda-01/ (Unraid 7.3.2), each with its own README
stating the exact OS/kernel and a file-by-file map to deployed paths,
since a script written for one host's conventions doesn't just work
unchanged on the other - the vlda-01 README in particular documents
the persistence gotchas that actually broke earlier attempts (RAM-
backed root filesystem, VFAT /boot with no execute bit, Unassigned
Device auto-mount). Root README.md now links directly to these files
from each relevant section instead of only describing them.
Explicitly not included: the dedicated SSH private key vlda-01 uses to
authenticate to alpha - noted in clients/vlda-01/README.md to
regenerate rather than ever commit one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three changes to the alpha -> vlda-01 backup pipeline:
- alpha now writes a sha256sum-format checksum alongside every dump,
ACL'd for the pull account same as the dump itself.
- vlda-01 verifies each freshly-pulled dump's checksum before trusting
it enough to delete the alpha-side copy - a mismatch removes the bad
local copy instead and leaves alpha's for a retry, so neither end can
end up trusting a corrupt file or losing the only good copy. This
required loosening the giteabackup account from strictly read-only:
the bind mount is now rw and the directory ACL grants rwx (needed for
delete, a directory-level operation in POSIX), but per-file ACLs stay
read-only - verified the boundary holds (rm succeeds, put/overwrite
still fails with Permission denied).
- vlda-01 now also runs the sync once at every array startup via a
second User Scripts entry (schedule.json only supports one schedule
per script path), not just the daily 03:30 - hit the same VFAT
execute-bit issue as before along the way (a wrapper can't directly
exec a /boot-resident file, has to invoke it via bash).
Verified the full pipeline end-to-end with a genuinely fresh dump:
checksum generated, fetched, verified locally, both files deleted from
alpha only after a confirmed match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Was landing in /mnt/user/appdata/ (array storage); moved to a dedicated
external disk per request. That disk isn't guaranteed to already be
mounted when the pull script runs, so it now resolves the device by
filesystem label (not a hardcoded /dev/sdX - unassigned device paths
aren't stable across reboots) and mounts it itself via the Unassigned
Devices plugin's own tool, refusing to proceed unless a real mountpoint
actually comes up - avoids the failure mode of silently writing backups
onto the array if the disk is unplugged or fails to mount. Migrated the
two already-pulled archives to the new location and verified the
updated script end-to-end through the real scheduled-invocation path.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Implements the sketch from SYNC-PLAN.md item 2b. gitea dump (not raw
rsync, to avoid grabbing the live SQLite DB mid-write) runs nightly on
alpha via cron, into a dedicated read-only-chrooted SFTP account
(giteabackup) reusing the existing ftp_jens/ftp_alex sshd pattern.
vlda-01 pulls whatever's new via a User Scripts plugin entry (not a raw
crontab - Unraid's / and /usr are RAM-backed and don't survive reboot,
only /boot does), fetching only files it doesn't already have so a
run after days offline catches up on everything missed.
Found and fixed two real issues along the way: a default ACL on the
backup directory doesn't actually grant read access to new dump files,
since gitea dump creates them 600 and the resulting ACL mask neuters
the inherited grant - fixed by applying the ACL explicitly per file.
And the restricted account's ForceCommand internal-sftp blocks a real
rsync invocation, so the transport is SFTP get, not rsync.
Verified end-to-end through the actual scheduled-invocation path on
both ends, including a second run correctly skipping an already-pulled
file instead of re-fetching it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Now that alpha is the canonical live Gitea, vlda-01's stopped copy from
the migration just goes stale without something feeding it. Sketched
(not built) a design: gitea dump (not raw rsync, to avoid grabbing the
live SQLite DB mid-write) on a nightly cron on alpha, pulled by a
vlda-01-initiated cron job whenever it happens to be online - same
network-direction constraint as the rest of this plan, since alpha has
no route into the home LAN. Point-in-time backup, not a hot standby;
restore is a manual, deliberate step. Confirmed gitea dump is available
on this install and checked disk headroom on both ends before sizing
the retention window.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
jayfield.org and its public subdomains all point at this host
(89.58.8.149) and are what this whole doc set is about. home.jayfield.org
is a separate, deliberately private namespace for the home intranet
(vlda-01 etc.) - confirmed it's genuinely absent from this host's
authoritative zone, consistent with being kept off public DNS by
design. Also fixed SYNC-PLAN.md's stale top-of-file status line, which
still said "nothing implemented yet" after the Gitea migration.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Renamed via Gitea's web admin panel (the code path that correctly
moves on-disk repo storage and sets up the old-path redirect together)
rather than a direct DB/API edit. Password rotated at the same time.
Updated this repo's local origin remote and stored git credentials to
match; verified new credentials authenticate and both the new and
redirected old repo paths return 200.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README.md documented the running service, but the from-scratch rebuild
guide was missing it entirely - added a section matching the existing
Portainer/Nextcloud format, including the uid-collision gotcha (host's
vmail user already owns 1000:1000) and the X-Forwarded-Proto fix
needed for correct clone URLs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds git.jayfield.org to the vhost checks, gitea to the container
list, and a genuine API check (not just an HTTP 200) confirming Gitea
is actually serving repo data. Verified with a live run: 47 pass, 0
fail.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
First item off SYNC-PLAN.md's build order. Full migration, not sync:
stopped vlda-01's Gitea container, copied its entire data volume
(SQLite DB, repos, LFS, SSH host keys - ~4GB) directly via a piped ssh
tar stream, brought it up on alpha as a permanent container on the
same web/cloud/portainer pattern (127.0.0.1-only, Apache reverse
proxy, certbot). Remapped file ownership from the source's uid 1000
(which collides with vmail on alpha) to a dedicated 1010:1010. Fixed
Gitea generating http:// clone URLs by adding X-Forwarded-Proto,
same fix cloud.jayfield.org already needed. Verified end-to-end: all
repos present via the API, a real git clone reproduces identical
history, and this repo's own origin now points at git.jayfield.org
with working push access. vlda-01's copy left stopped as a cold
backup, not decommissioned.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
root@jayfield.org didn't resolve to anything (missing from both
/etc/aliases and vmail.aliases) - added it as a new alias to
jens@jayfield.org, matching the existing postmaster/hostmaster/
webmaster/wlan pattern, so the weekly report doesn't just bounce.
Also added per-run dated report files under
/var/log/attacker-check/reports/ alongside the existing CSVs, for a
plain-text historical archive browsable by date.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Parses fail2ban.log for the past 7 days into two append-only CSVs
(per-IP and per-jail rollups) for longitudinal analysis, plus a
human-readable summary emailed weekly. Two detection heuristics run on
top of the raw data: distributed single-shot-per-IP scans across a /24
(shape of the apache-noscript campaign), and regular-timing evasion on
repeat hits (shape of the sshd/dovecot low-and-slow campaigns) - both
modeled on real campaigns found earlier this engagement. A live test
run against real logs correctly re-identified the same dovecot
campaign IPs found manually earlier, with matching interval timing.
Runs weekly via cron, Sunday 06:00, clear of logrotate's daily timer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Codifies every manual verification done throughout today's 22.04->24.04
upgrade work into a single script: package consistency, SSH hardening,
DNS/DNSSEC, full mail stack (including a live rspamd scan test - would
have caught today's phantom-active rspamd immediately), MariaDB/vmail,
fail2ban jails and custom hardening, all public vhosts, and Docker
containers. Runs automatically once per boot via a systemd oneshot
unit, since every significant change so far has ended in a reboot
anyway; also safe to run manually. Emails a pass/warn/fail summary to
jens@jayfield.org via the local mail stack. Tested live: manual run
and an actual reboot both produced identical clean results (44 pass,
1 warn - pending routine package updates, 0 fail).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Ran do-release-upgrade detached and non-interactive after the specific
failure mode that killed the 2026-07-18 attempt (an unattended
interactive prompt), validated with a full test reboot first. Caught
and fixed 12 more leftover packages from the original incomplete
upgrade that an earlier cleanup pass had missed. Completed successfully
into Ubuntu 24.04.4 / kernel 6.8.0-136.
Two real post-upgrade problems found and fixed same day: fail2ban
failed to start due to a genuine gap in the noble package (a
[DEFAULT] value referenced but never defined), fixed with an owned
jail.d drop-in; rspamd was fully removed (not just held back like
PHP), its service masked as a safety measure, its apt source migrated
to the new deb822 format and disabled - reinstalled from a re-enabled
noble-pointed repo and verified it's genuinely scanning again, not
just a phantom "active" service.
Full verification pass confirms mail, DNS/DNSSEC, dyndns, Docker,
and SSH hardening all intact. MariaDB auto-upgraded again to noble's
native 10.11.14 as part of this run; vmail data verified unchanged.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Backed up all databases (mysqldump) and /etc/mysql before starting;
dry-run confirmed the exact expected package transition. Ran cleanly -
postinst auto-detected and upgraded the old datadir's system tables.
Verified end-to-end: vmail schema intact, doveadm lists both real
mailboxes correctly, and Postfix's MySQL lookups resolve against the
upgraded database. No Galera clustering was actually active, so the
galera-3->4 swap was a no-op in practice.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found the underlying cause of the flaky third-party repos: a
do-release-upgrade from 2026-07-18 stalled mid-way at an interactive
dpkg prompt and was never resumed, leaving mariadb-server-10.3,
docker, rspamd, and php7.4/8.0 all on stale focal-era state. Fixed the
three third-party apt sources (docker, rspamd, ondrej PHP PPA) to
point at jammy - only actually worked once the DNSSEC bug was fixed,
since that was silently blocking repo resolution too. Reviewed
rspamd's 3.11->4.1.3 changelog against the live config before
upgrading (no breaking changes apply here), then ran apt upgrade
(dry-run confirmed zero removals, no mariadb involvement) covering
docker-ce, PHP, rspamd, and 74 other packages. Restarted all
needrestart-flagged services carefully, verifying each (ssh first and
deliberately given it's the only remote access, then DNS/mail/security
services, then containers). Everything confirmed healthy.
MariaDB 10.3->10.6 migration and the actual 22.04->24.04 upgrade are
still pending as deliberately separate next steps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found while investigating flaky third-party apt repo resolution
(download.docker.com, ppa.launchpad.net): BIND's own recursive
resolver was returning SERVFAIL for essentially any DNSSEC-signed
external domain, root-caused via dnssec.log to a stale/inconsistent
root trust-anchor state, likely dating to when full local recursion
was first enabled on 2026-07-19. Reset managed-keys.bind to force a
clean RFC 5011 re-bootstrap; verified previously-failing domains now
resolve correctly and the authoritative jayfield.org zone is
unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Account has no hostname or key left to act on since the auth-binding
fix, but its password predated the whole migration - rotated it via
htpasswd -b. Verified new password works, old one is rejected, and the
account is still fully inert for DNS updates either way.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Second real WAN IP change on vpn produced another clean authenticated
update with no test collision this time. Both uschi and vpn have now
each been confirmed live across a real IP change on their dedicated
per-host login - nothing further to track on this item.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README.md's credential-audit table and command-injection writeup still
described the per-user auth binding gap as open; it's been closed since
2026-07-26. SETUP.md's TSIG section still taught the old single-shared-
key pattern for a fresh install - replaced with the per-host key +
update-policy pattern actually running in production now, plus a
pointer to the phased-rollout approach documented in TODO.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real FritzBox update authenticated as uschi landed after a WAN IP
change, confirmed via dig. Documented the earlier test-value collision
where a verification write clobbered the device's first post-migration
update, resolved by its next natural IP change - no lasting impact,
but noted as a lesson for future verification steps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
uschi's device should be using its own dedicated login now, and with
jens/kack/test's own hostnames already removed, nothing needed the
fallback anymore. Removed the update-policy fallback grant, retired
the original shared TSIG key (in production since March 2022), and
deleted the now-dead legacy code path from nsupdate.php and its ipv4/
ipv6 variants. Per-host auth binding for dyndns is now unconditional
with no exceptions. Verified jens can no longer update any host, while
uschi/vpn continue to update themselves normally.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same treatment as kack/test: no device had actually updated jens's own
record since Feb 2024. Removed the DNS record and its dedicated
update-policy grant/key, but deliberately kept the jens login and the
legacy shared-key fallback grant in place, since uschi's device still
depends on it. Re-verified the jens->uschi fallback path still works
after the change.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Neither had shown real recent activity (kack not since Feb 2024, test
with no traffic in the per-vhost log's short history), so rather than
give them dedicated per-host logins like vpn/uschi, removed their DNS
records, TSIG keys, update-policy grants, and .htpasswd accounts
outright. Backed up before removal; jens/uschi/vpn unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
FritzBox reconfigured with its dedicated per-host login; a real WAN IP
change after recycling the connection produced a genuine authenticated
update, confirmed in the access log and via dig. First device fully
migrated off the legacy shared-key fallback.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the gap where any dyndns credential could update any of the
zone's 5 hostnames. Confirmed live that a FritzBox was already exposed
to this (authenticating as the shared 'jens' account while updating
'uschi'), so implemented a phased rollout: new per-host TSIG keys and
.htpasswd accounts enforce the binding at the BIND layer itself, with
a logged legacy fallback for the pre-existing shared account only,
so unmigrated devices keep working until reconfigured. Also fixed a
bug where nic/update.php reported "good" even on a rejected update.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both were left as "revisit if it escalates" notes from earlier audits.
Checked live logs on alpha via SSH (port 10022): postfix-sasl remains
scattered low-volume noise, not a coordinated low-and-slow campaign like
the dovecot spray; the original three /24s behind the apache-noscript
distributed scan are no longer active, current activity is normal
per-IP jail behavior. No fixes needed for either; also reconfirmed the
dyndns per-user auth binding gap is still present and unfixed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Manual dig confirms 89.58.8.149 still delisted as of 2026-07-26, matching
the automated 1h propagation check. Not conclusive yet - Barracuda's
temporary 48h reputation bump is still in effect; the scheduled 48h
routine on 2026-07-27 is the real test of whether removal stuck.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Mail IP 89.58.8.149 got flagged on Barracuda's BRBL; local checks found
no sign of actual abuse (empty queue, minimal legit outbound, no
compromised SASL accounts), so removal was requested. Tracking the
confirmation number and scheduled follow-up checks here until resolved.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same evasion pattern already fixed for sshd recurred against dovecot: a
coordinated IMAP password spray paced just outside the 30-minute findtime
window ran unbanned for days. Widened dovecot's findtime the same way,
and updated SETUP.md so future provisioning applies this to every
credential jail up front instead of waiting for it to reappear.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real root cause turned out to differ from the original TODO: there was
no broken global CustomLog, mod_vhost_combined was already logging
every vhost without its own CustomLog into other_vhosts_access.log.
Added a dedicated CustomLog per real site anyway for easier incident
review, verified live with curl against all 7 domains after reload.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
README's public-access table and "previously resolved" summary now
reflect the dyndns injection fix and the TSIG-permission regression it
uncovered. Adds two open TODO items found during that audit: the dyndns
app's lack of real per-user auth binding, and the fact that access
logging for every real vhost on this host is currently a no-op.
nsupdate.php/nsupdate_ipv4.php/nsupdate_ipv6.php rewritten to build the
nsupdate command as a string written to proc_open()'s stdin, using the
array form of the command so no shell is ever invoked -- no value can be
interpreted as shell syntax regardless of content. Added hostname/IP
validation as defense in depth. Originals backed up.
Also fixes a regression discovered while testing: www-data never got
read access to the TSIG key when it was tightened to 640 root:bind
(2026-07-19), so legitimate dyndns updates were silently failing
(REFUSED) independent of the vulnerability. Fixed with a POSIX ACL
scoped to just that key file rather than group membership, since the
bind group also owns rndc.key (full remote BIND control).
Verified end-to-end via the real PHP functions: injection payloads
rejected with no side effects, invalid IP rejected, legitimate update
succeeds and is visible in the zone, unauthenticated requests still 401.
nsupdate.php/nsupdate_ipv4.php/nsupdate_ipv6.php pass unsanitized $_GET
input straight into exec()'d shell commands. Gated by Apache Basic Auth
on the dyndns vhost, so not anonymously exploitable, but a real OS
command injection reachable by anyone with valid (or leaked) dyndns
credentials. Documents the finding and the fix approach; not yet applied.
A stray /var/www/html/.htaccess (dated 2025-02-18, predating everything
else in TODO.md) was silently requiring HTTP Basic Auth for jayfield.org
and www.jayfield.org's shared DocumentRoot, 401ing every real visitor to
the intentionally-public static site. Removed (backed up, not deleted).
Also adds a README table auditing which public hostnames do/don't
require credentials, prompted by checking this.
Moved Portainer's HTTPS UI behind an Apache reverse proxy on its own
subdomain (portainer.jayfield.org) instead of publishing it directly on
:9443 with a self-signed cert, which HSTS's includeSubDomains policy made
unreachable in Firefox with no click-through exception. Same pattern as
the existing web.jayfield.org/cloud.jayfield.org proxied subdomains: DNS
A record, Apache vhost proxying to 127.0.0.1:9443 over HTTPS, real
Let's Encrypt cert via certbot, standard HSTS header. Portainer container
recreated to bind 9443 to loopback only; verified end-to-end (cert chain,
HSTS header, redirect, and that :9443 is no longer reachable externally).
Firefox refuses https://alpha.jayfield.org:9443 with
MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT and no click-through: the
includeSubDomains HSTS policy on the real alpha.jayfield.org vhost pins
the hostname to trusted-cert-only HTTPS on every port, but Portainer
still serves its stock self-signed cert directly on 9443. Records the
root cause and the planned reverse-proxy fix; no server changes made yet.
Records the host's service inventory, setup runbook, mail account list,
sync plan, package diff vs. clean install, and hardening TODOs, including
today's SSH hardening (key-only, no root login) and the fail2ban sshd
findtime fix for a low-and-slow brute-force evasion pattern.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdTyvEJkfNLDVAWt8WQ6X9
jensandClaude Sonnet 5