Check in scripts/configs, organized and documented by host OS
Every script/config built this session (sanity-check, attacker-check, the Gitea backup pipeline on both ends) now lives in this repo as reference copies, not just described in prose - previously they only existed on the live servers. Split into scripts/alpha/ (Ubuntu 24.04.4 LTS) and scripts/clients/vlda-01/ (Unraid 7.3.2), each with its own README stating the exact OS/kernel and a file-by-file map to deployed paths, since a script written for one host's conventions doesn't just work unchanged on the other - the vlda-01 README in particular documents the persistence gotchas that actually broke earlier attempts (RAM- backed root filesystem, VFAT /boot with no execute bit, Unassigned Device auto-mount). Root README.md now links directly to these files from each relevant section instead of only describing them. Explicitly not included: the dedicated SSH private key vlda-01 uses to authenticate to alpha - noted in clients/vlda-01/README.md to regenerate rather than ever commit one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 5
@@ -0,0 +1,14 @@
|
||||
# Client hosts
|
||||
|
||||
Scripts and configs that run on other hosts this project administers —
|
||||
not on `alpha.jayfield.org` itself — one subdirectory per host, named
|
||||
after it. Each subdirectory's own `README.md` states that host's exact
|
||||
OS/version, since a script written for one host's conventions generally
|
||||
will not just work unchanged on another (see `../README.md` for why this
|
||||
split exists at all).
|
||||
|
||||
## Hosts
|
||||
|
||||
- `vlda-01/` — the home intranet server referenced throughout
|
||||
`SYNC-PLAN.md` (Unraid). Currently holds the pull side of the
|
||||
`alpha` → `vlda-01` Gitea backup.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Scripts running on `vlda-01`
|
||||
|
||||
**OS**: Unraid `7.3.2`, kernel `6.18.38-Unraid`. **Very different
|
||||
persistence model from Ubuntu/`alpha`** — this tripped up the first version
|
||||
of every script here, so it's worth stating plainly before anything else:
|
||||
|
||||
- `/` and `/usr` are RAM-backed (`rootfs` / an `overlay` on top of the
|
||||
read-only squashfs image loaded from the boot USB stick). **Anything
|
||||
written there — a plain crontab, a script dropped in `/usr/local/sbin`
|
||||
— is silently gone on the next reboot.** Only `/boot` (the actual USB
|
||||
stick, mounted read/write as a normal filesystem) survives.
|
||||
- `/boot` is **VFAT**, which has no concept of a Unix execute bit at all.
|
||||
A script stored there can never be `chmod +x`'d in any way that sticks —
|
||||
don't try to `exec` it directly from another script; invoke it via
|
||||
`bash <path>` instead. (The **User Scripts** plugin itself handles this
|
||||
correctly when it runs a script *through the plugin* — it copies the
|
||||
content to a RAM-backed tmp location and `chmod +x`s that copy before
|
||||
running it — but a script invoking *another* script directly, as the
|
||||
`-onboot` wrapper here does, has to work around it manually.)
|
||||
- Real, persistent storage lives on the array (`/mnt/user/...`) or on
|
||||
**Unassigned Devices** — extra disks attached but not part of the
|
||||
protected array, mounted at `/mnt/disks/<label>` — which is where these
|
||||
backups actually land. An Unassigned Device is **not guaranteed to
|
||||
already be mounted** when a script runs; see `gitea-backup-pull`'s own
|
||||
handling of this below.
|
||||
- Scheduled tasks persist via the **User Scripts** plugin (already
|
||||
installed, already used for this box's other maintenance), which stores
|
||||
scripts under `/boot/config/plugins/user.scripts/scripts/<name>/` and
|
||||
scheduling in `/boot/config/plugins/user.scripts/schedule.json` — both
|
||||
on the persistent USB stick. `update_cron` folds the JSON's cron-style
|
||||
entries into the live `/etc/cron.d/root` at every boot; `"frequency":
|
||||
"boot"` entries instead run once via the plugin's own `disks_mounted`
|
||||
array-start event hook, **not** through cron at all.
|
||||
- SSH keys under `/root/.ssh` needed no special persistence handling —
|
||||
Unraid already auto-mirrors that whole directory to
|
||||
`/boot/config/ssh/root/` on its own.
|
||||
|
||||
## Files, and where they're actually deployed
|
||||
|
||||
| File here | Deployed path | Purpose |
|
||||
|---|---|---|
|
||||
| `user-scripts/gitea-backup-pull/script` | `/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script` | Pulls new `gitea-dump` archives from `alpha` via the restricted `giteabackup` SFTP account, verifies each one's sha256 checksum, deletes both files on `alpha` only after a confirmed match, prunes its own local retention (30). Resolves and mounts the `WSD2L840` Unassigned Device itself first — see the script's own header comment for the full reasoning. |
|
||||
| `user-scripts/gitea-backup-pull/description` | `.../gitea-backup-pull/description` | Shown in the User Scripts UI. |
|
||||
| `user-scripts/gitea-backup-pull-onboot/script` | `.../gitea-backup-pull-onboot/script` | One-line wrapper: `bash`-invokes the script above. Exists only because `schedule.json` supports one schedule per script *path* — this is how the "also run once at every restart" requirement was added alongside the existing daily `03:30` schedule without disturbing it. |
|
||||
| `user-scripts/gitea-backup-pull-onboot/description` | `.../gitea-backup-pull-onboot/description` | Shown in the User Scripts UI. |
|
||||
| `user-scripts/schedule.json` | `/boot/config/plugins/user.scripts/schedule.json` | **Whole file, as deployed** — includes entries for this box's *other*, pre-existing scheduled maintenance (`delete_dangling_images`, `rc.fancontrol`, etc.) that predate this project and aren't described elsewhere in this repo. Only the `gitea-backup-pull` and `gitea-backup-pull-onboot` entries near the bottom belong to this project. |
|
||||
|
||||
## Redeploying from this checked-in copy
|
||||
|
||||
```bash
|
||||
# on vlda-01:
|
||||
mkdir -p /boot/config/plugins/user.scripts/scripts/gitea-backup-pull
|
||||
mkdir -p /boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot
|
||||
# copy script/description files into place from this repo, then:
|
||||
cp /boot/config/plugins/user.scripts/schedule.json /tmp/user.scripts/schedule.json
|
||||
/usr/local/sbin/update_cron
|
||||
```
|
||||
The dedicated ed25519 keypair this depends on (`/root/.ssh/gitea-backup/`,
|
||||
authorized on `alpha`'s `giteabackup` account) is **not** in this repo —
|
||||
regenerate it and re-authorize the new public key on `alpha` rather than
|
||||
ever committing a private key here.
|
||||
@@ -0,0 +1 @@
|
||||
Runs the gitea-backup-pull sync once at every array startup, in addition to its daily 03:30 schedule - so a restart after being offline catches up right away instead of waiting for the next scheduled time.
|
||||
@@ -0,0 +1,2 @@
|
||||
#!/bin/bash
|
||||
bash /boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script
|
||||
@@ -0,0 +1 @@
|
||||
Pulls new gitea-dump backup archives from alpha.jayfield.org (git.jayfield.org) via the restricted giteabackup SFTP account. Part of the alpha -> vlda-01 backup strategy, see docs-alpha.jayfield.org/SYNC-PLAN.md.
|
||||
@@ -0,0 +1,130 @@
|
||||
#!/bin/bash
|
||||
# vlda-01 side of the alpha -> vlda-01 Gitea backup strategy
|
||||
# (see SYNC-PLAN.md "Ongoing backup" section). Pulls whatever nightly
|
||||
# gitea-dump archives exist on alpha that aren't already here yet - so a
|
||||
# run after several days offline just catches up on all of them, not
|
||||
# just the latest. Verifies each dump's sha256 checksum after pulling
|
||||
# it, and only then deletes the alpha-side copy - a failed/incomplete
|
||||
# verification leaves the remote copy alone for a retry next run.
|
||||
#
|
||||
# Uses SFTP (not real rsync) because the alpha-side account is
|
||||
# restricted to `ForceCommand internal-sftp` - that blocks a real
|
||||
# `rsync --server` invocation, so plain SFTP get/rm is the compatible
|
||||
# transport here.
|
||||
#
|
||||
# Stored on the WSD2L840 Unassigned Device (not the array), which isn't
|
||||
# guaranteed to already be mounted - this script mounts it itself first
|
||||
# and refuses to proceed if that didn't actually result in a real
|
||||
# mountpoint, rather than risk silently writing backups onto the array.
|
||||
|
||||
set -u
|
||||
KEY=/root/.ssh/gitea-backup/id_ed25519
|
||||
KNOWN_HOSTS=/root/.ssh/gitea-backup/known_hosts
|
||||
REMOTE=giteabackup@alpha.jayfield.org
|
||||
REMOTE_PORT=10022
|
||||
UD_LABEL=WSD2L840
|
||||
UD_MOUNTPOINT=/mnt/disks/WSD2L840
|
||||
LOCAL_DIR="$UD_MOUNTPOINT/git"
|
||||
KEEP=30
|
||||
|
||||
# WSD2L840 is an Unassigned Device, not part of the array - not
|
||||
# guaranteed to already be mounted when this runs. Resolve by label
|
||||
# (not a hardcoded /dev/sdX - USB/UD device paths aren't stable across
|
||||
# reboots) and mount it via the Unassigned Devices plugin's own tool,
|
||||
# which is safe to call even if it's already mounted.
|
||||
UD_DEV=$(findfs LABEL="$UD_LABEL" 2>/dev/null)
|
||||
if [ -z "$UD_DEV" ]; then
|
||||
echo "$(date -Is) ERROR: no device labeled $UD_LABEL found - is it plugged in?" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
/usr/local/sbin/rc.unassigned mount "$UD_DEV" >/dev/null 2>&1
|
||||
|
||||
if ! mountpoint -q "$UD_MOUNTPOINT"; then
|
||||
echo "$(date -Is) ERROR: $UD_MOUNTPOINT did not come up as a real mountpoint after mount attempt - refusing to write backups onto the array by mistake" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$LOCAL_DIR"
|
||||
|
||||
SFTP="sftp -P $REMOTE_PORT -i $KEY -o UserKnownHostsFile=$KNOWN_HOSTS -o BatchMode=yes"
|
||||
|
||||
REMOTE_LISTING=$($SFTP "$REMOTE" <<< 'ls -1 backups/' 2>/dev/null)
|
||||
if [ -z "$REMOTE_LISTING" ]; then
|
||||
echo "$(date -Is) ERROR: could not list remote backups (host down or auth failed)" >&2
|
||||
exit 1
|
||||
fi
|
||||
REMOTE_ZIPS=$(echo "$REMOTE_LISTING" | grep '\.zip$')
|
||||
|
||||
GET_BATCH=$(mktemp)
|
||||
DEL_BATCH=$(mktemp)
|
||||
trap 'rm -f "$GET_BATCH" "$DEL_BATCH"' EXIT
|
||||
|
||||
TO_FETCH=0
|
||||
while read -r fname; do
|
||||
[ -z "$fname" ] && continue
|
||||
base=$(basename "$fname")
|
||||
if [ ! -f "$LOCAL_DIR/$base" ]; then
|
||||
echo "get $fname $LOCAL_DIR/" >> "$GET_BATCH"
|
||||
TO_FETCH=$((TO_FETCH + 1))
|
||||
# grab the checksum alongside it if alpha actually has one
|
||||
# (older dumps from before checksums existed won't)
|
||||
if echo "$REMOTE_LISTING" | grep -qxF "${fname}.sha256"; then
|
||||
echo "get ${fname}.sha256 $LOCAL_DIR/" >> "$GET_BATCH"
|
||||
fi
|
||||
fi
|
||||
done <<< "$REMOTE_ZIPS"
|
||||
|
||||
if [ "$TO_FETCH" -eq 0 ]; then
|
||||
echo "$(date -Is) up to date, nothing new to pull"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if ! $SFTP -b "$GET_BATCH" "$REMOTE" 2>&1; then
|
||||
echo "$(date -Is) ERROR: sftp batch fetch failed partway through" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# verify each freshly-fetched dump before trusting it enough to delete
|
||||
# the alpha-side copy
|
||||
VERIFIED=0
|
||||
UNVERIFIED=0
|
||||
while read -r fname; do
|
||||
[ -z "$fname" ] && continue
|
||||
base=$(basename "$fname")
|
||||
local_zip="$LOCAL_DIR/$base"
|
||||
local_sum="$LOCAL_DIR/$base.sha256"
|
||||
[ -f "$local_zip" ] || continue # wasn't newly fetched this run
|
||||
|
||||
if [ ! -f "$local_sum" ]; then
|
||||
echo "$(date -Is) WARNING: no checksum available for $base (older dump, predates checksums?) - keeping local copy, leaving alpha's copy alone" >&2
|
||||
UNVERIFIED=$((UNVERIFIED + 1))
|
||||
continue
|
||||
fi
|
||||
|
||||
if (cd "$LOCAL_DIR" && sha256sum -c "$(basename "$local_sum")") >/dev/null 2>&1; then
|
||||
echo "rm $fname" >> "$DEL_BATCH"
|
||||
echo "rm ${fname}.sha256" >> "$DEL_BATCH"
|
||||
VERIFIED=$((VERIFIED + 1))
|
||||
else
|
||||
echo "$(date -Is) ERROR: checksum mismatch for $base - deleting the bad local copy, leaving alpha's copy for a retry" >&2
|
||||
rm -f "$local_zip" "$local_sum"
|
||||
UNVERIFIED=$((UNVERIFIED + 1))
|
||||
fi
|
||||
done <<< "$REMOTE_ZIPS"
|
||||
|
||||
if [ -s "$DEL_BATCH" ]; then
|
||||
if ! $SFTP -b "$DEL_BATCH" "$REMOTE" 2>&1; then
|
||||
echo "$(date -Is) WARNING: some verified files could not be deleted on alpha - harmless, alpha's own retention prune will clean them up eventually" >&2
|
||||
fi
|
||||
fi
|
||||
|
||||
# prune to the last $KEEP archives (vlda-01 has far more headroom than
|
||||
# alpha, so this is the real long-term history now that alpha deletes
|
||||
# its own copies right after a verified pull)
|
||||
cd "$LOCAL_DIR" || exit 1
|
||||
ls -1t gitea-dump-*.zip 2>/dev/null | tail -n +$((KEEP + 1)) | while read -r old; do
|
||||
rm -f "$old" "${old}.sha256"
|
||||
done
|
||||
|
||||
echo "$(date -Is) pulled $TO_FETCH new archive(s), $VERIFIED verified+deleted from alpha, $UNVERIFIED kept unverified, $(ls -1 gitea-dump-*.zip 2>/dev/null | wc -l) retained locally"
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"/boot/config/plugins/user.scripts/scripts/delete_dangling_images/script": {
|
||||
"script": "/boot/config/plugins/user.scripts/scripts/delete_dangling_images/script",
|
||||
"frequency": "disabled",
|
||||
"id": "scheduledelete_dangling_images",
|
||||
"custom": ""
|
||||
},
|
||||
"/boot/config/plugins/user.scripts/scripts/delete.ds_store/script": {
|
||||
"script": "/boot/config/plugins/user.scripts/scripts/delete.ds_store/script",
|
||||
"frequency": "disabled",
|
||||
"id": "scheduledelete-ds_store",
|
||||
"custom": ""
|
||||
},
|
||||
"/boot/config/plugins/user.scripts/scripts/rc.fancontrol/script": {
|
||||
"script": "/boot/config/plugins/user.scripts/scripts/rc.fancontrol/script",
|
||||
"frequency": "boot",
|
||||
"id": "schedulerc-fancontrol",
|
||||
"custom": ""
|
||||
},
|
||||
"/boot/config/plugins/user.scripts/scripts/viewDockerLogSize/script": {
|
||||
"script": "/boot/config/plugins/user.scripts/scripts/viewDockerLogSize/script",
|
||||
"frequency": "disabled",
|
||||
"id": "scheduleviewDockerLogSize",
|
||||
"custom": ""
|
||||
},
|
||||
"/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script": {
|
||||
"script": "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull/script",
|
||||
"frequency": "custom",
|
||||
"id": "schedulegitea-backup-pull",
|
||||
"custom": "30 3 * * *"
|
||||
},
|
||||
"/boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot/script": {
|
||||
"script": "/boot/config/plugins/user.scripts/scripts/gitea-backup-pull-onboot/script",
|
||||
"frequency": "boot",
|
||||
"id": "schedulegitea-backup-pull-onboot",
|
||||
"custom": ""
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user