Add checksums, delete-after-verified-sync, and boot trigger to Gitea backup
Three changes to the alpha -> vlda-01 backup pipeline: - alpha now writes a sha256sum-format checksum alongside every dump, ACL'd for the pull account same as the dump itself. - vlda-01 verifies each freshly-pulled dump's checksum before trusting it enough to delete the alpha-side copy - a mismatch removes the bad local copy instead and leaves alpha's for a retry, so neither end can end up trusting a corrupt file or losing the only good copy. This required loosening the giteabackup account from strictly read-only: the bind mount is now rw and the directory ACL grants rwx (needed for delete, a directory-level operation in POSIX), but per-file ACLs stay read-only - verified the boundary holds (rm succeeds, put/overwrite still fails with Permission denied). - vlda-01 now also runs the sync once at every array startup via a second User Scripts entry (schedule.json only supports one schedule per script path), not just the daily 03:30 - hit the same VFAT execute-bit issue as before along the way (a wrapper can't directly exec a /boot-resident file, has to invoke it via bash). Verified the full pipeline end-to-end with a genuinely fresh dump: checksum generated, fetched, verified locally, both files deleted from alpha only after a confirmed match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -373,13 +373,20 @@ config together) — also what Gitea's own restore procedure expects.
|
|||||||
the date: `gitea dump` refuses to overwrite an existing filename, which
|
the date: `gitea dump` refuses to overwrite an existing filename, which
|
||||||
a plain per-day name would hit on any manual re-run (found this the hard
|
a plain per-day name would hit on any manual re-run (found this the hard
|
||||||
way testing it).
|
way testing it).
|
||||||
- Prunes to the last 14 archives.
|
- Writes a `sha256sum`-format `.sha256` file alongside each dump, so
|
||||||
|
`vlda-01` can verify integrity after transfer before trusting it enough
|
||||||
|
to delete the alpha-side copy (see below).
|
||||||
- Applies a POSIX ACL (`setfacl -m u:giteabackup:r`) to each new archive
|
- Applies a POSIX ACL (`setfacl -m u:giteabackup:r`) to each new archive
|
||||||
so the pull account below can read it. **Has to happen explicitly per
|
**and its checksum file** so the pull account below can read them.
|
||||||
file** — a default ACL on the directory alone isn't enough: `gitea dump`
|
**Has to happen explicitly per file** — a default ACL on the directory
|
||||||
creates each file `600`, and the resulting ACL mask neuters the
|
alone isn't enough: `gitea dump` creates each file `600`, and the
|
||||||
inherited grant (`getfacl` shows `#effective:---` if this step is
|
resulting ACL mask neuters the inherited grant (`getfacl` shows
|
||||||
skipped).
|
`#effective:---` if this step is skipped).
|
||||||
|
- Prunes to the last 14 archives (dump + checksum pairs together) — a
|
||||||
|
**safety net only**: in normal operation `vlda-01` deletes each dump
|
||||||
|
right after a verified pull, well before this would ever trigger. Only
|
||||||
|
matters if `vlda-01` is offline long enough for dumps to pile up
|
||||||
|
unswept.
|
||||||
|
|
||||||
**Transport — a dedicated, restricted account on alpha**, not a shared or
|
**Transport — a dedicated, restricted account on alpha**, not a shared or
|
||||||
personal credential:
|
personal credential:
|
||||||
@@ -387,18 +394,23 @@ personal credential:
|
|||||||
existing `sftp` group — reuses the `Match Group sftp` /
|
existing `sftp` group — reuses the `Match Group sftp` /
|
||||||
`ForceCommand internal-sftp` block already in `sshd_config` for
|
`ForceCommand internal-sftp` block already in `sshd_config` for
|
||||||
`ftp_jens`/`ftp_alex`; no new sshd config needed.
|
`ftp_jens`/`ftp_alex`; no new sshd config needed.
|
||||||
- Chrooted to `/var/sftp/giteabackup/`, containing only a **read-only bind
|
- Chrooted to `/var/sftp/giteabackup/`, containing only a bind mount of
|
||||||
mount** of `/srv/gitea/data/backups` (`/etc/fstab`, `bind,ro`) — even a
|
`/srv/gitea/data/backups`. **Read-write** at the filesystem level (`rw`
|
||||||
misused key can only ever read backup archives, never write, delete, or
|
in `/etc/fstab` — needed so `vlda-01` can delete a dump after verifying
|
||||||
reach anything else on the host. Verified: `touch` inside the mount
|
it), but the *account's* actual privilege is narrower than that sounds:
|
||||||
fails with "Read-only file system" even as root.
|
the directory-level ACL grants `giteabackup` `rwx` (needed for
|
||||||
|
delete/unlink, which is a directory operation in POSIX), while each
|
||||||
|
individual file's ACL only grants `r`. Verified the boundary actually
|
||||||
|
holds: an SFTP `rm` on an existing dump succeeds, but an SFTP `put`
|
||||||
|
attempting to overwrite one fails with "Permission denied" — it can
|
||||||
|
delete a file wholesale, never tamper with one in place.
|
||||||
- Key-only auth: a dedicated ed25519 keypair generated on `vlda-01`
|
- Key-only auth: a dedicated ed25519 keypair generated on `vlda-01`
|
||||||
(private key never leaves it), `authorized_keys` further restricted
|
(private key never leaves it), `authorized_keys` further restricted
|
||||||
(`restrict,no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty`)
|
(`restrict,no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty`)
|
||||||
on top of the chroot/`ForceCommand`.
|
on top of the chroot/`ForceCommand`.
|
||||||
- **Plain SFTP, not real `rsync`**: `ForceCommand internal-sftp` blocks a
|
- **Plain SFTP, not real `rsync`**: `ForceCommand internal-sftp` blocks a
|
||||||
genuine `rsync --server` invocation from the client, so the transport
|
genuine `rsync --server` invocation from the client, so the transport
|
||||||
is SFTP `get`, not rsync.
|
is SFTP `get`/`rm`, not rsync.
|
||||||
|
|
||||||
**On `vlda-01`** — via the **User Scripts** plugin, not a raw
|
**On `vlda-01`** — via the **User Scripts** plugin, not a raw
|
||||||
crontab/`/usr/local/sbin` script: Unraid's `/` and `/usr` are RAM-backed
|
crontab/`/usr/local/sbin` script: Unraid's `/` and `/usr` are RAM-backed
|
||||||
@@ -412,13 +424,36 @@ box's other scheduled maintenance, no new persistence mechanism invented.
|
|||||||
The SSH keypair needed no special handling: Unraid already auto-mirrors
|
The SSH keypair needed no special handling: Unraid already auto-mirrors
|
||||||
`/root/.ssh` to `/boot/config/ssh/root/` on its own.
|
`/root/.ssh` to `/boot/config/ssh/root/` on its own.
|
||||||
- Lists what's actually on alpha via SFTP, fetches only filenames not
|
- Lists what's actually on alpha via SFTP, fetches only filenames not
|
||||||
already present locally — a run after several days offline catches up
|
already present locally (dump + its `.sha256`, when one exists — older
|
||||||
on all of them, not just the newest, without re-downloading anything
|
dumps from before checksums existed won't have one, and are just kept
|
||||||
already pulled.
|
as unverified without touching alpha's copy) — a run after several days
|
||||||
|
offline catches up on all of them, not just the newest, without
|
||||||
|
re-downloading anything already pulled.
|
||||||
|
- **Verifies each freshly-fetched dump's checksum before deleting
|
||||||
|
anything on alpha.** A match triggers an SFTP `rm` of both the dump and
|
||||||
|
its checksum file on alpha; a mismatch deletes the *bad local copy*
|
||||||
|
instead and leaves alpha's copy alone for a retry next run — a failed
|
||||||
|
or partial transfer never results in either end trusting a corrupt file
|
||||||
|
or losing the only good copy.
|
||||||
|
- **Also runs once at every array startup** (`gitea-backup-pull-onboot`,
|
||||||
|
a one-line wrapper around the same script, registered with
|
||||||
|
`"frequency": "boot"` — `schedule.json` only supports one schedule per
|
||||||
|
script path, so this needed a second entry rather than combining it with
|
||||||
|
the daily one), in addition to the daily `03:30` schedule — so a restart
|
||||||
|
after being offline catches up right away instead of waiting for the
|
||||||
|
next scheduled time. Fired via the User Scripts plugin's own
|
||||||
|
`disks_mounted` boot-event hook, which calls `startSchedule.php boot`
|
||||||
|
once per real boot; verified by invoking that exact call manually
|
||||||
|
(without a real reboot) and confirming the wrapper actually ran the
|
||||||
|
underlying script. **Gotcha hit twice building this**: a script that
|
||||||
|
tries to directly `exec` another file living on `/boot` fails with
|
||||||
|
"Permission denied", since `/boot` is VFAT and can never carry a Unix
|
||||||
|
execute bit — the wrapper has to invoke it via `bash <path>` instead of
|
||||||
|
relying on the file's own (nonexistent) `+x`.
|
||||||
- Lands in `/mnt/disks/WSD2L840/git/` — an **Unassigned Device** (external
|
- Lands in `/mnt/disks/WSD2L840/git/` — an **Unassigned Device** (external
|
||||||
disk, not part of the protected array), not the RAM-backed root — keeps
|
disk, not part of the protected array), not the RAM-backed root — keeps
|
||||||
the last 30 locally, deeper history than alpha's own 14, given the
|
the last 30 locally, now the real long-term history given alpha deletes
|
||||||
disk's ample headroom.
|
its own copies right after a verified pull.
|
||||||
- **The device isn't guaranteed to already be mounted** when the script
|
- **The device isn't guaranteed to already be mounted** when the script
|
||||||
runs, so it resolves the disk by filesystem label (`findfs
|
runs, so it resolves the disk by filesystem label (`findfs
|
||||||
LABEL=WSD2L840`, not a hardcoded `/dev/sdX` — USB/Unassigned-Device
|
LABEL=WSD2L840`, not a hardcoded `/dev/sdX` — USB/Unassigned-Device
|
||||||
@@ -433,11 +468,13 @@ The SSH keypair needed no special handling: Unraid already auto-mirrors
|
|||||||
Gitea container, unpack the chosen dump's DB/repos/etc. into `/data` per
|
Gitea container, unpack the chosen dump's DB/repos/etc. into `/data` per
|
||||||
Gitea's documented restore steps, restart, verify. Not automated.
|
Gitea's documented restore steps, restart, verify. Not automated.
|
||||||
|
|
||||||
**Verified end-to-end**: a real archive produced on alpha, pulled to
|
**Verified end-to-end**, including the full checksum-verify-then-delete
|
||||||
`vlda-01` via the actual scheduled-invocation code path (`startCustom.php`,
|
flow with a genuinely fresh dump (not just a re-run against existing
|
||||||
not just the raw script) and confirmed as a genuinely new file; a second
|
files): dump + checksum produced on alpha, pulled to `vlda-01` via the
|
||||||
run correctly reported "up to date, nothing new to pull" rather than
|
actual scheduled-invocation code path (`startCustom.php`), checksum
|
||||||
re-fetching.
|
verified locally (`sha256sum -c` → `OK`), both files then deleted from
|
||||||
|
alpha (confirmed empty afterward) — all through the real production
|
||||||
|
script, not a manual approximation of it.
|
||||||
|
|
||||||
### Password vault sync (KeePass, via Nextcloud)
|
### Password vault sync (KeePass, via Nextcloud)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user