diff --git a/README.md b/README.md index a756c44..141d5eb 100644 --- a/README.md +++ b/README.md @@ -373,13 +373,20 @@ config together) — also what Gitea's own restore procedure expects. the date: `gitea dump` refuses to overwrite an existing filename, which a plain per-day name would hit on any manual re-run (found this the hard way testing it). -- Prunes to the last 14 archives. +- Writes a `sha256sum`-format `.sha256` file alongside each dump, so + `vlda-01` can verify integrity after transfer before trusting it enough + to delete the alpha-side copy (see below). - Applies a POSIX ACL (`setfacl -m u:giteabackup:r`) to each new archive - so the pull account below can read it. **Has to happen explicitly per - file** — a default ACL on the directory alone isn't enough: `gitea dump` - creates each file `600`, and the resulting ACL mask neuters the - inherited grant (`getfacl` shows `#effective:---` if this step is - skipped). + **and its checksum file** so the pull account below can read them. + **Has to happen explicitly per file** — a default ACL on the directory + alone isn't enough: `gitea dump` creates each file `600`, and the + resulting ACL mask neuters the inherited grant (`getfacl` shows + `#effective:---` if this step is skipped). +- Prunes to the last 14 archives (dump + checksum pairs together) — a + **safety net only**: in normal operation `vlda-01` deletes each dump + right after a verified pull, well before this would ever trigger. Only + matters if `vlda-01` is offline long enough for dumps to pile up + unswept. **Transport — a dedicated, restricted account on alpha**, not a shared or personal credential: @@ -387,18 +394,23 @@ personal credential: existing `sftp` group — reuses the `Match Group sftp` / `ForceCommand internal-sftp` block already in `sshd_config` for `ftp_jens`/`ftp_alex`; no new sshd config needed. -- Chrooted to `/var/sftp/giteabackup/`, containing only a **read-only bind - mount** of `/srv/gitea/data/backups` (`/etc/fstab`, `bind,ro`) — even a - misused key can only ever read backup archives, never write, delete, or - reach anything else on the host. Verified: `touch` inside the mount - fails with "Read-only file system" even as root. +- Chrooted to `/var/sftp/giteabackup/`, containing only a bind mount of + `/srv/gitea/data/backups`. **Read-write** at the filesystem level (`rw` + in `/etc/fstab` — needed so `vlda-01` can delete a dump after verifying + it), but the *account's* actual privilege is narrower than that sounds: + the directory-level ACL grants `giteabackup` `rwx` (needed for + delete/unlink, which is a directory operation in POSIX), while each + individual file's ACL only grants `r`. Verified the boundary actually + holds: an SFTP `rm` on an existing dump succeeds, but an SFTP `put` + attempting to overwrite one fails with "Permission denied" — it can + delete a file wholesale, never tamper with one in place. - Key-only auth: a dedicated ed25519 keypair generated on `vlda-01` (private key never leaves it), `authorized_keys` further restricted (`restrict,no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty`) on top of the chroot/`ForceCommand`. - **Plain SFTP, not real `rsync`**: `ForceCommand internal-sftp` blocks a genuine `rsync --server` invocation from the client, so the transport - is SFTP `get`, not rsync. + is SFTP `get`/`rm`, not rsync. **On `vlda-01`** — via the **User Scripts** plugin, not a raw crontab/`/usr/local/sbin` script: Unraid's `/` and `/usr` are RAM-backed @@ -412,13 +424,36 @@ box's other scheduled maintenance, no new persistence mechanism invented. The SSH keypair needed no special handling: Unraid already auto-mirrors `/root/.ssh` to `/boot/config/ssh/root/` on its own. - Lists what's actually on alpha via SFTP, fetches only filenames not - already present locally — a run after several days offline catches up - on all of them, not just the newest, without re-downloading anything - already pulled. + already present locally (dump + its `.sha256`, when one exists — older + dumps from before checksums existed won't have one, and are just kept + as unverified without touching alpha's copy) — a run after several days + offline catches up on all of them, not just the newest, without + re-downloading anything already pulled. +- **Verifies each freshly-fetched dump's checksum before deleting + anything on alpha.** A match triggers an SFTP `rm` of both the dump and + its checksum file on alpha; a mismatch deletes the *bad local copy* + instead and leaves alpha's copy alone for a retry next run — a failed + or partial transfer never results in either end trusting a corrupt file + or losing the only good copy. +- **Also runs once at every array startup** (`gitea-backup-pull-onboot`, + a one-line wrapper around the same script, registered with + `"frequency": "boot"` — `schedule.json` only supports one schedule per + script path, so this needed a second entry rather than combining it with + the daily one), in addition to the daily `03:30` schedule — so a restart + after being offline catches up right away instead of waiting for the + next scheduled time. Fired via the User Scripts plugin's own + `disks_mounted` boot-event hook, which calls `startSchedule.php boot` + once per real boot; verified by invoking that exact call manually + (without a real reboot) and confirming the wrapper actually ran the + underlying script. **Gotcha hit twice building this**: a script that + tries to directly `exec` another file living on `/boot` fails with + "Permission denied", since `/boot` is VFAT and can never carry a Unix + execute bit — the wrapper has to invoke it via `bash ` instead of + relying on the file's own (nonexistent) `+x`. - Lands in `/mnt/disks/WSD2L840/git/` — an **Unassigned Device** (external disk, not part of the protected array), not the RAM-backed root — keeps - the last 30 locally, deeper history than alpha's own 14, given the - disk's ample headroom. + the last 30 locally, now the real long-term history given alpha deletes + its own copies right after a verified pull. - **The device isn't guaranteed to already be mounted** when the script runs, so it resolves the disk by filesystem label (`findfs LABEL=WSD2L840`, not a hardcoded `/dev/sdX` — USB/Unassigned-Device @@ -433,11 +468,13 @@ The SSH keypair needed no special handling: Unraid already auto-mirrors Gitea container, unpack the chosen dump's DB/repos/etc. into `/data` per Gitea's documented restore steps, restart, verify. Not automated. -**Verified end-to-end**: a real archive produced on alpha, pulled to -`vlda-01` via the actual scheduled-invocation code path (`startCustom.php`, -not just the raw script) and confirmed as a genuinely new file; a second -run correctly reported "up to date, nothing new to pull" rather than -re-fetching. +**Verified end-to-end**, including the full checksum-verify-then-delete +flow with a genuinely fresh dump (not just a re-run against existing +files): dump + checksum produced on alpha, pulled to `vlda-01` via the +actual scheduled-invocation code path (`startCustom.php`), checksum +verified locally (`sha256sum -c` → `OK`), both files then deleted from +alpha (confirmed empty afterward) — all through the real production +script, not a manual approximation of it. ### Password vault sync (KeePass, via Nextcloud)