Write-up · Home lab · September 2026

The backup that said SUCCESS and was missing a photo

A verified restore of 15,123 files from an encrypted offsite copy, and the three things that were quietly wrong before I checked.

The setup

Immich is the self-hosted photo platform my family's phones back up to. It runs on a TrueNAS VM with two ZFS mirrors: the library on one pool, Postgres on the other. When I built it, the box had zero backup coverage: no snapshot tasks, no replication, cloud sync disabled. The weekly Proxmox backup job that appeared to cover the NAS could not see either pool, because the disk controller is passed through to the VM; it was imaging the 32 GB OS disk and writing the result to an NFS export served by the same NAS. The dashboard said "backed up". Nothing that mattered was.

So the first job was three real layers:

1. ZFS snapshots as the undo button: hourly for a day, daily for two weeks, weekly for eight, with separate naming schemes so retention never collides.
2. Nightly database dumps from Immich's own backup job (it handles the vector extension correctly, which a hand-rolled pg_dump does not), copied to the other pool by cron.
3. An encrypted offsite copy: rclone crypt to a Swift-compatible object store, run from a ZFS snapshot so the copy is point-in-time consistent, filenames encrypted too, regenerable trees (thumbnails, transcodes) excluded. The key and salt live in a password manager, not on the box.

Then I did the part most people skip.

The restore test that failed silently

With rclone at its defaults, restoring a six-file sample returned five files and reported SUCCESS, "There was nothing to transfer". Five attempts in a row, always missing the same file. The object was on the remote, at the right size; walking the encrypted tree by hand found it.

The cause was the provider's directory listings. Listing the same unchanged folder six times returned three entries four times and two entries twice, always dropping the last one alphabetically. Root listings were stable; nested listings were not. Any client that walks directories will lose files, and it will not know.

The fix was one setting. With fast_list = true, rclone asks for a single flat listing of every object under the prefix and rebuilds the tree on the client, so it never touches the unreliable per-directory listings. Same restore, one flag changed: six of six, every file byte-identical, the photo's SHA256 matching exactly. I turned it on for the push too, because sync decides what to delete on the far side from that same listing.

Scaling it up

After a 100 GB import (an old SD card, OneDrive, Google Photos, a console's clip folder, 15,105 assets in total) the nightly sync reported FAILED three nights running while the app looked perfectly healthy. An independent check told the truth:

rclone check --one-way --size-only --fast-list
15123 / 15123 matching, 0 missing, 0 differing

The data was never missing. Two separate faults were producing the failures:

• Mid-transfer: HTTP 503 or Put: EOF, always after everything queued had already been sent. TrueNAS passes rclone no retry flags at all, so it was running on bare defaults. Fixed with --low-level-retries 20 --retries 5 --retries-sleep 30s --timeout 300s --contimeout 60s in the task's arguments.
• At startup: Failed to create file system for "encrypted:/": HTTP Error 404. This one fires while rclone is still constructing the encrypted filesystem, before any transfer machinery exists, so no retry flag can catch it. It hit roughly one run in two.

The only cure for the second fault is running the whole task again, so a small Python wrapper runs four hours after the sync: it reads the last job's state, exits if that is RUNNING or a SUCCESS newer than 20 hours, and otherwise re-runs the sync up to three times with five-minute gaps. Its first real run recovered the three-night failure on the first attempt, in 5 minutes 26 seconds.

The restore, for real

A restore test is only honest if it starts from nothing, the way it would after losing the NAS. So I built a fresh rclone configuration on a clean path from the password-manager entry alone and pulled data back:

• all 8 database dumps: byte-identical by SHA256, valid gzip, valid pg_dump output;
• 60 random library files including the five largest: 60 of 60 byte-identical.

Two things came out of that. First, the password and salt were not enough: the encrypted remote also had to be declared with filename_encryption = standard, directory_name_encryption = true, fast_list = true and the exact remote path, or it silently returned nothing. Those four lines now sit in the password manager beside the key. Second, a full 26 GB library restore needed more than one pass: 1,396 of 1,400 files, then 1,398, with Object Not Foundon objects the listing had just advertised. Every file that did arrive was byte-identical, so nothing was corrupt; the rule is simply "re-run until the count matches". That is acceptable for a second copy and unacceptable for the only one, which is why a second provider is still on my list.

Why nobody had told me

The three failed nights had raised alerts. I never saw one. The NAS's mail alert service had an empty from-address and no outgoing server, and the admin account had no email, so every alert died on the box. The Proxmox node was blind in exactly the same way: its only notification target was local sendmail with no relay host, and the previous week's backup-failure mail was still sitting in the postfix queue addressed to a user that does not exist. That is also how a weekly backup of the NAS configuration, the one that holds the encryption key material, stayed switched off for ten days after a maintenance window.

Fixed the same day: a Telegram alert service on the NAS at warning level and above, a Telegram webhook target on Proxmox with the bot token stored as a secret rather than in the URL, the dead mail channel disabled so it stops looking like coverage, and the config backup re-enabled. Each one tested with the platform's own test call, not assumed.

What I would tell someone building the same thing

• A backup you have never restored is a hypothesis. Test it with files you can checksum, and test it from a clean machine with only what the password manager holds.
• Watch the listing, not just the transfer. Sync tools decide what to keep and what to delete from the destination listing; if that lies, the tool lies with it, and it still reports success.
• Alert delivery is part of the backup. Test the path end to end, and disable dead channels so they cannot pass for coverage.
• Write the restore recipe where it survives the box. Mine needed four settings the key alone did not carry.
• Keep two independent copies. One provider that needs multiple passes is a fine second copy.

Numbers as of 2026-09-01: 15,105 assets, 109 GB, 15,123 of 15,123 objects verified offsite, restore samples byte-identical, nightly job self-healing, failures paged to my phone.

Back to the lab