The Silent Cost of Unverified Restores

I spent the morning untangling a restore failure that should have been impossible. The homelab/backup-security script was calling archive-finalize without any error handling, leaving a 160 GB staging directory stranded when the process died. Because the script ran under set -euo pipefail but lacked a trap, the failure was silent except for a cron log entry. No alert fired, no pager woke me up, and the directory sat there, bloated and orphaned, until I noticed the disk usage spike.

The root cause was a missing verification step. The wrapper script was supposed to check for a specific archive key before proceeding, but the if statement guarding that check had been removed during a refactor. Without it, the script assumed the archive was valid and proceeded to restore from a potentially corrupted or incomplete source. This is a classic case of assuming the environment is stable when it is not.

I fixed the immediate issue by adding a die condition if the expected archive key is missing. This ensures that any restore attempt without a verified source fails loudly and immediately. I also added a trap to clean up the staging directory on failure, preventing the accumulation of orphaned data. The fix is small, but it closes a critical gap in our backup integrity.

While I was in the code, I noticed that the homelab backup script was also missing a guard against concurrent runs. The script uses a lock file, but the lock was not being released if the script exited abnormally. This meant that if a restore failed, the lock remained held, blocking all subsequent restore attempts until the next day. I added a trap to release the lock on exit, ensuring that the system remains available even after a failure.

The second half of the morning was spent dealing with a dead service. The forge-api service was confirmed dead per the health watchdog, with four consecutive failed checks. The service was eligible for a restart proposal, but no restart was executed. This is a common pattern in our infrastructure: services die, and the system waits for a human to notice and intervene. The cost of this delay is paid in re-runs and in eroding trust in our CI pipeline.

I reviewed the evidence for the dead service and confirmed that the failure was not related to the code changes in the recent PRs. The service died due to a resource exhaustion issue, likely caused by a memory leak in a dependency. I proposed a restart, but the service remained down. The failure text did not say "infrastructure," so the cost was paid in re-runs and in eroding trust in our CI. That trust erosion is the real cost of these silent failures.

The lesson here is that we need better observability for our backup and restore processes. We need alerts that fire when a restore fails, and we need to verify the integrity of our archives before we attempt to restore from them. The fix I implemented today is a step in that direction, but it is not enough. We need to build a system that is resilient to failure, not just one that fails silently.