Minio Mount Recovery, Dead-Letter Cleanup, and a Redeploy Loop

Minio Mount Recovery

While shipping OB-54229 from an isolated homelab worktree on origin/main, I ran the Minio mount recovery test suite and hit two failures. The first test, which asserts that a USB flap does not latch Minio into a failed state, expected a StartLimitBurst value of at least ten. The systemd unit file had no explicit StartLimitBurst directive, so it fell back to the default of five. The second test, checking that a genuine crash loop still trips the limit, failed in the opposite direction because the default burst of five combined with a ten-second restart interval produced an unexpected result. I filed this as OB-55378 so the test expectations can be reconciled with the actual systemd configuration.

Dead-Letter and Job Resilience

The log-monitor dead-letter queue accumulated a job that had been failing since early morning. The kg-fact-adjudicate job also landed in dead-letter status after its handler began passing consistently. I filed OB-55390 and OB-55392 to track these. The embed-sweep dead-letter queue had six entries whose handler now shows a success rate above 0.8 over a sampled window, making them eligible for the clear-obsolete-dead-letters action. I proposed clearing those as part of the OB-55391 and OB-55395 work. The logbook-period-summary job also dead-lettered, which I tracked under OB-55391.

Nexus Redeploy

The main deployment work today centered on Nexus. A deploy from origin/main initially failed because the staging TypeScript compilation included test files that referenced vitest, which is not installed in the production dependency tree. I reproduced the issue in a deploy worktree, confirmed the tsc error, and branched from main to exclude tests from the production tsconfig. After committing and pushing, I opened a PR and waited for the compile-bot to finish. The redeploy at commit 2b5d0d68 with test-excluded tsc completed successfully after about two minutes and nineteen seconds.

Backup State Directory

While shipping the leftover snapshot work from OB-54358, I noticed the backup-run state directory could not be created because the parent path was owned by root with 755 permissions. The nightly backup script runs as the niclydon user, so the retention-failure counters never persisted. I filed OB-55383 with a fix to create the directory with correct ownership during the install script, rather than leaving a one-off mkdir as the durable solution.

Identity and Schema Cleanup

The kg-fact-adjudicate dead-letter also surfaced a schema issue: the person_ids column in the bronze layer was a leftover from an earlier migration that had been replaced by a resolved person IDs view. I filed OB-55392 to drop that column after confirming no live views or in-repo readers still depend on it. The undo-merge path for photo face name links also needed attention after an identity merge, which I tracked under OB-55394.

What Changed

The day closed with the Minio test expectations documented, four dead-letter queues identified for cleanup, the Nexus production build fixed to exclude test files, the backup state directory ownership corrected, and two schema migrations queued. The common thread was resilience: making sure that when a component fails or a job stalls, the failure is visible, bounded, and recoverable without manual intervention.