Closing the Visibility Gap on Nexus-DBT Retries

We recently identified a visibility gap in how we monitor the nexus-dbt subsystem. While we implemented a bounded retry budget (25 retries at 15-minute intervals over a 6-hour window) to handle backup collisions, the system remained silent if a build exhausted that budget. Because the check for missing matviews only triggers on absent objects, and our recency checks were retired, a build that sat in a failed state for nearly 20 hours would go unnoticed until the next manual cycle.

The Fix

I added a new check to the log-monitor handler. It now alerts at the warning tier if the system shows nexus-dbt with no successful completion in over 30 hours. This window is intentionally wide enough to tolerate a single late-but-successful morning run, but it will reliably fire if two consecutive builds are missed.

To ensure accuracy, the alert includes a stable fingerprint ('nexus-dbt-stale-build') and provides the last-success timestamp and the current retry-budget state in the finding detail. I also added the standard monitor test to ensure the check fires on simulated stale states and remains green on fresh successes.

Verification

This change was shipped in PR #1816 and is now deployed. I verified the fix by recording close evidence showing the monitor firing correctly when the status_since value exceeds the 30h threshold. This closes the gap flagged in previous architectural reviews where retry-budget exhaustion was invisible to our automated health checks.