Hardening Worker Resilience and Cleaning Dead-Letters

We have been focusing heavily on worker resilience within the JobFoundry system. Several jobs, including kg-salience-recompute, agent-gap-judge, and embed-sweep, were identified in the dead-letter queue. Our primary goal was to determine whether these were systemic failures or transient issues requiring specific tuning.

Tuning and Cleanup

For the agent-gap-judge job, we identified that the failures were primarily due to timeouts. We have since tuned the timeout_ms parameters and cleared the obsolete dead-letters. Similarly, we cleared dead-letters for kg-salience-recompute and plaud-enrich after verifying that their respective handlers are now passing with a success_rate above 0.8 over a sampled window. This ensures our monitoring remains clean and actionable.

Infrastructure and Hygiene

Beyond specific job tuning, we shipped a broader fix for dead-letter handler health-gating and timeout tuning (OB-51538). This was a scoped fix intended to address symptoms across 14 auto-filed dead-letter items. We also performed some hygiene on the decision-lane classifier to ensure it isn't reverting correct operator reroutes back to investigation status unnecessarily.

Verification and Status

We verified these changes by monitoring the success rates of the handlers and ensuring the dead-letter counts dropped to zero. We also addressed a critical migration issue where missing columns in a table were causing plaud-speaker-resolve alerts. Currently, the system is stabilizing, and we are moving toward more autonomous handling of these resilience patterns.