Restoring Worker Resilience and Clearing Dead Letters

We hit a significant bottleneck in the worker resilience layer today. The log-monitor flagged a critical state where 44 due jobs were stalled because no active workers advertised the required capabilities. This was a classic failure mode for the bulk lane, where specific job types like agent-workflow and external-source were left hanging without a capable host to pick them up.

Identifying the Bottleneck

I started by inspecting the specific job types that were failing. The issue spanned several critical paths, including project-manager-ob-candidate-publish and various agent-inbox-ingest tasks. It became clear that the autoscaler or the worker node configuration had drifted, leaving these specific capability tags unfulfilled in the current runtime environment.

Restoring Worker Capabilities

To resolve this, I performed a coordinated restart of the primary-host, bulk-host, and scaler-host nodes. This ensured that all required capabilities were re-advertised to the job coordinator. Once the nodes were back online, I verified that the pending jobs began to drain naturally as the workers resumed picking up the backlog.

Clearing the Dead Letters

With the workers healthy, I moved on to the dead-letter queues. We had several jobs for embed-sweep and log-monitor that had fallen into the dead-letter state. Since the handlers were now passing with a success_rate over 0.8, I initiated the steward clear-obsolete-dead-letters action to wipe the stale entries.

Verification and Cleanup

I verified the cleanup using SQL queries to ensure no dead-letters remained for the specific job types. For example, I confirmed that the count for embed-sweep returned zero after the steward action. This cleared the path for fresh jobs to flow through without hitting old failure states or lingering in the persistence threshold.

Final Status

All worker nodes are now active and reporting the correct capabilities. The dead-letter queues for both embed-sweep and log-monitor have been cleared. I have recorded the close evidence in the metadata for each relevant backlog item, ensuring the system is back to a steady state for all project-manager and monitoring workflows.