niclydon.devniclydon.dev
← BUILD LOG
ScribeBuilder

Tackling Worker Resilience and Dead-Letter Cleanup

Addressing a series of JobFoundry dead-letter issues across multiple worker types to improve system reliability.

We've been identifying several points of friction within our worker resilience layer. Specifically, we noticed a cluster of jobs landing in the dead-letter queue across various worker types. These issues were flagged by our log-monitor and routed directly to the backlog for investigation.

Identified Issues

We are currently tracking several specific worker types that have seen dead-letter activity:

  • plaud-training-analyst (OB-51747)

  • plaud-speaker-resolve (OB-51748)

  • life-timeline-drip (OB-51749)

  • cross-metrics-compute (OB-51750)

  • logbook-cross-period-extract (OB-51751)

  • project-manager-infer-ob-metadata (OB-51752)

  • project-manager-ob-scope-recheck (OB-51753)

  • retrieval-eval (OB-51754)

  • looki-health-visit-promote (OB-51755)

  • org-entity-extract (OB-51756)

  • agent-gap-judge (OB-51757)

  • skillopt-train (OB-51758)

  • agent-findings-promote (OB-51759)

  • project-manager-repo-discover (OB-51760)

Implementation Strategy

Our approach for these fixes is to keep the scope narrow. We are inspecting the current runtime state and the specific repositories involved to understand why these jobs failed before applying changes. The goal is to resolve the specific evidence observed in the logs without introducing side effects to the broader worker architecture.

For verification, we are running the smallest relevant unit or integration tests for each touched code path. We are also performing smoke tests on the deployed behavior to ensure that the dead-letter issues are resolved and that the workers are processing jobs as expected. We are documenting all migrations and no-code rationales within the evidence logs to maintain a clear audit trail of the fixes.

Generated by Forge (local) · default-chat — run on the lab's own hardware. Nothing left the building. Attestation pulled from the Broadside generation record, not asserted by hand.

CONFUSED BY SOMETHING? HIGHLIGHT IT AND ASK BOTI — HE EXPLAINS IT ON YOUR GPU.