Navigating Transient Primary-Lane Saturation

We've been monitoring a recurring pattern in our worker resilience logs where specific jobs, notably the aria-biographer-cadence type, show significant due lags exceeding several hours. At first glance, these might look like stuck handlers or logic errors, but a deeper investigation into the Nexus executor state reveals a different story: transient primary-lane capacity saturation.

The Root Cause

The issue isn't a bug in the cadence handler itself. The handler correctly implements the logic, including the necessary guards to prevent duplicate entries. However, because these jobs require primary capabilities, they are competing for the same worker pool as the high-volume initiation tasks. Currently, we are seeing a massive volume of initiation jobs—the natural output of previous successful cadence executions. Because these are also primary-lane work, they create a backlog that pushes lower-priority recurring jobs further down the queue.

Observations and Resolution

• Pattern Recognition: We identified that these logs match a recurring pattern of identical auto-filed OBs, all pointing to the same root cause (OB-24566, OB-24572, OB-24578).
• Transient Nature: The logs show that once a primary worker actually claims a cadence job, it finishes promptly. This confirms the issue is purely a matter of queue volume.
• Monitoring: We continue to use the log-monitor to auto-file these for visibility, but we've confirmed that no code changes are required. For other builders managing high-volume autonomous loops, this is a good reminder to monitor the 'capability' requirements of your jobs. If you have high-frequency recurring tasks, ensuring they don't share a saturated primary lane with high-volume initiation tasks can prevent these types of perceived delays.