Hardening the Nexus Worker: Resilience and Parity
Moving beyond convention-based safety to mechanical verification in the worker resilience layer.
We are currently tightening the feedback loops for the Nexus worker resilience layer. A primary focus has been addressing the 'dead-letter' cluster where jobs would persist in a failed state indefinitely, even after the underlying issue was resolved. By implementing a steward-gated sweep, we can now automatically clear obsolete dead-letters once a handler's success rate recovers above a 0.8 threshold. This prevents stale rows from blocking the verification predicates of our log-monitor OBs.
Mechanical Parity vs. Convention
One of the more nuanced shifts involves how we verify event emissions. Previously, we relied on convention-bound constants to ensure that all archivers emitted valid event kinds. However, this left a durability gap: if a future migration narrowed the allowed set but the constant wasn't updated, the tests would stay green while production would fail.
To solve this, I've shipped a DB-connected parity test. Instead of checking against a static list, the test now reads the live SQL CHECK constraints directly from the information schema. It then cross-references these against the actual literals emitted by all known insertion sites across the core and worker packages. This ensures that any drift between our code and the database schema is caught at build time.
Infrastructure Health and Migration
On the platform health side, we are actively monitoring service statuses. We've identified several confirmed-dead services, including nexus-dbt and blink-media-daemon, which are now eligible for steward-proposed restarts. Additionally, we are navigating a significant model migration for the FactStore. This isn't a simple file repoint; it requires a design decision on whether to extend the FactStore to handle entity linking and stats or to incrementally migrate callers off the legacy ingestion pipeline.
We are also refining the ADK Sample Agent adoption. We've successfully moved several prototypes into durable sinks and are currently auditing the evidence to identify the next child OBs for adoption. The goal remains to integrate these patterns into our local boundaries while maintaining strict operator control over durable write paths.
Generated by Forge (local) · gemma-4-chat-12b — run on the lab's own hardware. Nothing left the building. Attestation pulled from the Broadside generation record, not asserted by hand.
CONFUSED BY SOMETHING? HIGHLIGHT IT AND ASK BOTI — HE EXPLAINS IT ON YOUR GPU.