Managing Data Pipeline Hygiene and Dead-Letter Cleanup

Today's focus was on refining the data pipeline's structural integrity. We identified a class of naming convention violators—specifically five tables that do not adhere to the discoverability contract. While some are active external ingests, others are derived summaries or frozen archives. The plan is to audit each individually to decide between a rename-with-compat-view or a simple alias, ensuring no reader breaks during the migration.

Retiring Legacy Paths

We also initiated a cleanup of the old Mac push-staging path. A critical finding during investigation showed that simply dropping these tables would cause permanent data loss. Specifically, nine unique sessions from the codex_cli path exist nowhere else in the crawler silver. To handle this safely, we need to drain the pending rows, repoint the evidence-seed handlers, and then deactivate the payload schemas before archiving the old tables.

Worker Resilience and Steward Actions

On the operational side, we've been monitoring JobFoundry dead-letters. We identified a job in the dead-letter queue for the type 'ai-conversation-to-kg'. We are currently evaluating these for manual intervention while also running steward actions to clear obsolete dead-letters where the success rate has stabilized above 0.8 over the recent window.

Infrastructure Health

Finally, we are looking into the adoption of Google ADK sample-agent patterns into local boundaries. This involves deploying and building evidence to identify the next concrete ADK adoption child OB. We are currently in the investigation phase to ensure these patterns integrate smoothly with our existing infrastructure.