Retrieval Latency, Deploy Gates, and Agent Governance

We started the day by tackling a critical latency regression in our hybrid retrieval system. Cold natural-language queries were exceeding the 20-second time budget, resulting in empty results. The root cause was twofold: the embedding step carried a 60-second timeout with average cold latencies near 11 seconds, and 25 of 27 configuration tables ran full sequential cosine scans on 4096-dimensional vectors without HNSW indexes. To fix this, we adopted a two-pronged strategy. First, we reduced the embedding timeout to approximately 8 seconds to ensure failures surface within budget. Second, we plan to migrate the per-row tables to use the central embeddings table, which already benefits from the existing HNSW substrate. This approach leverages existing infrastructure to permanently reduce scan costs.

Deployment Stability

A separate but equally impactful issue emerged in the deployment pipeline, which was silently failing because a governance gate ran before the service restart loop. The gate was blocked by stale watchdog entries, causing the MCP service to never restart with new code. We identified this as a deploy-ordering defect where the gate aborts the script under set -e before services can be cycled. The recommended fix is to switch the gate to a warning mode, mirroring an established pattern elsewhere in the platform. This allows deploys to proceed while logging the hygiene warning, ensuring that critical tool updates actually reach the live environment.

Agent Runtime Governance

We also spent time refining the governance layer for our autonomous agents. The SecurityAnalyst agent was blocked from certification due to a desync between its active database grants and its grant profile. The profile had zero items despite six live grants. The fix is straightforward: populate the profile to match the manifest, ensuring the reconciler sees consistent state. Additionally, we reviewed two candidates proposing to embed lessons from tool failures into agent memory. We rejected both because one failure was a permanent stub returning a placeholder, and the other was a single non-recurring event. Embedding such noise would mislead future agent iterations.

Other Updates

Several other tasks moved forward. We deferred building an additional ANN index until sequential retrieval latency becomes a measurable bottleneck. We shipped the movie half of the recommendation feedback loop, capturing play and skip signals into a feedback ledger. Finally, we audited worker embedding call sites to ensure query versus document roles are correctly passed to the asymmetric embedding model, improving cosine similarity for search tasks.