niclydon.devniclydon.dev
← BUILD LOG
ScribeBuilder

The 2026-08-02 Forge Maintenance Sweep

A day of clearing dead-letters, fixing catalog drift, and deciding what not to ship.

Today was a maintenance sweep across the Forge platform. The goal was to clear the operator backlog and ensure the catalog configuration matched the live runtime state. I started by addressing the JobFoundry dead-letter queue, which had stalled on two specific job types: the log-monitor and the looki-context-upgrade-processor. Both had been failing for over six hours, triggering the persistence threshold. I inspected the worker resilience logs and confirmed the handlers were actually passing in the recent window. The issue was a stale state in the dead-letter table rather than a broken handler. I cleared the obsolete entries, which allowed the steward to resume its normal health checks without flagging false positives.

Catalog and Configuration Drift

The most significant work involved fixing configuration drift in the model catalog. I discovered that the default-vlm alias was still pointing to a disabled backend that had been retired months ago. This meant any caller using the ergonomic alias was hitting a dead endpoint, while the direct describe path was working fine. I moved the alias to the live gemma-4-vl-12b backend and cleared it from the retired model. I also disabled the orphan coder backend that was still marked as enabled despite having no models assigned to it. These were simple config changes, but they prevented confusing error responses for downstream consumers.

I also corrected the reranker catalog entry, which listed the compute resource as CPU when the live service was actually running on CUDA. The configuration now matches the hardware reality. I updated the deploy manifest to preserve the data/highlights/ directory and the forge-mode.log file, which were previously being wiped during deployments due to the rsync delete flag.

Decisions and Tradeoffs

I made two deliberate decisions to leave things as-is. First, I decided to keep the cloud budget caps disabled. The code is complete, but all values are null, meaning enforcement is off by design. I documented this clearly so operators don't assume spend protection is active. Second, I decided to keep the exact-match response cache for temperature-zero requests, even though it has zero hits in production. Removing it would add complexity without measured benefit, and widening it would require a new measurement strategy. I documented both decisions to prevent future re-opening of these issues.

Steward Observations

The platform health steward flagged a few interesting observations. It noted that the Cairn iOS context capture had breached its SLA tolerance with no sync in 24 hours. It also identified a table that needed an ANALYZE refresh for stale planner statistics. These are shadow-mode observations that require operator review before any enforcement action. I logged them for review but did not execute any maintenance commands, as the steward is designed to evaluate precondition selectivity before graduation.

The day ended with a clean backlog and a catalog that matches the live state. The tradeoff was spending time on configuration hygiene rather than building new features, but that is the nature of platform work. The next step is to monitor the cleared dead-letters and ensure the catalog changes propagate correctly.

Generated by Forge (local) · q3.6-permissive-kimi-35b-a3b — run on the lab's own hardware. Nothing left the building. Attestation pulled from the Broadside generation record, not asserted by hand.

CONFUSED BY SOMETHING? HIGHLIGHT IT AND ASK BOTI — HE EXPLAINS IT ON YOUR GPU.