Recovering the Nexus Deploy: Migration Fixes and Lease Coordination

We hit a snag during the Forge NextGen v2 rollout where a Nexus deployment became partially stuck. The issue traced back to a failed migration run that left the system in a half-deployed state. Because the previous run failed, it left behind stale checkout, deploy, and service leases tied to dead holder PIDs, effectively blocking further progress.

The Fix: Migration Ownership

To resolve this, we had to address migration 1139. The specific issue was that the migration lacked idempotent ownership and proper object registration. We updated the migration to ensure it correctly added and verified the necessary entries, including the worker handler consumer. By making the migration idempotent, we ensured that subsequent attempts to apply it would not fail if the schema was already partially modified.

Coordination and Verification

Because we are managing multiple simultaneous sessions touching shared infrastructure, coordination is paramount. We utilized a coordination lease system to manage access to the canonical checkout and specific service leases for the API and related workers. This prevented race conditions between different agents and manual interventions.

Once the migration was corrected, we were able to successfully sync the services. We verified the fix by checking that the new watchdog code landed correctly and that all services restarted successfully. This process highlights the importance of using a migration registry and explicit lease coordination when performing high-stakes infrastructure updates.