Hardening Deployment Pipelines Against Long-Running Locks

During a recent deployment for OB-51219, we hit a significant bottleneck. A migration intended to drop a table was blocked by an active pg_dump process that had been running for nearly two hours. Because the backup held an AccessShareLock on numerous relations, our migration—which required an AccessExclusiveLock—sat idle while the runtime services remained paused. This type of contention can turn a routine deployment into a prolonged outage.

The Solution

To solve this, we implemented two coordinated guards in our migration and deployment tooling:

• Bounded Lock Timeouts: We updated the migration runner to execute each migration on a dedicated client with a specific lock_timeout. This ensures that if a DDL statement is blocked by a long-running process like a backup, it will fast-fail with a clear error rather than hanging the entire deployment pipeline.

• Pre-emptive Backup Detection: We modified the deployment script to probe for active pg_dump sessions before attempting to stop services or apply migrations. If a backup is detected, the system will now surface a dedicated deploy-blocker or fail closed, preventing the system from entering a stalled state.

Verification and Tradeoffs

We verified these changes by ensuring the migration runner correctly logs the likely lock-holder on a fresh connection when a timeout occurs. We also ensured that configuration settings are reset before the connection is released to prevent leaks into the connection pool. These changes were hardened through careful review to ensure we don't lose evidence of the blocker during the migration phase. This approach prioritizes system availability by failing fast over waiting indefinitely for resources that may be held by non-interactive processes.