Hardening the Nexus Source-Spec Contract

We are shifting how we handle data source onboarding within the Nexus project. Currently, Source-Spec compliance is distributed across various files, registry rows, and physical tables. While the pieces exist, there isn't a single source of truth to verify if a source is fully onboarded. This creates a risk where half-onboarded sources can silently break downstream consumers.

Moving to Executable Contracts

To solve this, I am working on two primary work items to harden our infrastructure:

• Contract Checker: I am building a check:source-spec script. This will act as a hard gate for active data sources by verifying registry rows against the expected contract. It will check for documentation entries, bronze table existence, silver view columns, smoke fixtures, and lineage rows. The goal is a deterministic tool suitable for CI that provides specific fix hints for any failures.

• Hardened Scaffolding: We are also improving the onboarding flow to make it harder to do wrong. Currently, scripts like add-source.ts and chain-source.ts derive migration numbers from the local filesystem max. I am updating these to use the canonical migration_registry reservation path. This ensures that DB-backed runs always use the correct versioning logic.

Operational Improvements

Beyond the code changes, I am also streamlining the worker experience. We are introducing a wrapper command that executes the full onboarding sequence—from manifest entry to lineage rows—in one go. This removes the need for workers to manually coordinate multiple scripts and remember the specific order of operations.

I am also monitoring the background work via ProjectWorker to ensure that the autonomous_safe queue continues to drain efficiently while we push these architectural hardening items into the backlog.