outlabs-release

The release train and rollout discipline tooling for schema-owning Outlabs libraries (outlabs-taskq, outlabs-auth) and the applications that embed them.

Why this exists

A library that owns database tables gives every consumer a second version axis: the code a deploy ships, and the schema a migration produces. Nothing ties them together by default, so a deploy can silently move one without the other — the classic failure is code shipped, migration forgotten, and every health check green while a feature quietly breaks.

The industry answers are old and boring, and this package is just those:

• Attestation (Rails' PendingMigrationError, generalized): every process compares what its installed code expects against what each database says, at boot and on demand. Auth/host behind = refuse loudly. A TaskQ contract behind or ahead is a rollout state only when the installed runtime's closed set explicitly supports that exact contract.
• An explicit skew policy (Kubernetes-style): each schema-owning runtime declares the exact database contracts it can run against. TaskQ migrations use runtime-first rollout because an older process may reject the newer SQL contract even when both versions share a major number.
• A release train (Spring Boot BOM / Debian stable): one certified combination of package versions, so consumers track one number instead of a matrix.

The current 2026.10 train certifies outlabs-taskq==0.1.0a41 (SQL contract 0.6.12, migration 0048_queue_admission_owner_recovery) with outlabs-auth==0.1.0a36 (Auth schema head 20260802_0025).

The three verbs

Every consumer, every topology, the same three motions:

outlabs-release preflight   # read-only: drift verdicts + taskq pending fingerprints
# deploy the compatible artifact everywhere and fence older TaskQ processes
outlabs-release migrate --database primary --confirm --runtime-first-confirmed
# boot prints one attestation line per domain, all "match"

Plus outlabs-release attest (preflight without the fingerprints) and outlabs-release train show|check (the certified combination vs what is installed).

The manifest — shape, never versions

A consumer commits outlabs-release.toml next to its pyproject.toml. It declares which domains live in which databases and nothing else; expected versions always derive from installed code at run time.

[database.primary]
dsn_env = "POSTGRES_DSN"
domains = ["taskq", "outlabs_auth", "host"]

[domain.taskq]
# Boot/attestation uses the least-privilege runtime credential.
dsn_env = "TASKQ_RUNTIME_DATABASE_URL"
# Operator preflight uses the short-lived owner credential to read the ledger.
plan_dsn_env = "OWNER_DATABASE_URL"
# Set false only when the consumer has a separate history-aware empty-DB bootstrap.
allow_fresh_install = true

[domain.outlabs_auth]
schema = "outlabs_auth"

[domain.host]
alembic_dir = "alembic"
version_table = "public.alembic_version"

A worker whose queue lives in a remote database declares only that:

[database.queue]
dsn_env = "TASKQ_DSN"
domains = ["taskq"]

Declaring a domain asserts it must exist there (missing blocks). A consumer without auth simply never declares outlabs_auth.

Boot gate

from outlabs_release.boot import enforce_at_boot

# in the application lifespan / worker startup
await enforce_at_boot("outlabs-release.toml", enforce=settings.SCHEMA_DRIFT_ENFORCE)

All reads use the restricted runtime role. Non-TaskQ behind and every missing verdict raise when enforcement is on. A TaskQ database may be numerically behind (compatible-behind) or ahead only when the installed TaskQ runtime lists that exact contract in its closed compatibility set; otherwise attestation reports incompatible and raises. This lets the new runtime boot against the old supported contract before the migration while still failing closed for an unknown contract.

The migrate verb

outlabs-release migrate replaces per-consumer closeout scripts whose operators had to retype every expected version as a flag. It derives expectations from installed code, orders the domains correctly (auth -> taskq -> host, because host migrations may verify the taskq manifest), passes the owner DSN to child processes only through the environment, and ends by attesting the database — refusing to report success unless every declared domain exactly matches. Compatible rollout skew is accepted at boot and preflight, never as migration completion. The operator supplies exactly three things: the owner DSN in the environment, --confirm, and (for a production-bound taskq target) the installation-id acknowledgement. When an existing TaskQ database has pending migrations, it also requires --runtime-first-confirmed. This is the operator's assertion that every API, worker, scheduler, operator, migrator, and maintenance process already runs the compatible artifact and that older processes cannot reconnect. The guard is evaluated before Auth, TaskQ, or host migrations mutate the target.

The train

src/outlabs_release/train.toml is the one legitimate place versions are restated: the certified combination. train check compares installed packages against it. Attestation never reads it.

Boundary

This repository is consumer-agnostic. Consumer names, topologies, and deployment specifics stay in the consumers.

The complete verdict table, runtime-first sequence, mutation gates, and recovery boundary are in docs/Runtime-First Rollout.md.

Release-specific changes and verification are recorded in docs/RELEASE-<version>.md; the current train was certified in docs/RELEASE-0.1.0a5.md.

Releasing

Releases publish to PyPI through trusted publishing: the Publish release workflow (.github/workflows/publish-pypi.yml) exchanges a short-lived GitHub OIDC token for the upload, so no PyPI token is stored anywhere. The PyPI project trusts exactly this publisher:

To cut a release:

1. Merge a pull request that sets version in pyproject.toml and adds docs/RELEASE-<version>.md. A train change also updates train.toml, the exact development pins, and uv.lock.

2. Tag the merge commit on an up-to-date main with v plus the package version, and push the tag:

   git switch main && git pull --ff-only
   git tag -a v0.1.0a5 -m "outlabs-release 0.1.0a5"
   git push origin v0.1.0a5
   

3. The tag push runs Publish release. Its verify job fails unless the tag equals the package version, then runs ruff check, ruff format --check, mypy, and pytest, builds the sdist and wheel with uv build, and smoke-tests the installed wheel. Only v* tags trigger it; pushes and pull requests run ci.yml alone.

4. The publish-pypi job runs in the pypi GitHub environment with id-token: write and uploads the verified distributions with pypa/gh-action-pypi-publish. When the environment has required reviewers, approve the pending deployment on the workflow run; nothing reaches PyPI until that approval. A publish-github job attaches the same distributions to a GitHub prerelease.

5. Confirm the version on https://pypi.org/project/outlabs-release/ and that its file hashes match the run's python-package-distributions artifact.

A published version is immutable on PyPI: fix a bad release with the next version, never by moving a tag.