👻 wraith-modelgen

Sovereign dbt Scaffolding for the Ghost Stack

wraith-modelgen is a BigQuery dbt model scaffolder. Feed it a YAML data contract, get back a pair of sovereign data products: an Origin (raw) layer that passes everything through, and a Consumption (staging) layer that holds the stable line for downstream consumers.

Source schemas change. Columns get renamed, types get tightened, and new fields appear without warning. In most analytics codebases that change lands downstream: dashboards break, FDPs fail, and the analytics team finds out from a support ticket.

wraith-modelgen makes the data contract the change-management mechanism. The source team updates it as part of their release. wraith-modelgen regenerates the models. Downstream sees what it sees. Nothing breaks unless something it actually depended on disappears, which triggers a generation failure with the column name.

🚀 Installation

uv tool install wraith-modelgen

You'll also need Application Default Credentials for BigQuery introspection:

gcloud auth application-default login

Everything except new also runs without them. --from-contract generates from the contract's declared columns alone (and is the automatic fallback when no credentials are found). That's how CI runs it.

⚙️ Workflow

modelgen init                                  # scaffold .modelgen.yml + dirs (once)
modelgen new my-project.landing.user_signups   # draft a contract from a live table
modelgen validate contracts/user_signups.yml   # offline semantic lint, all findings at once
modelgen run                                   # regenerate every layer of every contract
modelgen check --diff                          # CI: fail if committed models drifted

Single layers when you need them. --raw and --staging are mutually exclusive:

modelgen gen contract.yml --raw     -o ./models/raw
modelgen gen contract.yml --staging -o ./models/staging
modelgen gen contract.yml --raw --dry-run  # preview without writing

📄 Contract anatomy

Contracts are Open Data Contract Standard (ODCS) v3.0.2 documents, validated against the bundled ODCS JSON schema. modelgen's BigQuery/dbt specifics ride in customProperties.

version: "1.0.0"
kind: DataContract
apiVersion: v3.0.2
id: 8b1f6c2e-5a3d-4e7a-9f21-3c4d5e6f7a8b
status: active
name: user_signed_up

servers:
  - server: landing                    # source project + dataset
    type: bigquery
    project: my-gcp-project
    dataset: landing

schema:
  - name: user_signed_up               # the entity → dbt model stem
    physicalName: user_signups_raw      # source table
    physicalType: table

    customProperties:
      - property: modelgenLoadedAtField  # SOURCE column for incremental loads
        value: RECEIVED_AT
      - property: modelgenRaw
        value:
          dataset: raw
          incrementalStrategy: merge
          dedup: true                    # row_number() partition by primaryKey
          partitionBy: { field: RECEIVED_AT, dataType: timestamp, granularity: day }
          clusterBy: [USER_ID]
          freshness:                     # -> dbt source freshness
            warnAfter:  { count: 12, period: hour }
            errorAfter: { count: 24, period: hour }
      - property: modelgenStaging
        value:
          dataset: staging
          materialization: snapshot   # ds snapshot; derives partitionBy: ds
          clusterBy: [user_id]

    properties:
      - name: event_id                  # staging (consumer-facing) name
        physicalName: EVENT_ID          # source column name
        logicalType: string
        physicalType: STRING            # cast target
        description: "..."              # >= 50 chars, and not just the name
        primaryKey: true                # contributes to dbt unique_key
        primaryKeyPosition: 1
        required: true                  # → not_null test
        unique: true                    # → unique test
      # accepted_values → quality: [{ type: library, rule: validValues, validValues: [...] }]

🔐 Classification and erasure

A property marked classification: pii or psi is encrypted on the way into staging and never materialised in plaintext. Same seam as required: true → not_null: the contract says it, the generator does it, no human in the loop.

      - property: modelgenSubjectField
        value: USER_ID                 # who the row is about

      - name: user_id
        physicalName: USER_ID
        classification: pii            # subject → DETERMINISTIC_ENCRYPT
      - name: email
        physicalName: EMAIL
        classification: psi            # payload → AEAD.ENCRYPT

The subject column is encrypted deterministically, so its ciphertext is stable per keyset and joins, row counts and aggregates all survive erasure. Everything else is randomised. Both resolve their keyset from a keystore holding one wrapped key pair per subject.

Right to erasure is one DELETE. Destroy the subject's keystore row and every ciphertext written under those keys becomes unrecoverable, across every table, without rewriting a single partition. A historical regulatory position still reconstructs in aggregate; the individual does not.

Full design, key hierarchy (CMEK vs KEK vs DEK), erasure timeline and citations: Keystore & Erasure.

🔄 Schema evolution

Validation runs as part of modelgen gen --staging. If it passes, the generated staging model still presents the same contract to downstream consumers.

✅ Contract standards

modelgen validate --strict is the gate. Beyond the structural checks, it holds contracts to a few standards that only matter once more than one team reads them:

An exemption is scoped either to the whole contract or to a single column, and the two never cross:

      - name: internal_note
        customProperties:
          - property: modelgenExempt
            value:
              - rule: low-quality-column-description
                reason: "deprecated in TJP-00042, drops next quarter"

Exempted findings still print on every validate run, labelled exempt with their reason, and the summary counts them. They stop blocking, they do not stop being visible. Full rule list: Lint Rules.

🏗️ What gets generated

For --raw:

• raw__event.sql: dbt incremental model with {{ source(...) }}, optional dedup window, partition and cluster config.
• raw__event.yml: dbt sources entry plus model definition. Columns mirror the introspected source schema.

For --staging:

• stg__event.sql: dbt model reading {{ ref('raw__event') }}. Casts and renames applied.
• stg__event.yml: model definition with column tests from the contract.

Staging defaults to a datestamped daily snapshot: insert_overwrite partitioned on a generated ds column, one full copy of the dimension per day. History is where ds = '<date>', and any date backfills in parallel with the same SQL. The orchestrator injects it:

dbt run --select stg__event --vars '{ds: 2026-07-27}'

The grain is (key, ds), so the generated unique test is scoped to the partition the run writes. materialization: incremental on modelgenStaging opts back into a high-water-mark load, and then incrementalStrategy and partitionBy are yours to declare.

Both layers load incrementally. Raw merges on the unique key, staging replaces one ds partition per run. Neither rebuilds the whole table, and no generated model ever emits CREATE OR REPLACE TABLE ... AS SELECT. Full rebuild is available as a documented exception rather than a default, which is covered in Tooling Decisions.

🧪 Developer Quality Gate

# Clone and set up
git clone https://git.thomaspeoples.com/thomaspeoples/wraith-modelgen
cd wraith-modelgen
uv run poe setup        # syncs deps + installs pre-commit hooks

# The quality gate
uv run poe test         # pytest (interactive)
uv run poe test-ci      # pytest with coverage enforcement (≥80%)
uv run poe lint         # ruff check
uv run poe format       # ruff format

The test suite uses FakeIntrospector and runs without BigQuery credentials. Covers contract parsing, semantic linting, both layers, schema evolution scenarios, composite keys, REPEATED/RECORD types, determinism, drift detection, contract scaffolding, and error surfaces.

Committing

All commits go through commitizen with the Ghost Stack convention:

👻 <type>/<ticket>: <message>

uv run cz commit

📜 Sovereign Principles

1. One layer per invocation. Layers have different lifecycles; conflating them makes things harder to reason about.
2. Introspection at gen time. The warehouse is the source of truth for column names and types. Drift is impossible because nothing is duplicated.
3. Deterministic output. Same contract + same source schema = byte-identical files. modelgen check turns this into a CI gate.
4. Strict failure on missing columns. No silent passes. If the contract references a column the source no longer has, generation fails with the column name.
5. Standards are enforced, exceptions are written down. A rule that is wrong for one contract is exempted in the contract itself, with a reason, and the exemption stays visible on every run.
6. BigQuery only. The introspection module is BQ-native. Add a different Introspector implementation if you need another warehouse.

Part of the Ghost Stack. Sovereign. Self-hosted. No nonsense.