dbt-dqm

dbt-dqm is an open-source dbt package for persistent, row-level tracking of dbt test failures, plus a single-user local review app. Each failed row becomes an issue with a stable identity, a lifecycle (Active → Archived when the test passes) and reviewer annotations that survive later runs.

Status and supported versions

The Python app pins dbt-core 1.12.5; its optional extras use dbt-bigquery 1.12.1 and dbt-postgres 1.11.x; the dbt package accepts the wider dbt range above. Python support is >=3.11,<3.15. The release checklist lists what must pass before a release is tagged; the 0.2.0a4 evidence records this alpha's results. Package contracts (relations, migrations, grants, recovery, retention, maintenance) are in warehouse interfaces.

Install the dbt package

Install the alpha through Git by adding it to your consumer project's packages.yml:

packages:
  - git: "https://github.com/a-adamovich/dbt-dqm.git"
    revision: v0.2.0a4

The alpha is not listed on dbt Hub: Hub's SemVer tag parser does not accept v0.2.0a4, so install this alpha through the Git revision above. Hub registration is planned for 0.2.0.

The 0.2 app expects the matching 0.2 package; don't mix it with 0.1.

Then run dbt deps. The review app is a separate Python package (dbt-dqm); installing it doesn't install the dbt package, and vice versa.

On Postgres, create the extension once (the package doesn't, because it usually needs elevated privileges): create extension if not exists pgcrypto;

Install the Python app and start reviewing

In a Python 3.14 environment (recommended; 3.11–3.14 supported), install the app separately:

python3.14 -m venv .venv
source .venv/bin/activate
python -m pip install 'dbt-dqm[bigquery]==0.2.0a4'

Warehouse adapters are optional. In a checkout, install only the adapter you need with uv sync --extra bigquery or uv sync --extra postgres; use uv sync --all-extras for both. Select the extra in your installed environment:

python -m pip install 'dbt-dqm[bigquery]==0.2.0a4'          # BigQuery
python -m pip install 'dbt-dqm[postgres]==0.2.0a4'          # Postgres
python -m pip install 'dbt-dqm[bigquery,postgres]==0.2.0a4' # Both

A plain install includes the app and dbt-core but no warehouse adapters. Shared UI dependencies such as PyArrow remain installed. Existing environments retain adapters already installed. Earlier 0.2.0a2 installs included both adapters.

On Windows PowerShell, create and activate the environment with:

py -3.12 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install 'dbt-dqm[bigquery]==0.2.0a4'

Activation is optional: uv run --extra bigquery dbt-dqm app uses the checkout environment. For an installed environment, run its dbt-dqm command directly.

Configure your dbt profile, add the Git package above, and set a fresh schema in your consumer project's dbt_project.yml when upgrading from 0.1:

vars:
  dbt_dqm_schema: dqm_alpha

After adding a DQM test as shown below, run it separately from your regular model workflow:

dbt deps
dbt-dqm run
dbt-dqm app

The dedicated command initializes tracking relations, captures results, and reconciles even when quality tests fail. Project, profiles and target flags remain optional for both commands.

In the browser, Sync, edit an issue's notes or review verdict, then Apply. Health shows setup, tests and maintenance status.

Configuration defaults and troubleshooting

From your dbt project or a subfolder, launch with dbt-dqm app; dbt-dqm workspace info and dbt-dqm workspace purge use the same defaults. Explicit flags take precedence:

• Project: --project-dir, then DBT_PROJECT_DIR, then dbt's current-directory/parent lookup.
• Profiles: --profiles-dir, then DBT_PROFILES_DIR, then profiles.yml in the launch directory, then your home directory's .dbt folder.
• Target: --target, then DBT_TARGET, then the rendered profile's target, or default.

DBT_PROFILE, DBT_TARGET_PATH, profile env_var() values, and GOOGLE_APPLICATION_CREDENTIALS are inherited. Only explicit CLI overrides are passed as flags to dbt. Paths use the native operating system's conventions; quote paths containing spaces. The app uses dbt beside its own Python interpreter before searching PATH, and logs that executable and its version. Technical details include failed commands, exit codes and both output streams. If tracking relations are missing, run dbt-dqm run first. This package command works at the normal multithreaded setting; its setup models are ordered.

Custom generate_schema_name and generate_alias_name macros are supported for package models. The app and SQL helpers use the parsed model relations. Macro-created internal tracking tables, such as dqm_issue_occurrences, keep their fixed identifiers in the reconciliation model's schema. Changing naming macros does not migrate or delete existing warehouse data.

BigQuery authentication through WIF

The app and dbt use Application Default Credentials (ADC) for an oauth profile. Supply a trusted external-account configuration from your existing Workload Identity Federation environment; it exchanges the external identity for short-lived service-account access without a downloaded service-account key:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/trusted-wif-adc.json

consumer:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth
      project: your-project
      dataset: your_dataset
      location: US
      threads: 4

Use your consumer project's profile name in place of consumer. Keep the configuration and its external token source available to both commands and the app. WIF configurations belong to the environment owner; dbt-dqm does not create pools, providers, service accounts or IAM bindings. Use the normal ADC environment path, not an explicit credentials-file override, for WIF. Only use configurations from an environment you trust.

The offline WIF smoke test exercises ADC loading, token exchange, service-account impersonation and refresh through the installed app/dbt authentication libraries, with simulated responses. It does not verify live WIF or IAM. Live release QA uses existing local service-account credentials; no new keys are created or downloaded. Optional scheduled BigQuery verification stays inactive until configured.

Migrating existing tests

DQM is disabled by default and runs as a separate workflow. Install the matching warehouse extra, update the Git package revision above, and run dbt deps. Existing installations require these steps:

1. Stop reviewer apps and preserve any annotations you need outside the package. No history migration is provided.
2. Mark each DQM test with a meta.dbt_dqm object. Tags are optional business categories; the legacy dqm tag remains stored but is excluded from app displays and tag filters.
3. Set enabled=var('dqm_enabled', false), store_failures=true, and meta.dbt_dqm.granularity to the output columns identifying the failed record.
4. Use native limit, normally limit=var('dbt_dqm_default_test_limit', 100). Set dbt_dqm_default_test_limit in dbt_project.yml to change that default, or give an individual test its own positive integer limit. A missing native limit warns and uses the project default. Keep fail_calc=count(*) and table failure storage.
5. Set ownership using a nonblank output column dqm_owner, meta.dbt_dqm.owner, or meta.owner, in that order. Blank values fall through. Conflicting row owners use the metadata fallback and produce a Health diagnostic. Ownership initializes new issues; later reviewer edits persist.
6. All output keys and attributes are captured. Remove capture_mode, dbt_dqm_capture_mode, and context_columns; they warn and are ignored. Remove poc_responsible, owner_column, dqm_granularity, and dbt_dqm_test_tag; these obsolete aliases are unsupported.
7. Configure retention for captured observations and event payloads. The dedicated workflow warns when either full-value history has no corresponding retention setting.
8. Run dbt-dqm run --full-refresh. This explicitly resets known DQM tracking tables and manifest-resolved package relations, assigns a fresh installation identity, then runs DQM. It preserves consumer models, stored-failure relations, unrelated tables, and the dataset. On PostgreSQL an external dependent view prevents reset; deletion never cascades to consumer objects. On BigQuery empty replacement copies preserve destination table access; reset verifies table policies.
9. Clear each old local workspace with dbt-dqm workspace purge, then restart reviewer apps. Old annotation submissions are bound to the previous installation and cannot be applied after reset.

Example singular test (the same configuration belongs under config: for a generic YAML test):

{{ config(
    enabled=var('dqm_enabled', false),
    store_failures=true,
    limit=var('dbt_dqm_default_test_limit', 100),
    tags=['customer', 'contact'],
    meta={'owner': 'analytics', 'dbt_dqm': {
      'granularity': ['customer_id'], 'owner': 'customer_support', 'priority': 'high'
    }}
) }}
select customer_id, reason, assigned_team as dqm_owner
from {{ ref('customers') }} where is_invalid

# dbt_project.yml
vars:
  dbt_dqm_default_test_limit: 100
  dbt_dqm_retention_days: 30
  dbt_dqm_event_retention_days: 90

Run the dedicated workflow

From your dbt project, after your regular models have run:

dbt-dqm run                         # enable DQM, validate, test, capture, reconcile, maintain
dbt-dqm run --select tag:customer --exclude tag:experimental --threads 4
dbt-dqm run --vars '{dbt_dqm_default_test_limit: 250}'
dbt-dqm run --full-refresh          # explicit reset; tracking history is discarded

PowerShell uses the same commands. To avoid shell differences for YAML variables:

dbt-dqm run --vars "dbt_dqm_default_test_limit: 250"
dbt-dqm run --full-refresh
dbt-dqm workspace purge

Project, profiles and target flags are optional and follow the dbt/environment defaults described above. The worker uses dbt's programmatic runner in this Python environment, preserving the launch working directory and inherited credentials. Large selections and result payloads stay out of the Windows subprocess command line. Only metadata-marked tests matching the selection run; invalid selected configurations stop before initialization or reset. App startup only parses metadata and reads warehouse state; it does not run tests, capture results or execute package models.

Test failures still produce captured evidence and reconciliation. Exit codes are 0 for success, 1 for failed tests, and 2 for configuration or execution errors. A dbt warn result retains dbt's successful exit behavior. Automatic capture and retention on-run-end hooks have been removed. Ordinary dbt commands leave DQM disabled unless you explicitly enable it.

Limits apply to raw failed rows before deduplication. Samples are ordered by granularity identity and full payload before the native limit. Reaching the exact limit is ambiguous: the full failure count is unknown. Such executions have collection_status=limit_reached, plus failure_limit and captured_issue_count. They reconcile observed positive evidence but never close omitted issues through absence. A later complete execution can resolve them. Health shows limited collections.

Serialize DQM workflows sharing stored-failure tables: equal counts do not prove two invocations read the same rows. BigQuery supports one reconciliation per dataset at a time. Per-test capture commits preserve earlier successful captures if a later test fails during collection.

dbt_dqm_schema optionally chooses the tracking schema. With standard dbt schema naming, dbt_dqm_schema: governance produces <target_dataset>_governance. Package model relations respect custom schema and alias macros; macro-owned tracking identifiers remain fixed under the reconciliation node's database and schema. Changing naming macros does not migrate or delete old tables.

Retention and maintenance

• dbt_dqm_retention_days (a positive integer) prunes raw executions and observations that reconciliation has already processed, and old run-ledger rows.
• dbt_dqm_event_retention_days prunes the dqm_issue_events history.

These run as the last phase of dbt-dqm run, or on demand with dbt run-operation cleanup_dqm_logs --vars "{dqm_enabled: true}". A SQL error inside a maintenance step never fails the dbt run; outcomes are logged and shown in dqm_maintenance_health and the app's Health tab. The exact guarantee and its limits are in warehouse interfaces.

Event values outlive raw logs by default. With dbt_dqm_event_payloads: full (the default) and no dbt_dqm_event_retention_days, captured before/after values are kept forever. Under full capture they can contain personal data. Set event retention, or choose changed_columns (column names plus hashes, which are not anonymization) or none.

Local review app

dbt-dqm app \
  --project-dir /path/to/consumer-project \
  --profiles-dir ~/.dbt \
  --target demo

• It uses the selected dbt profile (validated with dbt debug), including env_var() authentication and custom generate_schema_name behavior, and synchronizes at the start of each browser session.
• It caches a project-and-target-specific snapshot plus pending edits in an owner-only SQLite workspace under the user-data directory. Credentials are never copied into it. dbt-dqm workspace info and dbt-dqm workspace purge (with --project-dir and --target) inspect or delete it.
• Edits use compare-and-set versions: a concurrent change produces an explicit conflict rather than a silent overwrite. On Postgres, a write blocked by a running reconciliation gives up after --lock-timeout seconds (default 5) and reports the warehouse as busy.
• Each sync is verified before it replaces the cache; an inconsistent read keeps the cache and pending edits.
• Cache limits: at most 50,000 issues and 128 MiB of serialized issue data (--max-cache-issues and --max-cache-mib can only lower them). The cache holds Active issues plus 90 days of Archived history; --archive-cache-days 0 keeps Active issues only. An oversized sync keeps the existing cache and pending edits. These limits bound serialized data, not process memory; see scale.
• Backlogs above the cache limits (--browse, default auto): the app syncs a local snapshot while the review population fits the limits, and above them reads issues from the warehouse page by page instead. Filters, search, counts and the sort then run in the warehouse; only recently loaded pages (up to 2,000 issues or 32 MiB) and issues with pending edits are kept locally, and edits keep the same compare-and-set versions. When the warehouse can't be read, the cached pages are shown offline. --browse snapshot or --browse warehouse forces a mode.
• Warehouse cost in that mode: every page, count and search is a query. On BigQuery the app reads 2,000 issues at a time (about 1 GB billed at 1M occurrences, serving 80 pages of 25), a header summary costs about 0.16 GB, and a search is refused before it runs if a dry run estimates more than --max-search-gib (default 1). Postgres needs migration 0007's index for fast pages. Measured figures: scale.

Demo projects

integration_tests/demo_postgres and integration_tests/demo_bigquery contain the same synthetic consumer: one seed, three tracked tests of different shapes, and an assertion of the expected outcome for each scenario. No company data is included.

cd integration_tests/demo_postgres   # or integration_tests/demo_bigquery
dbt deps
dbt run-operation reset_demo_dqm
dbt seed
dbt run --select demo_records --vars '{demo_scenario: initial}'
dbt-dqm run --vars '{demo_scenario: initial}'
dbt test --select assert_scenario_outcomes --vars '{demo_scenario: initial, dqm_enabled: true}'

Repeat the last three commands with demo_scenario: passed (every issue archives as a genuine pass), then demo_scenario: recurrence (fresh occurrences for selected identities). The scenario assertion checks the exact active and archived counts, and assumes this reset → initial → passed → recurrence order.

Adapter notes:

• Postgres: create pgcrypto first; start from the demo's profiles.yml.example.
• BigQuery: keep the service-account key outside the repository (for example ~/.credentials/dbt-dqm/service-account.json). The demo's schema macro stores test failures in the same pre-created dataset, so its account needs only BigQuery Job User on the project and BigQuery Data Editor on that dataset.

Roadmap

The package and the local app are, and will stay, Apache-2.0 open source. A hosted offering may follow, but nothing in the package or app is gated behind it. See the roadmap.

License

Copyright 2026 Aliaksei Adamovich. Licensed under the Apache License, Version 2.0. See LICENSE.md.