vowl

vowl (vee-owl 🦉) is a validation engine for Open Data Contract Standard (ODCS) data contracts. Define your validation rules once in a declarative YAML contract and get rich, actionable reports on your data's quality.

🏆 Official ODCS Vendor: vowl is actively maintained and listed on the official ODCS vendors list as a natively compatible tool.

Who it's for

• Data engineers and platform teams that need declarative, standards-based data quality validation
• Teams wanting a deterministic quality gate that guards pipelines against unreliable data, including data produced by increasingly automated or AI-assisted workflows
• Anyone working with ODCS data contracts who wants an automated way to enforce them

Table of Contents

Part 1 · Getting Started

• Who it's for
• Features
• Installation
• Validate in 3 lines

Part 2 · Core Concepts

• Data Contracts
• Auto-Generated Checks
• Library Metrics (type: library)
• Format Checks
• Validation Results
• Architecture

Part 3 · Usage Patterns

• Common sources — Local DataFrame · PySpark · Ibis (20+ backends) · Concurrent Checks
• Filtering & cross-source — Filter Conditions · Multi-Source · Compatibility Mode
• Advanced & extending — Servers in Contract · Custom Adapters · Remote Contracts (Git/S3)

More

• Roadmap
• Status
• Contributing
• Owner
• Security
• License

Part 1 · Getting Started

Features

• Extensible Check Engine: Ships with a SQL check engine out of the box, with the architecture designed to support custom check types beyond SQL.
• Auto-Generated Rules: Checks are automatically derived from contract metadata (logicalType, logicalTypeOptions, enum, required, unique, primaryKey) and library metrics (nullValues, missingValues, invalidValues, duplicateValues, rowCount).
• Any DataFrame, Any Backend: Load any Narwhals-compatible DataFrame type (pandas, Polars, PySpark, etc.) or connect to 20+ backends via Ibis. SQL dialect translation is handled by SQLGlot.
• Server-Side Execution: SQL checks run server-side through Ibis without materialising tables on the client.
• Multi-Source Validation: Validate across tables in different source systems with cross-database joins.
• Declarative ODCS Contracts: Define validation rules in YAML following the Open Data Contract Standard.
• Flexible Filtering: Filter conditions with wildcard pattern matching, ideal for incremental validation of new data.
• Rich Reporting: Detailed summaries, row-level failure analysis, saveable reports, and a chainable ValidationResult API.
• DQ Metrics & OpenTelemetry: Row-level pass rates per table, dimension and check, saved to dq_metrics.json or exported as OTLP metrics, traces and logs.
• No Silent Gaps: Unimplemented or unrecognised checks surface as ERROR, not quietly skipped, so nothing slips through the cracks.

Installation

pip install vowl

Or install from source:

pip install git+https://github.com/govtech-data-practice/vowl.git

Optional extras are available: vowl[spark], vowl[otel], vowl[all]. vowl[otel] adds the OpenTelemetry SDK and OTLP exporter that export_otel() needs. vowl[spark] supports both classic Spark and Spark Connect (pyspark[connect], requires pyspark >= 3.4). On runtimes pinned to pyspark 3.0–3.3, use vowl[spark-classic] for classic Spark only (no Spark Connect). For local development, testing, and release workflow, see CONTRIBUTING.md.

Validate in 3 lines

import pandas as pd  # or any Narwhals-compatible DataFrame
from vowl import validate_data

df = pd.read_csv("tests/hdb_resale/HDBResaleWithErrors.csv")
result = validate_data("tests/hdb_resale/hdb_resale_simple.yaml", df=df)
result.display_full_report()

=== Data Quality Validation Results ===
   Contract Version:      v3.2.0
   Contract ID:           c11443ee-542f-4442-b28d-2d224342be37
   Schemas:               hdb_resale_prices

 OVERALL DATA QUALITY
   Overall:
     Checks Pass Rate:          7 / 9 (77.7%)

   hdb_resale_prices:
     Overall:
       Checks Pass Rate:          7 / 9 (77.7%)
       ERRORED Checks:            0
       Failed Rows (approximate): 14
     Single Table:
       Checks Pass Rate:          7 / 9 (77.7%)
       ERRORED Checks:            0
     Multi Table:
       Checks Pass Rate:          0 / 0 (N/A)
       ERRORED Checks:            0
       Failed Rows (approximate): N/A


 CHECK RESULTS
+------------------------------------+----------------------------------+-------------------+--------+----------+----------+--------+----------------+
| check_id                           | Target                           | tables_in_query   | status | operator | expected | actual | execution time |
+------------------------------------+----------------------------------+-------------------+--------+----------+----------+--------+----------------+
| Month                              | hdb_resale_prices.month          | hdb_resale_prices | FAILED | mustBe   | 0        | 2      | 18.60 ms       |
| floor_area_must_be_less_than_200   | hdb_resale_prices.floor_area_sqm | hdb_resale_prices | FAILED | mustBe   | 0        | 12     | 15.28 ms       |
+------------------------------------+----------------------------------+-------------------+--------+----------+----------+--------+----------------+
| flat_type_column_exists_check      | hdb_resale_prices.flat_type      | hdb_resale_prices | PASSED | mustBe   | 0        | 0      | 4.44 ms        |
| flat_type_enum_check               | hdb_resale_prices.flat_type      | hdb_resale_prices | PASSED | mustBe   | 0        | 0      | 8.24 ms        |
| floor_area_sqm_column_exists_check | hdb_resale_prices.floor_area_sqm | hdb_resale_prices | PASSED | mustBe   | 0        | 0      | 4.43 ms        |
| month_column_exists_check          | hdb_resale_prices.month          | hdb_resale_prices | PASSED | mustBe   | 0        | 0      | 4.81 ms        |
| month_logical_type_check           | hdb_resale_prices.month          | hdb_resale_prices | PASSED | mustBe   | 0        | 0      | 10.04 ms       |
| resale_price_column_exists_check   | hdb_resale_prices.resale_price   | hdb_resale_prices | PASSED | mustBe   | 0        | 0      | 4.32 ms        |
| resale_price_must_not_exceed_2m    | hdb_resale_prices.resale_price   | hdb_resale_prices | PASSED | mustBe   | 0        | 0      | 14.53 ms       |
+------------------------------------+----------------------------------+-------------------+--------+----------+----------+--------+----------------+
Total Execution:       84.69 ms

=== Failed Checks and Rows (up to 5 row(s) per failed check) ===

  hdb_resale_prices
    Single checks

      [Month]
        Operator:   mustBe
        Expected:   0
        Actual:     2
        Target:   hdb_resale_prices.month
        Details:  Based on ISO 8601 | YYYY-MM
        Rule:     SELECT COUNT(*) FROM "hdb_resale_prices" WHERE NOT REGEXP_MATCHES(TRY_CAST(month AS TEXT), '^[0-9]{4}-(0[1-9]|1[0-2])$')
        Rows shown: 2 of 2
+----------+--------+-----------+-------+--------------+--------------+----------------+---------------+---------------------+--------------------+--------------+
| month    | town   | flat_type | block | street_name  | storey_range | floor_area_sqm | flat_model    | lease_commence_date | remaining_lease    | resale_price |
+----------+--------+-----------+-------+--------------+--------------+----------------+---------------+---------------------+--------------------+--------------+
| 2017-jan | BEDOK  | 5 ROOM    | 21    | CHAI CHEE RD | 07 TO 09     | 130.0          | Adjoined flat | 1972                | 54 years 06 months | 530000.0     |
| 2017-jan | BISHAN | 3 ROOM    | 105   | BISHAN ST 12 | 04 TO 06     | 4.0            | Simplified    | 1985                | 67 years 11 months | 395000.0     |
+----------+--------+-----------+-------+--------------+--------------+----------------+---------------+---------------------+--------------------+--------------+

      [floor_area_must_be_less_than_200]
        Operator:   mustBe
        Expected:   0
        Actual:     12
        Target:   hdb_resale_prices.floor_area_sqm
        Details:  Check failed: expected mustBe 0, got 12
        Rule:     SELECT COUNT(*) FROM "hdb_resale_prices" WHERE TRY_CAST(floor_area_sqm AS BIGINT) >= 200
        Rows shown: 5 of 12
+---------+-----------------+-----------+-------+---------------------+--------------+----------------+--------------------+---------------------+--------------------+--------------+
| month   | town            | flat_type | block | street_name         | storey_range | floor_area_sqm | flat_model         | lease_commence_date | remaining_lease    | resale_price |
+---------+-----------------+-----------+-------+---------------------+--------------+----------------+--------------------+---------------------+--------------------+--------------+
| 2017-06 | KALLANG/WHAMPOA | 3 ROOM    | 38    | JLN BAHAGIA         | 01 TO 03     | 215.0          | Terrace            | 1972                | 54 years 01 month  | 830000.0     |
| 2017-09 | CHOA CHU KANG   | EXECUTIVE | 641   | CHOA CHU KANG ST 64 | 16 TO 18     | 215.0          | Premium Maisonette | 1998                | 79 years 04 months | 888000.0     |
| 2017-12 | KALLANG/WHAMPOA | 3 ROOM    | 65    | JLN MA'MOR          | 01 TO 03     | 249.0          | Terrace            | 1972                | 53 years 07 months | 1053888.0    |
| 2018-01 | CHOA CHU KANG   | EXECUTIVE | 639   | CHOA CHU KANG ST 64 | 10 TO 12     | 215.0          | Premium Maisonette | 1998                | 79 years           | 900000.0     |
| 2018-09 | KALLANG/WHAMPOA | 3 ROOM    | 41    | JLN BAHAGIA         | 01 TO 03     | 237.0          | Terrace            | 1972                | 52 years 10 months | 1185000.0    |
+---------+-----------------+-----------+-------+---------------------+--------------+----------------+--------------------+---------------------+--------------------+--------------+

Next, Part 2 · Core Concepts explains what a contract looks like and what you get back. If you'd rather see runnable code, jump to Part 3 · Usage Patterns for PySpark, Ibis connections, multi-source validation, and more.

Part 2 · Core Concepts

This section walks through the four ideas you need: how you declare rules (Data Contracts), what vowl generates for you (Auto-Generated Checks), the declarative metrics you can add (Library Metrics and Format Checks), and what you get back (Validation Results). It closes with the Architecture.

Data Contracts

Instead of writing validation logic in Python, you declare it in a YAML file following the Open Data Contract Standard (ODCS). This separates your rules from your code, making them easier to manage, version, and share.

Example hdb_resale_simple.yaml:

kind: DataContract
apiVersion: v3.2.0
version: 1.0.0
id: c11443ee-542f-4442-b28d-2d224342be37
name: HDB Resale Flat Prices
schema:
  - name: hdb_resale_prices # This becomes the table name in your SQL queries
    properties:
      # --- SQL Check: regex-based format validation ---
      - name: month
        logicalType: string
        quality:
          - type: sql
            name: Month
            description: Based on ISO 8601 | YYYY-MM
            mustBe: 0
            query: |-
              SELECT COUNT(*)
              FROM "hdb_resale_prices"
              WHERE CAST(month AS TEXT) !~ '^[0-9]{4}-(0[1-9]|1[0-2])$';
            dimension: conformity

      # --- Generated Check: allowed values from enum ---
      - name: flat_type
        enum:
          - value: 1 ROOM
          - value: 2 ROOM
          - value: 3 ROOM
          - value: 4 ROOM
          - value: 5 ROOM
          - value: EXECUTIVE
          - value: MULTI-GENERATION

      # --- SQL Check: business rule ---
      - name: floor_area_sqm
        quality:
          - name: floor_area_must_be_less_than_200
            type: sql
            query: SELECT COUNT(*) FROM "hdb_resale_prices" WHERE floor_area_sqm >= 200
            mustBe: 0
            dimension: consistency

      # --- SQL Check: resale price cap ---
      - name: resale_price
        quality:
          - name: resale_price_must_not_exceed_2m
            type: sql
            query: SELECT COUNT(*) FROM "hdb_resale_prices" WHERE resale_price > 2000000
            mustBe: 0
            dimension: consistency

Auto-Generated Checks

You don't have to write every check by hand. When a contract is loaded, vowl automatically derives checks from your column metadata, so simply declaring logicalType, required: true, unique: true, and similar gives you validation for free. These auto-generated checks run before any explicit quality checks you've authored.

The check types currently generated:

For example, a property like this:

- name: block
    logicalType: string
    logicalTypeOptions:
        maxLength: 10
    required: true

produces three generated checks: a column-exists check, a maxLength option check, and a required (no-NULL) check. Because string does not currently generate a SQL cast-based type check, the logicalType entry above contributes metadata for option checks rather than a standalone type-validation query. If you use integer, number, boolean, date, timestamp, or time, vowl also generates a logicalType SQL check automatically. You only need to define extra quality entries when you want custom business rules beyond the contract metadata.

Relationships (foreign keys)

A relationships entry of type foreignKey says every non-NULL value in a column must exist in another table's column. vowl runs this as a real check, so you get referential integrity from metadata without writing the cross-table SQL by hand (compare the employee_id_exists_in_master_list example under Residues below).

Declare the relationship on the property that holds the key. The to target uses shorthand <object>.<property> notation, resolved by property name:

schema:
  - name: demo_employee_list # the master list (referenced side)
    properties:
      - name: employee_id
        logicalType: string
        primaryKey: true
  - name: demo_employee_payroll # references the master list
    properties:
      - name: employee_id
        logicalType: string
        relationships:
          - type: foreignKey
            to: demo_employee_list.employee_id

This generates a check named demo_employee_payroll_employee_id_to_demo_employee_list_employee_id_foreign_key_check. A payroll row whose employee_id is NULL is skipped, so an optional foreign key is allowed. A value fails only when it is present but has no match in the master list. Because failed rows carry only the payroll table's columns, they appear in the demo_employee_payroll annotated table rather than a separate residue.

Need composite keys, cross-file references, or cross-source joins? See Data Contracts: Relationships (Foreign Keys).

Library Metrics (type: library)

Instead of writing SQL by hand, you can declare common data quality metrics using type: library in your quality blocks. vowl auto-generates the appropriate SQL at runtime.

Column-level metrics (under a property's quality):

Table-level metrics (under a schema's quality):

All library metrics support unit: "percent" to return the result as a percentage of total rows instead of an absolute count. They also accept any of the standard check operators (mustBe, mustBeGreaterThan, etc.).

Example:

properties:
  - name: town
    quality:
      - type: library
        metric: nullValues
        mustBe: 0
        dimension: completeness

  - name: flat_type
    quality:
      - type: library
        metric: invalidValues
        mustBe: 0
        dimension: conformity
        arguments:
          validValues:
            - 3 ROOM
            - 4 ROOM
            - 5 ROOM
            - EXECUTIVE

quality:
  - type: library
    metric: rowCount
    mustBeGreaterThan: 0
    dimension: completeness

  - type: library
    metric: duplicateValues
    mustBe: 0
    dimension: uniqueness
    arguments:
      properties:
        - month
        - block
        - street_name

Format Checks

The logicalTypeOptions.format key validates that column values conform to a declared format. The check generated depends on the column's logicalType. In short:

• Integer formats (i8…u64) — range-check a fixed-width integer type.
• String formats (uuid, email, ipv4, ipv6, hostname, uri) — match a built-in regex.
• Date/timestamp/time formats — a JDK DateTimeFormatter pattern (e.g. yyyy-MM-dd) is converted to a regex and matched against string-cast values.
• Number formats (f32, f64) — recognised but metadata-only (no check).

- name: age
  logicalType: integer
  logicalTypeOptions:
    format: u8 # 0 – 255

- name: request_id
  logicalType: string
  logicalTypeOptions:
    format: uuid

- name: created_at
  logicalType: timestamp
  logicalTypeOptions:
    format: "yyyy-MM-dd'T'HH:mm:ss.SSSXXX"

Validation Results

The validate_data function returns a powerful ValidationResult object that provides multiple ways to interact with your validation results.

Core Methods

Row Quality

get_dq_metrics_df() returns the row counts. The summary does not show them. It shows Failed Rows (approximate), a sum of the scalar counts of the failed checks. For each table, get_dq_metrics_df() gives how many rows failed at least one row-level check, and the share that passed. A row that fails two checks counts once, and every copy of a duplicated row counts. vowl counts inside your data source where it can, so the numbers do not depend on max_failed_rows, and each one carries an approximate flag for the cases where it could be off. A check whose failed rows vowl can't attribute to the table is left out and counted in checks_not_attributable. See When counting and annotating differ for how the row counts relate to annotated output.

result.get_dq_metrics_df()                  # one row per table
result.get_dq_metrics_df(by="dimension")    # one row per table and dimension
result.get_dq_metrics_df(by="check")        # which checks are row-level, and how they were counted

By default only failed checks add rows. A check that passed within its tolerance (for example 50 rows under mustBeLessThan: 100) adds nothing. Set ValidationConfig(fetch_tolerated_rows=True) to fetch its rows too, so they count in the row counts and appear in every output, marked as tolerated. See Tolerated rows.

Checks that are plain row filters are counted inside your data source wherever it supports it. For other checks, such as ones with DISTINCT or a join, vowl attributes each failed row to the source table, so a row that fails several checks counts once. On DuckDB, SQLite, Spark, Databricks and PostgreSQL the data source does this. Elsewhere vowl downloads the table and holds it in memory, about 1 to 1.5 GiB for 1 million rows at 6 columns. Annotated output reuses the same download. This work runs only when you ask for DQ metrics or annotated output, which includes the default save() and export_otel(). print_summary(), get_check_results_df() and save(outputs=["consolidated_query_outputs"]) use the scalar counts only.

When a count could be off, for example because a check ended in ERROR and may hide failed rows, its approximate flag is set. The same flag is exported as the vowl.<level>.row.approximate gauges, so you can alert on it. See Approximate row counts.

See How Attributed Rows Work for how each failed row is matched to its table, and Counting Mechanisms.

Annotated Output

get_annotated_output() returns the full in-scope table with a check_info column that annotates which rows failed which checks. Passing rows have null in the check_info column. This is useful when you need to see failures in the context of the full dataset rather than just the isolated failed rows.

New to this? Check Results explains failed rows, annotated output and residues in plain language, with a small worked example.

It returns a nested dict with two reserved keys:

• "annotated": a {schema: table} dict where each table is your full in-scope data plus a check_info column. Every original row is present; check_info is null for rows that passed everything and holds a JSON array of objects describing the failing check(s) otherwise.
• "residues": failed rows for checks that cannot be merged onto a single table (checks whose failed rows lack the table's primary key or, when it declares none, don't have exactly its columns, such as column-subset checks). Checks that return one number, such as an average, a sum, a minimum or a maximum, produce none. Most single-table contracts produce none. Residues are per-check (one entry per non-mergeable check, keyed "<schema>.<column>::<check_name>", or "<schema>::<check_name>" for a schema-level check) and carry the same check_info column as the annotated tables (a single-element JSON array, shaped by the same preset) plus tables_in_query, so everything get_annotated_output() returns is read the same way. (A cross-table check can merge onto its home schema if you shape its row query to project only that schema's columns: see Annotating the failed rows of a cross-table check.)

The check_info parameter ("names" default, "summary", or "full") shapes each array element. Every preset emits a JSON array of objects so consumers parse uniformly via item["check_name"]; they differ only in how many keys each object carries:

• "names" — [{"check_name": ...}, ...] (just the failing check name(s)).
• "summary" — [{"check_name", "dimension", "tags", "target"}, ...].
• "full" — [<full check_definition> + "check_name" + "target", ...].

import json

result = validate_data("contract.yaml", df=df)
output = result.get_annotated_output(check_info="summary")

annotated = output["annotated"]["hdb_resale_prices"].to_pandas()
# columns: <original columns> + check_info
# output["residues"] — cross-table or non-mergeable failures (empty for single-table contracts)

# check_info is a JSON string; parse it to read each failing check's dimension/tags/target.
failing = annotated[annotated["check_info"].notna()].iloc[0]
print(json.loads(failing["check_info"]))
# [{"check_name": "Year", "dimension": "conformity",
#   "tags": ["SG-DRM v5.0"], "target": "hdb_resale_prices.lease_commence_date"}]

Annotated Table

# Sort so flagged rows surface first; passing rows (check_info = null) sink to the bottom.
flagged_first = annotated.sort_values("check_info", na_position="last").reset_index(drop=True)

flagged_first[["month", "town", "block", "floor_area_sqm",
               "lease_commence_date", "check_info"]]

clean = annotated[annotated["check_info"].isna()].drop(columns=["check_info"])
# 201,861 clean rows of 201,879, ready for downstream use — no join back to the source needed

A check whose failed rows can't be matched to rows of one table becomes a residue: its rows lack the table's declared primary key or, when the table declares none, don't have exactly the table's columns. Column-subset checks, such as one that returns only distinct values, are the usual case. A cross-table check merges when its failed rows carry the table's primary key, or when its row query projects only its home schema's columns. See Annotating the failed rows of a cross-table check. Checks that return one number, such as an average, a sum, a minimum or a maximum, have no failed rows and appear only in the summary. Residues are per-check, one entry per non-mergeable check, keyed "<schema>.<column>::<check_name>" ("<schema>::<check_name>" for a schema-level check). Each carries its own failed rows plus the same check_info column the annotated tables use (a single-element JSON array) and tables_in_query:

Residues

- name: phone_numbers_missing_from_master_list
  type: sql
  query: >-
    SELECT COUNT(*) FROM (
      SELECT DISTINCT payroll.phone_number
      FROM demo_employee_payroll payroll
      LEFT JOIN demo_employee_list ref ON payroll.phone_number = ref.phone_number
      WHERE payroll.phone_number IS NOT NULL AND ref.phone_number IS NULL
    ) AS missing_numbers
  mustBe: 0

output = result.get_annotated_output()

print("Residue keys:", list(output["residues"].keys()))
for key, residue in output["residues"].items():
    df = residue.to_pandas()
    print(f"\nResidue '{key}': {len(df)} failed row(s)")
    print(df)

For the full eligibility rules and worked examples of each non-mergeable category, see Where each failed check ends up. The Basic Tutorial notebook walks through these examples end-to-end.

save() always writes <prefix>_check_results.csv and <prefix>_summary.json. The outputs argument lists the other files to write:

# Every output but "failed_query_outputs". This is the default.
result.save()

# Shape the check_info column: "names" (default), "summary", or "full"
result.save(outputs=["annotated_table"], check_info="summary")

# Only the grouped failed-rows CSVs
result.save(outputs=["consolidated_query_outputs"])

Cost: "annotated_table" and "dq_metrics" attribute failed rows to each table and may download it, which can be slow on a large table. They share that work, so writing both costs no more than writing one. The other outputs do nothing to the rows. "all_query_outputs" runs the row query of each check that passed.

output_mode is deprecated. It still works with a FutureWarning and will be removed in a future release. See Deprecated output_mode.

You can also set the outputs globally via ValidationConfig (see Run Settings):

from vowl import ValidationConfig, validate_data

config = ValidationConfig(outputs=["annotated_table", "dq_metrics"], annotated_check_info="summary")
result = validate_data("contract.yaml", df=df, config=config)
result.save()  # uses the configured outputs and check_info

save() can also write straight to cloud storage. Pass a URI instead of a folder:

result.save("s3://my-bucket/dq-results/run-1/")

It works for s3://, gs://, abfs:// and hdfs:// using the filesystems built into pyarrow, so there's nothing extra to install. Credentials come from the usual place for each cloud, such as environment variables, ~/.aws, or an IAM role. To use a custom endpoint or explicit credentials, pass filesystem=. See Saving Results to Cloud Storage.

OpenTelemetry Export

export_otel() sends a run's results to an OpenTelemetry Collector or any monitoring tool that accepts OTLP. Install the extra first with pip install 'vowl[otel]'.

result = validate_data("contract.yaml", df=df)

run_id = result.export_otel(
    endpoint="http://localhost:4317",
    service_name="orders-dq",
    custom_attributes={"deployment.environment": "prod"},
)

vowl sends three signals by default:

• Metrics: the DQ metrics at check, dimension, schema and run level, for dashboards and alerts.
• Traces: one vowl.validate span per run, with a vowl.check span per check and a vowl.save span for each save() made before the export.
• Logs: one record per check that failed or ended in ERROR.

Nothing is re-run against the data. The returned run_id matches the one in dq_metrics.json, so you can link a saved run to its telemetry. In a deployed pipeline, leave out endpoint and set the standard OTEL_EXPORTER_OTLP_* environment variables instead.

To try it locally, the repo ships a Collector, Prometheus, Tempo, Loki and Grafana stack with a ready-made DQ dashboard. Run make otel-up, then make otel-add-vowl-runs to send sample runs (needs Docker). See the local stack guide.

See Exporting to OpenTelemetry for every parameter, the span and log attributes, and how to use your own OTel providers.

Architecture

vowl has a modular architecture built around Ibis as the universal query layer.

┌─────────────────────────────────────────────────────────────────────────────┐
│                              validate_data()                                │
│                           (Main Entry Point)                                │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           DataSourceMapper                                  │
│              (Auto-detects input type → creates adapter)                    │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
          ┌──────────────────────────┼──────────────────────────┐
          ▼                          ▼                          ▼
┌──────────────────┐      ┌──────────────────┐      ┌──────────────────┐
│   IbisAdapter    │      │ MultiSourceAdapter│      │  Custom Adapter  │
│                  │      │                  │      │                  │
│ • pandas/Polars  │      │ • Cross-database │      │ • Extend         │
│ • PySpark        │      │   validation     │      │   BaseAdapter    │
│ • PostgreSQL     │      │ • Data federation│      │                  │
│ • Snowflake      │      │                  │      │                  │
│ • BigQuery       │      │                  │      │                  │
│ • 20+ backends   │      │                  │      │                  │
└──────────────────┘      └──────────────────┘      └──────────────────┘
          │                          │                          │
          └──────────────────────────┼──────────────────────────┘
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                              Executors                                      │
│                                                                             │
│  ┌─────────────────┐  ┌─────────────────────┐  ┌─────────────────────┐     │
│  │  IbisSQLExecutor│  │MultiSourceSQLExecutor│  │  Custom Executor   │     │
│  │                 │  │                     │  │                     │     │
│  │ Runs SQL checks │  │ Mode 1: delegate to │  │ Extend BaseExecutor │     │
│  │ via Ibis        │  │ backend (same conn) │  │ or SQLExecutor      │     │
│  │ (server-side)   │  │ Mode 2: materialise │  │                     │     │
│  │                 │  │ to DuckDB via Arrow │  │                     │     │
│  └─────────────────┘  └─────────────────────┘  └─────────────────────┘     │
└─────────────────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           ValidationResult                                  │
│                                                                             │
│  • Per-check failed rows with check_id & tables_in_query columns            │
│  • Detailed check results and metrics                                       │
│  • Export to CSV/JSON                                                       │
└─────────────────────────────────────────────────────────────────────────────┘

Key Components

Part 3 · Usage Patterns

Interactive demo: Try the example notebooks for a hands-on walkthrough of the examples below — start with the Basic Tutorial.

The patterns are grouped from most common to most advanced:

• Common sources — point vowl at the data you already have: a local DataFrame, PySpark, or any of 20+ Ibis backends.
• Filtering & cross-source — validate a subset of rows, or join across multiple source systems — optionally through DuckDB ATTACH.
• Advanced & extending — drive connections from servers in the contract, build custom adapters/executors, or load contracts from Git and S3.

Common sources

Local DataFrame (Pandas/Polars)

import pandas as pd
from vowl import validate_data

df = pd.read_csv("data.csv")
result = validate_data("contract.yaml", df=df)
result.display_full_report()

PySpark

from pyspark.sql import SparkSession
from vowl import validate_data

# Create SparkSession (user-managed)
spark = SparkSession.builder.appName("vowl").getOrCreate()

try:
    spark_df = spark.read.table("my_table")
    result = validate_data("contract.yaml", df=spark_df)
    result.display_full_report()
finally:
    # User is responsible for stopping the SparkSession
    spark.stop()

Note: The library does not manage the SparkSession lifecycle. You must create and stop it yourself. This is by design - SparkSession is a heavy, application-owned resource with specific configuration requirements.

Ibis Connections (20+ Backends)

# Ibis supports: Amazon Athena, BigQuery, ClickHouse, Dask, Databricks, DataFusion,
# Druid, DuckDB, Exasol, Flink, Impala, MSSQL, MySQL, Oracle, pandas, Polars,
# PostgreSQL, PySpark, RisingWave, SingleStoreDB, Snowflake, SQLite, Trino, ...
# Find out more at https://github.com/ibis-project/ibis

import ibis
from vowl import validate_data
from vowl.adapters import IbisAdapter

con = ibis.postgres.connect(...)  # Redshift can be supported via Postgres connections too

result = validate_data("contract.yaml", adapter=IbisAdapter(con))
result.display_full_report()

For MySQL, select the database when you create the connection, for example via ibis.mysql.connect(..., database="my_db") or a connection URI that already includes the database name. vowl does not issue USE database during validation; it runs read-only SELECT queries against the active database on the existing connection. If you need to avoid relying on the connection's default database, use qualified table names such as my_db.my_table in your contract queries.

Concurrent Checks (PooledAdapter)

When a contract has many independent checks and the backend can serve several queries at once, run them concurrently by wrapping a connection factory in a PooledAdapter. It keeps a thread-safe pool of connections (one per worker) and dispatches checks across them; the verdicts are identical to a sequential run.

import ibis
from vowl import validate_data
from vowl.adapters import IbisAdapter, PooledAdapter

# factory: returns a fresh adapter (new connection) on each call. Called once
# per pooled connection, so the table must be available on every connection.
def make_adapter():
    con = ibis.duckdb.connect("my_db.duckdb")
    return IbisAdapter(con)

pooled = PooledAdapter(factory=make_adapter, max_concurrency=4)

# Pass the pool like any other adapter: adapter=pooled for every schema,
# or adapters={"my_table": pooled, ...} to give each schema its own.
result = validate_data("contract.yaml", adapter=pooled)
result.display_full_report()

max_concurrency caps the checks in flight, and so the connections open, for the whole run. Schemas given the same pool share that cap and run side by side. A join between tables on one pool runs in the database on one of the pool's connections. A join across two pools, or between a pool and another adapter, is copied to a local DuckDB. See PooledAdapter.

The pool keeps its adapters after the run, so you can pass it to another validate_data call. Call pooled.cleanup() when you are done with it. This drops the pooled adapters and calls cleanup() on each one that defines it. It does not close Ibis connections, so close those yourself if your factory opens ones that need closing.

Filtering & cross-source

Explicit Adapter with Filter Conditions

from vowl import validate_data
from vowl.adapters import IbisAdapter
from datetime import datetime, timedelta
import ibis

date_limit = (datetime.today() - timedelta(days=7)).strftime("%Y-%m-%d")
con = ibis.postgres.connect(...)

# Using dict for filter conditions with wildcard patterns
# Wildcards use glob-style matching: * (any chars), ? (single char), [seq] (char in seq)
adapter = IbisAdapter(
    con,
    filter_conditions={
        # Exact match
        "TableA": {
            "field": "date_dt",
            "operator": ">=",
            "value": date_limit
        },
        # Wildcard: matches employees, emp_history, emp_details, etc.
        "emp*": {
            "field": "date_dt",
            "operator": ">=",
            "value": date_limit
        },
        # Wildcard: matches orders_archive, customers_archive, etc.
        "*_archive": {
            "field": "is_deleted",
            "operator": "=",
            "value": False
        },
        # Apply to ALL tables
        "*": {
            "field": "tenant_id",
            "operator": "=",
            "value": 123
        },
    }
)
# Note: If multiple patterns match a table, conditions are combined with AND

# Multiple filter conditions on same table (combined with AND)
adapter = IbisAdapter(
    con,
    filter_conditions={
        "TableA": [
            {"field": "date_dt", "operator": ">=", "value": date_limit},
            {"field": "status", "operator": "=", "value": "active"},
        ]
    }
)

result = validate_data("contract.yaml", adapter=adapter)
result.display_full_report()

Multi-Source Validation

There are two ways to validate across tables in different databases.

Option A: DuckDB ATTACH (streams rows per check, no up-front download)

import ibis
from vowl import validate_data
from vowl.adapters import IbisAdapter

con = ibis.duckdb.connect()

# Attach multiple remote databases
con.raw_sql("ATTACH 'postgresql://user:pass@host:5432/salesdb' AS pg_sales (TYPE postgres, READ_ONLY)")  # trufflehog:ignore
con.raw_sql("ATTACH 'sqlite:///path/to/users.db' AS sqlite_users (TYPE sqlite, READ_ONLY)")

# Switch back to local DuckDB so views live in memory
con.raw_sql("USE memory")

# Create views as prefix-free shortcuts to the attached tables
con.raw_sql("CREATE VIEW transactions AS SELECT * FROM pg_sales.transactions")
con.raw_sql("CREATE VIEW users AS SELECT * FROM sqlite_users.users")

# Now vowl (and your contract queries) can reference tables without alias prefixes
result = validate_data("contract.yaml", adapter=IbisAdapter(con))
result.display_full_report()

Note: A view is a saved query, not a copy, and it gives your contracts clean, prefix-free table names. DuckDB still runs on your machine, so each check streams the rows it reads from the attached databases, then discards them. For PostgreSQL and MySQL, simple filters are sent to the source. Joins and aggregates such as COUNT(*) run inside DuckDB, so the rows they read cross the network, and a table used by several checks is read once per check. DuckDB ATTACH supports PostgreSQL, MySQL, and SQLite.

Option B: Multi-Source Adapters (downloads tables for cross-table checks)

from vowl import validate_data
from vowl.adapters import IbisAdapter
import ibis

con_a = ibis.postgres.connect(...)
con_b = ibis.sqlite.connect(...)

adapters = {
    "table_a": IbisAdapter(con_a),
    "table_b": IbisAdapter(con_b)
}

result = validate_data("contract.yaml", adapters=adapters)
result.display_full_report()

Why this exists: A fallback for backends that DuckDB ATTACH does not support (e.g. Snowflake, BigQuery, Databricks, Oracle, MSSQL). Single-table checks run inside each table's own database. For a cross-table check across different connections, the MultiSourceAdapter downloads each table it reads in full (after filter conditions) via Arrow into a local DuckDB instance, so prefer ATTACH for large tables when your sources support it. DuckDB ATTACH only supports PostgreSQL, MySQL, and SQLite. It cannot be used as a general-purpose multi-source strategy because of namespace, credential, and filter limitations. Both options share a known dark pattern: SQL checks can reference tables not declared in the contract's schema block. vowl reads an undeclared table through the connection of the schema the check sits under, whether the check reads it alone or joins it to other tables, and the check runs whenever that connection can see the table.

Compatibility Mode (DuckDB ATTACH)

import ibis
from vowl import validate_data
from vowl.adapters import IbisAdapter

# ATTACH lets DuckDB query your remote database directly.
# Data is streamed on demand, not materialised locally.
# All SQL is evaluated by DuckDB, so dialect differences are eliminated.
con = ibis.duckdb.connect()
con.raw_sql("ATTACH 'postgresql://user:pass@host:5432/mydb' AS pg (TYPE postgres, READ_ONLY)")  # trufflehog:ignore
con.raw_sql("USE pg")  # Allows querying tables without the pg. alias

result = validate_data("contract.yaml", adapter=IbisAdapter(con))
result.display_full_report()

When to use this: Your remote backend doesn't support a SQL feature that a check needs, or you want a single local engine for reproducible results regardless of the source database. DuckDB ATTACH supports PostgreSQL, MySQL, and SQLite.

Advanced & extending

Using Servers Defined in Data Contract

from vowl import validate_data
from vowl.contracts import Contract
from vowl.adapters import IbisAdapter
import ibis

# Load the contract and get server configuration
contract = Contract.load("contract.yaml")
server = contract.get_server("my-postgres-server")  # Match by server name
# Or: contract.get_server("uat")        # falls back to matching by environment
# Or: contract.get_server()             # returns the first server

# Create connection based on server config
con = ibis.postgres.connect(
    host=server["server"],
    port=server.get("port", 5432),
    database=server.get("database", ""),
)

# Create adapter and validate
adapter = IbisAdapter(con)
result = validate_data("contract.yaml", adapter=adapter)
result.display_full_report()

Custom Adapters and Executors

BaseAdapter, BaseExecutor, and SQLExecutor are intended as boilerplate extension points for teams building custom integrations. The typical pattern is to wrap an existing adapter, register custom executors, and then add backend-specific behavior incrementally.

from typing import Optional

import ibis

from vowl.adapters import BaseAdapter, IbisAdapter
from vowl.executors import BaseExecutor, SQLExecutor


class CustomAdapter(BaseAdapter):
    def __init__(self, con, **kwargs):
        super().__init__(executors={
            "sql": CustomSQLExecutor,
            "xxx": CustomEngineExecutor,
        })
        self._wrapped = IbisAdapter(con, **kwargs)

    def get_connection(self):
        return self._wrapped.get_connection()

    @property
    def filter_conditions(self):
        return self._wrapped.filter_conditions

    def test_connection(self, table_name: str) -> Optional[str]:
        return self._wrapped.test_connection(table_name)


class CustomEngineExecutor(BaseExecutor):
    ...


class CustomSQLExecutor(SQLExecutor):
    ...


con = ibis.duckdb.connect()
adapter = CustomAdapter(con)

executors = adapter.get_executors()
assert "sql" in executors

validate_data accepts any BaseAdapter through adapter= or adapters=, including IbisAdapter, PooledAdapter and your own subclasses. A custom adapter runs its checks through the executors it registers. A cross-table check runs in the database only if is_compatible_with says the adapters can share a query. The default returns False, so vowl copies the tables to a local DuckDB, which needs export_table_as_arrow. When the tables in such a join have different filter conditions, the adapter also needs with_filter_conditions, or vowl copies the tables.

Loading Contracts from Remote Sources (Git/S3)

Contracts don't have to live on local disk — validate_data accepts GitHub/GitLab URLs and S3 URIs directly.

Git (GitHub/GitLab):

from vowl import validate_data

# GitHub - blob URL (auto-converted to raw)
result = validate_data(
    "https://github.com/org/repo/blob/main/contracts/my_contract.yaml",
    df=df
)
result.display_full_report()

# GitHub - raw URL
result = validate_data(
    "https://raw.githubusercontent.com/org/repo/main/contracts/my_contract.yaml",
    df=df
)
result.display_full_report()

# GitLab - blob URL (auto-converted to raw)
result = validate_data(
    "https://gitlab.com/org/repo/-/blob/main/contracts/my_contract.yaml",
    df=df
)
result.display_full_report()

# Note: `requests` is included in base install.

S3:

from vowl import validate_data

# S3 URI format
result = validate_data("s3://my-bucket/contracts/my_contract.yaml", df=df)
result.display_full_report()

# Note: `boto3` is not included in the base install.
# Install it with: pip install vowl[all]  or  pip install boto3
# Uses default AWS credentials (environment variables, ~/.aws/credentials, IAM role, etc.)

Roadmap

Completed

Planned

Status

Active. Under ongoing development, with issues and PRs monitored.

Contributing

We welcome contributions! Please see CONTRIBUTING.md for guidelines on how to get started.

Owner

Maintained by GovTech's Data Practice (govtech-data-practice).

Security

See SECURITY.md for how to report a vulnerability and which versions are supported.

License

This project is licensed under the MIT License.