Veridelta

Semantic diffing for data pipelines. Define expected variance, validate intentional changes, and detect regressions with confidence.

Veridelta is a declarative data comparison engine powered by Polars. It applies semantic and mathematical rules to filter out expected pipeline noise, such as floating-point drift or formatting changes, and programmatically isolates true data regressions at scale.

Documentation & API Reference

Installation

uv add veridelta

Warehouse and lakehouse drivers are optional extras (snowflake, databricks, delta, iceberg, or all):

uv add 'veridelta[snowflake]'

YAML examples, routing rules, and time travel live in the configuration guide.

Usage

Veridelta supports configuration-driven CLI execution and programmatic Python API integration.

CLI Configuration (YAML)

Declare comparison rules and tolerances via YAML:

# veridelta.yaml
primary_keys: ["transaction_id"]
schema_mode: "intersection"

source:
  path: "legacy_system.parquet"
  format: "parquet"

target:
  path: "modern_system.parquet"
  format: "parquet"

rules:
  - column_names: ["grand_total"]
    relative_tolerance: 0.01

  - column_names: ["contact_number"]
    regex_replace: {"[^0-9]": ""}

Execute the comparison:

veridelta run --config veridelta.yaml

Python API

Integrate directly into programmatic workflows (e.g., Airflow, Databricks). The DiffEngine natively consumes Polars LazyFrame objects for memory-safe, big-data execution.

import polars as pl
from veridelta import DiffConfig, DiffRule, DiffEngine

source_lazy = pl.scan_parquet("legacy_data.parquet")
target_lazy = pl.scan_parquet("modern_data.parquet")

config = DiffConfig(
    primary_keys=["user_id"],
    default_treat_null_as_equal=True,
    rules=[
        DiffRule(
            pattern="^AMT_.*",
            absolute_tolerance=0.05
        )
    ]
)

engine = DiffEngine(config, source_lazy, target_lazy)
summary = engine.run().summary

if not summary.is_match:
    print(f"Regression detected: {summary.changed_count} rows differ.")

Core Capabilities

Structural Alignment: Map legacy column names to modern schemas automatically.
Semantic Normalization: Coerce string markers to nulls, standardize whitespace, and cast types dynamically before mathematical comparison.
Warehouse Pushdown: Compile comparison SQL for Snowflake and Databricks so diffs run in-warehouse (changed, added, and removed counts) without extracting full tables.
Lakehouse Native: Scan Delta Lake and Apache Iceberg tables as unevaluated Polars LazyFrames, including optional version and snapshot time travel.
Discrepancy Artifacts: Export isolated Parquet files detailing added, removed, and changed records for downstream auditing.

Roadmap

Upcoming work:

Advanced Heuristics: Fuzzy string matching and ML-driven schema mapping.
Reporting: Interactive HTML diff dashboards and CI/CD status checks.
Additional warehouses: BigQuery pushdown.

View Detailed Roadmap

License

Distributed under the Apache 2.0 License.

Metadata

Release files for veridelta 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for veridelta 0.8.0
File	Size	Uploaded
veridelta-0.8.0.tar.gz	396.8 kB	Sep 11, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for veridelta 0.8.0
File	Interpreter	ABI	Platform	Reset
veridelta-0.8.0-py3-none-any.whl 61.6 kB Sep 11, 2026	Python 3	none	any	Details

Total release size: 458.3 kB

Release files / veridelta-0.8.0.tar.gz

Download URL	veridelta-0.8.0.tar.gz
Size	396.8 kB
Tags	Source
SHA-256 checksum How to use checksums	`0e6ecb7917f6c42a93a0e7b801311dffe3a5b3865ea4b47e2bb55eaff9e6b86f`
BLAKE2b-256 checksum How to use checksums	`e923c181e9f2e9cdeef5486a7c6e50338534259b9cf4d839f0df6680b991cf8f`
Upload date	Sep 11, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / veridelta-0.8.0-py3-none-any.whl

Download URL	veridelta-0.8.0-py3-none-any.whl
Size	61.6 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`6bb9dc1f4ff55463a3a66c3c630c7e681df933ace965a021292598c36b08087f`
BLAKE2b-256 checksum How to use checksums	`0b1cf67eb6fd69524dfd5357ca93fc43657e434653a1568b5fb33c9aad17f4b8`
Upload date	Sep 11, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`