dqbench 1.0.0

The standard benchmark for data quality and validation tools

DQBench

The standard benchmark for data quality and validation tools.

The ImageNet of data quality — standardized benchmarks for validation tools.

Why DQBench?

Every data validation tool claims to be the best. But there's no standard way to compare them. DQBench fixes that with:

• Three difficulty tiers — basics, realistic, and adversarial
• Ground truth — every planted issue is documented with affected rows
• Fair scoring — recall AND precision matter (no gaming by flagging everything)
• One number — DQBench Score (0-100) for easy comparison
• 20-line integration — implement one method to benchmark any tool

Install

pip install dqbench

Quick Start

# Run with built-in GoldenCheck adapter
pip install goldencheck
dqbench run goldencheck

# Run with a custom adapter
dqbench run --adapter my_adapter.py

Tiers

Tier	Rows	Columns	Domain	Difficulty
1 — Basics	5,000	20	Customer DB	Obvious errors, baseline
2 — Realistic	50,000	30	E-commerce	Subtle issues + false positive traps
3 — Adversarial	100,000	50	Healthcare	Encoding traps, semantic errors, cross-column logic

Each tier has columns WITH planted issues and columns WITHOUT (false positive traps). Tools that flag clean columns lose precision points.

Scoring

Metric	Description
Recall	% of planted-issue columns detected
Precision	% of flagged columns that actually have issues
F1	Harmonic mean of recall and precision
FPR	Clean columns incorrectly flagged (WARNING/ERROR only)
DQBench Score	Tier1_F1 × 20% + Tier2_F1 × 40% + Tier3_F1 × 40%

Write Your Own Adapter

Implement one class to benchmark any tool:

from dqbench.adapters.base import DQBenchAdapter
from dqbench.models import DQBenchFinding
from pathlib import Path

class MyToolAdapter(DQBenchAdapter):
    @property
    def name(self) -> str:
        return "MyTool"

    @property
    def version(self) -> str:
        return "1.0.0"

    def validate(self, csv_path: Path) -> list[DQBenchFinding]:
        # Run your tool on the CSV
        # Return a list of DQBenchFinding objects
        return [
            DQBenchFinding(
                column="email",
                severity="error",      # "error", "warning", or "info"
                check="format",         # what kind of issue
                message="Invalid email format",
                confidence=0.9,         # optional, 0.0-1.0
            )
        ]

Then run:

dqbench run --adapter my_adapter.py

CLI Reference

Command	Description
dqbench run <adapter>	Run benchmark
dqbench run --adapter <path>	Run with custom adapter file
dqbench run <adapter> --tier 2	Run specific tier only
dqbench run <adapter> --json	JSON output
dqbench generate	Generate/cache datasets
dqbench generate --force	Regenerate datasets

Built-in Adapters

Adapter	Tool	Install
goldencheck	GoldenCheck	pip install goldencheck

Want to add your tool? See CONTRIBUTING.md.

Reproducibility

• Datasets are generated deterministically (random.seed(42), stdlib only)
• Canonical datasets committed as release artifacts
• Version-locked: published benchmark versions are immutable

License

MIT

---

From the maker of GoldenCheck and GoldenMatch.