DQBench

The standard benchmark for data quality and validation tools.

The ImageNet of data quality — standardized benchmarks for validation tools.

Why DQBench?

Every data validation tool claims to be the best. But there's no standard way to compare them. DQBench fixes that with:

• Three difficulty tiers — basics, realistic, and adversarial
• Ground truth — every planted issue is documented with affected rows
• Fair scoring — recall AND precision matter (no gaming by flagging everything)
• One number — DQBench Score (0-100) for easy comparison
• 20-line integration — implement one method to benchmark any tool

Install

pip install dqbench

Quick Start

# Run with built-in GoldenCheck adapter
pip install goldencheck
dqbench run goldencheck

# Run with a custom adapter
dqbench run --adapter my_adapter.py

Tiers

Each tier has columns WITH planted issues and columns WITHOUT (false positive traps). Tools that flag clean columns lose precision points.

Scoring

Write Your Own Adapter

Implement one class to benchmark any tool:

from dqbench.adapters.base import DQBenchAdapter
from dqbench.models import DQBenchFinding
from pathlib import Path

class MyToolAdapter(DQBenchAdapter):
    @property
    def name(self) -> str:
        return "MyTool"

    @property
    def version(self) -> str:
        return "1.0.0"

    def validate(self, csv_path: Path) -> list[DQBenchFinding]:
        # Run your tool on the CSV
        # Return a list of DQBenchFinding objects
        return [
            DQBenchFinding(
                column="email",
                severity="error",      # "error", "warning", or "info"
                check="format",         # what kind of issue
                message="Invalid email format",
                confidence=0.9,         # optional, 0.0-1.0
            )
        ]

Then run:

dqbench run --adapter my_adapter.py

CLI Reference

Built-in Adapters

Want to add your tool? See CONTRIBUTING.md.

Reproducibility

• Datasets are generated deterministically (random.seed(42), stdlib only)
• Canonical datasets committed as release artifacts
• Version-locked: published benchmark versions are immutable

License

MIT

From the maker of GoldenCheck and GoldenMatch.