anofox-tabular Python package

Python wrapper for the anofox-tabular DuckDB extension — data quality, PII detection, email/phone validation, anomaly detection, diffing, money, and VAT primitives.

Installation

pip install anofox-tabular

Optional extras for DataFrame support:

pip install "anofox-tabular[pandas]"   # adds pandas
pip install "anofox-tabular[polars]"   # adds polars
pip install "anofox-tabular[pandas,polars]"

Quick start

import anofox

# In-memory database (extension downloaded automatically)
with anofox.connect() as conn:
    # Email validation
    print(conn.execute("SELECT anofox_tab_email_is_valid('hi@example.com', 'regex')").fetchone())

# Or use a locally built extension
conn = anofox.connect(
    extension_path="/path/to/anofox_tabular.duckdb_extension"
)

Python-native API

import anofox
from anofox import validate, quality, pii, diff

conn = anofox.connect()

# ── Email validation ──────────────────────────────────────────────────
validate.email_is_valid(conn, "hi@example.com")                # True
validate.email_is_valid(conn, "hi@example.com", mode="dns")    # True (DNS checked)

import pandas as pd
df = pd.DataFrame({"email": ["a@b.com", "bad-email", "c@d.org"]})
result_df = validate.email_is_valid(conn, df, column="email")
# Returns DataFrame with added 'email_is_valid' column

# ── Phone validation ──────────────────────────────────────────────────
validate.phone_is_valid(conn, "+14155552671", region="US")     # True
validate.phone_format(conn, "+14155552671", "US", "INTERNATIONAL")

# ── Data quality ──────────────────────────────────────────────────────
conn.execute("CREATE TABLE orders AS SELECT * FROM read_parquet('orders.parquet')")

quality.volume(conn, "orders", min_rows=100)
# {"status": "pass", "min_rows": 100, ...}

quality.null_rate(conn, "orders", "amount", max_null_rate=0.05)
quality.distinct_count(conn, "orders", "status", min_distinct=2, max_distinct=10)
quality.schema_check(conn, "orders", ["id", "amount", "created_at"])

# ── High-level profile ────────────────────────────────────────────────
summary = conn.profile(df)   # returns pd.DataFrame with per-column metrics

# ── PII detection ─────────────────────────────────────────────────────
pii.pii_contains(conn, "Call me at +1-415-555-2671")  # True
pii.pii_detect(conn, "Email: test@example.com")        # [{"type": "EMAIL", ...}]
pii.pii_mask(conn, "test@example.com", strategy="redact")

scan_result = pii.pii_scan_table(conn, "orders")  # pd.DataFrame

# ── Diff ──────────────────────────────────────────────────────────────
# Table names or DataFrames both work
changes = diff.joindiff(conn, "orders_v1", "orders_v2", primary_keys="id")
changes = diff.joindiff(conn, df_before, df_after, primary_keys="id")
# Returns pd.DataFrame with diff_type: 'added', 'removed', 'changed', 'unchanged'

# ── Schema validation ─────────────────────────────────────────────────
from anofox.validate import EmailRule, PhoneRule

result = conn.validate(df, schema={
    "email": EmailRule(mode="dns"),
    "phone": PhoneRule(region="DE"),
})
print(result.passed)      # True / False
print(result.failures)    # pd.DataFrame of failed rows

Module overview

CLI

# Profile any file (colored table output)
anofox profile data.parquet
anofox profile data.csv --format json

# Quality checks (exit 0 = pass, exit 1 = fail)
anofox quality data.parquet --volume-min 1000
anofox quality data.csv --null-max 0.05 --column email

Supported formats: .parquet, .csv, .tsv, .json, .ndjson

pytest plugin

# Run with: pytest --anofox-check
import pytest

@pytest.mark.anofox_quality("orders", volume_min=100)
def test_orders_table_has_data(anofox_conn):
    ...

The anofox_conn session-scoped fixture is provided automatically. Tests skip if the extension is unavailable.

Extension resolution

The package resolves the extension binary in this order:

1. extension_path argument to connect() (path to a local binary)
2. ANOFOX_EXT_PATH environment variable
3. Cached binary in ~/.anofox/extensions/<duckdb version>/<platform>/
4. Download from the DuckDB community registry, then the S3 mirror (https://get.erpl.io), both laid out as <base>/v<duckdb version>/<platform>/anofox_tabular.duckdb_extension.gz (duckdb 1.5.6 downloads from v1.5.6/, duckdb 2.0.0 from v2.0.0/)

Downloads exist only for DuckDB releases. With a pre-release, development or post-release duckdb package (2.0.0rc1, 1.5.7.dev42, 1.5.6.post1) connect() raises anofox.UnsupportedDuckDBVersionError (a RuntimeError) before any download is attempted; install a released duckdb, or set ANOFOX_EXT_PATH / extension_path to an extension built for that DuckDB. The package declares duckdb>=1.0.0 without an upper bound: the extension is published for each supported DuckDB release line (currently 1.5.x and 2.0).

Downloads are verified against a .sha256 sidecar published next to the archive. The S3 mirror publishes sidecars for every build; the community registry does not, so by default downloads come from the mirror. Set ANOFOX_ALLOW_INSECURE_DOWNLOAD=1 to accept archives without a sidecar.

Development

# Build the extension first
make release

# Install package in dev mode
cd python
pip install -e ".[dev]"

# Run tests
ANOFOX_EXT_PATH=../build/release/extension/anofox_tabular/anofox_tabular.duckdb_extension \
  pytest tests/ -v

# Loader/utils tests run without extension (no env var needed)
pytest tests/test_loader.py tests/test_utils.py -v

License

MIT