polspec

Declare a Polars schema once. Generate data that matches it, and validate data against it — from the same declaration.

Early alpha. The API, the YAML format, and the exact values a given seed produces are all still moving. See Roadmap and stability before depending on any of it.

import polars as pl
from polspec import ColSpec, FrameSpec

class Orders(FrameSpec):
    order_id = ColSpec(pl.Int64, bounds=(1, None))
    status   = ColSpec(pl.Enum(["NEW", "PAID", "SHIPPED"]))
    total    = ColSpec(pl.Float64, bounds=(0.0, None))
    placed   = ColSpec(pl.Date, nullable=True)

df = Orders.generate(1_000_000, seed=42)   # a million rows in well under a second
Orders.validate(df)                        # raises ValidationError on any breach

The generator is a Rust extension that fills columns in parallel — ten million rows across four columns generates in well under 50ms on a release build. validate() compiles every check across every column into a single Polars aggregation, so validating a wide table costs about the same as a narrow one.

Why generate and validate from one declaration

A validation library tells you when production data drifted. A fixture library gives you something to test against. Keeping both behind one declaration means the fixtures and the contract can't quietly disagree — and where they still can today, it's written down in Known limitations, each backed by a test that fails the moment it's fixed.

Install

Package install (when published to PyPI)

uv add polspec           # preferred
pip install polspec      # alternative
uv add "polspec[all]"    # extras: PyArrow for Parquet/IPC

Not published to PyPI yet — install from a checkout. The generator is a compiled Rust extension, so this needs a Rust toolchain and maturin:

git clone https://github.com/MaxwellB13/polspec.git
cd polspec
uv sync --group dev        # or: pip install -e ".[all]" && pip install maturin
maturin develop --release

maturin develop builds the extension and installs the package into your active environment, editable. Writing Parquet or Arrow IPC needs PyArrow, included via [all]/the dev group above.

What you can declare

• Types and shape — every scalar and temporal Polars dtype, nullability, bounds (including open-ended, bounds=(0, None)), string lengths, and value domains with weights.
• Rules and invariants — conditional values (ColRule), single-column validators, multi-column checks, composite uniqueness, and foreign keys between specs.
• Data on demand — random or coverage-guaranteeing (method="cartesian") generation, reproducible seeds, batched streaming straight to Parquet, CSV, Arrow IPC or NDJSON.
• Specs from elsewhere — infer a spec by profiling an existing DataFrame, or load one from YAML.
• Shared categories — a CatSpec registry so several tables agree on an Enum/Categorical domain instead of each restating it.

class Categories(CatSpec):
    STATUS   = pl.Enum(["NEW", "PAID", "SHIPPED"])
    CURRENCY = pl.Categorical(pl.Categories("CURRENCY", physical=pl.UInt8))

ColSpec(Categories.STATUS)

Command line

polspec schema infer orders.parquet -o orders.yaml   # profile data into a schema
polspec schema new Payments -o payments.py           # or start from a blank one
polspec test orders.yaml -o test_orders.py           # generate a round-trip pytest file

The generated test disables whatever validate() flags a constraint generation can't yet satisfy (unique=True, __checks__, ...) with a comment explaining why, and skips a foreign key that needs parent data it wasn't given — see Command line.

Documentation

Full docs: maxwellb13.github.io/polspec

• Getting started · Comparison to other approaches
• Declaring columns · Constraints
• Generating data · Validating data
• Shared categories · YAML specs
• Testing pipelines · Command line
• Architecture · Known limitations · Roadmap and stability

To edit and preview the docs locally instead, see zensical:

zensical serve

Benchmarks — polspec vs NumPy vs pure Python

The benchmarks/bench_generate.py script compares the Rust-backed generator to:

• a NumPy-based generator that builds equivalent columns, and
• a pure-Python implementation using random.

Run locally:

uv run --group bench python benchmarks/bench_generate.py

Example results (illustrative; your hardware will differ):

Ran on Intel 13900K + 64GB DDR5 RAM

Notes

• All three implementations emit a Polars DataFrame with the same schema to keep the comparison fair. The NumPy version uses a fixed-width trick for strings (since vectorized ragged strings are not available), and the pure-Python version builds lists then constructs a DataFrame.
• The script warms each implementation once, then times increasing sizes; the pure-Python path is skipped for very large sizes once it exceeds a cutoff.

Development

After the install steps above:

pytest
ruff check . && ruff format --check .
cargo clippy --release

License

Not yet specified.