Skip to main content

polspec

One schema for your Polars data. Generate it, validate it, watch it drift.

PyPI Python Tests License

Your tests need fixtures, and your pipeline needs a contract. Written separately, they drift apart. polspec lets you write the schema once, as a Polars class, and use it for both: realistic data to test against, and validation of the data that turns up.

import datetime as dt

import polars as pl
from polspec import ColSpec, FrameSpec

class Orders(FrameSpec):
    order_id = ColSpec(pl.Int64, bounds=(1, 10**9), unique=True)
    status   = ColSpec(pl.Enum, choices={"NEW": 0.2, "PAID": 0.5, "SHIPPED": 0.3})
    total    = ColSpec(pl.Float64, bounds=(0, 5_000), distribution="lognormal",
                       distribution_params={"mean": 4, "std": 1})
    placed   = ColSpec(pl.Date, bounds=(dt.date(2025, 1, 1), dt.date(2025, 12, 31)),
                       nullable=True, null_probability=0.05)
    email    = ColSpec(pl.String, format="email")

orders = Orders.generate(1_000_000, seed=42)   # a million rows in ~8 ms
Orders.validate(orders)                        # ~26 ms; raises on any breach
┌───────────┬────────┬────────────┬────────────┬───────────────────────┐
│ order_id  ┆ status ┆ total      ┆ placed     ┆ email                 │
│ i64       ┆ enum   ┆ f64        ┆ date       ┆ str                   │
╞═══════════╪════════╪════════════╪════════════╪═══════════════════════╡
│ 830916752 ┆ PAID   ┆ 9.187332   ┆ 2025-12-30 ┆ dupebca@xdcbivwq.dev  │
│ 366447256 ┆ PAID   ┆ 19.02604   ┆ 2025-11-28 ┆ hinro@ewzc.com        │
│ 922459406 ┆ PAID   ┆ 336.930617 ┆ 2025-04-11 ┆ jp5uo7@mgayhywoxn.com │
└───────────┴────────┴────────────┴────────────┴───────────────────────┘

What you get

  • Fast. The generator is a Rust extension that fills columns in parallel. validate() compiles every check on every column into one Polars aggregation, so a wide table costs about what a narrow one does. Timings here are from a laptop; more are in Comparison.
  • Lazy and streaming. scan() is a LazyFrame: filter and project it like any other. The sinks write Parquet, IPC, CSV or NDJSON in batches, never holding the frame in memory, and a unique column stays unique across every batch.
  • Every Polars dtype. Integers to 128 bits, Decimal, time-zoned Datetime, Enum, Categorical, and List, Array and Struct nested to any depth (plus Map on Polars 2). Bounds, choices with weights, distributions, formats such as email and uuid4, regex patterns, uniqueness, nulls and NaNs are all declared per column, and all of them generate.
  • Tables that relate. Foreign keys within and across specs, conditional rules, columns derived from others, composite keys, row order and hierarchies are generated to hold, and validated; a row count is validated. Joining two specs gives the spec of the joined frame. Checks written as Polars expressions are enforced at validation.
  • Validation as data. inspect() returns every finding at once, with the offending rows a lazy filter away, and splits a frame into what passed and what to quarantine. Declared defaults can fill what is missing first.
  • Learn from real data. profile() turns a frame into a spec, finding formats, the patterns codes share, frequencies, distributions and null rates — and, with parse_text=True, the numbers, flags and dates in a file read as text, in the format they are written in. profile_all() profiles related tables at once and finds the keys between them; synthesize() and synthesize_all() make stand-ins with the same shape, a tenth the size or ten times it, every key kept.
  • Drift with statistics behind it. A move is reported only when it is both significant and large, as breaking or compatible.
  • Specs as files and docs. Save and load specs as YAML or Python. Render a Markdown data dictionary or a Mermaid ER diagram. A spec pickles, and reads as an Arrow schema: pl.Schema(Orders).
  • Tests that say what broke. assert_valid fails a test with the findings, the failing rows and the seed to rerun it with. A pytest fixture gives each test frames from a seed of its own, and strategy hands a spec to Hypothesis.
  • A CLI for your pipeline. polspec validate, generate, drift and synthesize, with exit codes for CI.
  • Deterministic. The same seed gives the same data on every Polars from 1.39 to 2.x, on Windows, Linux and macOS, checked case by case in CI.

A quick tour

Generate lazily

scan() returns a LazyFrame, so Polars pushes filters and projections into it:

paid = (
    Orders.scan(10_000_000, seed=1)
    .filter(pl.col("status") == "PAID")
    .select("order_id", "total")
    .collect()                                            # ~0.06 s
)
Orders.sink_parquet("orders.parquet", 1_000_000, seed=1)  # streamed in batches

Validate and quarantine

inspect() never raises: it reports every finding, with the rows behind it.

broken = orders.with_columns(
    pl.when(pl.col("order_id") % 1_000 == 0).then(-1.0).otherwise("total").alias("total")
)
report = Orders.inspect(broken)          # 1 finding: 1,053 totals out of bounds
clean = report.passing_rows().collect()  # the other 998,947 rows

Profile real data

from polspec import profile, synthesize

spec = profile(orders.head(100_000))
spec["total"].distribution                         # 'lognormal', fitted
fake = synthesize(orders.head(100_000), seed=1)    # a stand-in with the same shape
import polspec

customers = pl.DataFrame({"id": range(1, 101), "tier": ["free", "pro"] * 50})
purchases = pl.DataFrame({"id": range(1, 401), "customer_id": [i % 37 + 1 for i in range(400)]})
real = {"Customers": customers, "Purchases": purchases}

registry = polspec.profile_all(real)               # Purchases.customer_id -> Customers.id
fake = polspec.synthesize_all(real, scale=0.1, seed=1)
registry.validate_all(fake)                        # every key held by a fake parent

Catch drift

print(Orders.drift(orders.filter(pl.col("status") != "NEW")))
Drift: 0 breaking, 2 compatible, DataFrame against 'Orders'
  - [compatible] Column 'status': 1 of 3 declared value(s) never appear: ['NEW']
  - [compatible] Column 'status': frequencies moved by 20.0% over 800,231 value(s): ...

From the command line

polspec schema infer orders.csv -o orders_spec.py
polspec schema infer customers.csv orders.csv -o shop.yaml   # a registry, keys found
polspec validate orders_spec.py orders.csv --failing rejected.csv
polspec drift orders_spec.py orders.csv --markdown
polspec generate orders_spec.py -n 1000000 -o fixture.parquet --seed 1

Install

uv add polspec           # or: pip install polspec

polspec works with Polars 1.39 and later, Polars 2 included, on Python 3.12 to 3.15 (on 3.15, Polars 2: Polars 1 does not run on it). Wheels are published for Linux (x86_64, aarch64), macOS (Intel and Apple silicon) and Windows (x86_64), so you don't need a Rust toolchain.

Stable. A public name, a spec file's format, a seed's output and the command line change only in a major release, after a deprecation. Stability says what is promised, each promise held by a test, and Known limitations what polspec does not do.

Documentation

maxwellb13.github.io/polspec: start with Getting started, find a task among the how-to guides, or look a name up in the API reference.

Contributing

Setting up a checkout, the checks CI runs and the release process are in CONTRIBUTING.md. What changed, and when, is in CHANGELOG.md.

License

MIT.

Metadata

Release files for polspec 1.6.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for polspec 1.6.1
File Size Uploaded Provenance Actions
polspec-1.6.1.tar.gz 1.0 MB With provenance Details for polspec-1.6.1.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for polspec 1.6.1
File Actions
polspec-1.6.1-cp312-abi3-win_amd64.whl CPython 3.12 abi3 Windows x86-64 Details for polspec-1.6.1-cp312-abi3-win_amd64.whl
polspec-1.6.1-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.12 abi3 Linux glibc 2.17+ x86-64 Details for polspec-1.6.1-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
polspec-1.6.1-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl CPython 3.12 abi3 Linux glibc 2.17+ ARM64 Details for polspec-1.6.1-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
polspec-1.6.1-cp312-abi3-macosx_11_0_arm64.whl CPython 3.12 abi3 macOS 11.0+ ARM64 Details for polspec-1.6.1-cp312-abi3-macosx_11_0_arm64.whl
polspec-1.6.1-cp312-abi3-macosx_10_12_x86_64.whl CPython 3.12 abi3 macOS 10.12+ x86-64 Details for polspec-1.6.1-cp312-abi3-macosx_10_12_x86_64.whl

Total release size: 34.4 MB

Release files / polspec-1.6.1.tar.gz

Download URL polspec-1.6.1.tar.gz
Size 1.0 MB
Tags Source
SHA-256 checksum
How to use checksums
7d7899152c75e4263e69db6f4f2baa6202e74b90a49882b4364dffceb786dadd
BLAKE2b-256 checksum
How to use checksums
2829738f4e6f330f168bdb482c8048b879bd01dc69b3bfc7644f0e9b18571308
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / polspec-1.6.1-cp312-abi3-win_amd64.whl

Download URL polspec-1.6.1-cp312-abi3-win_amd64.whl
Size 7.2 MB
Tags CPython 3.12 Windows x86-64 abi3
SHA-256 checksum
How to use checksums
67c2633406b18fc11442176cdd7dc86d0f475e8c5f1006ba24e9a045cc75349d
BLAKE2b-256 checksum
How to use checksums
650a144d9a03e798bdd06ef7da950a25caf4be2cb2dc5958c48bf031bb660d09
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / polspec-1.6.1-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL polspec-1.6.1-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 6.9 MB
Tags CPython 3.12 Linux glibc 2.17+ x86-64 abi3
SHA-256 checksum
How to use checksums
a183f2f94a53966f4f764d44b8fe1559c20d7f0466146030b426f80a1165b551
BLAKE2b-256 checksum
How to use checksums
f9817ca5e73c2dc93edb1b42932eed826a6d1f79609876a39a3a50082166ff2f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / polspec-1.6.1-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl

Download URL polspec-1.6.1-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Size 6.4 MB
Tags CPython 3.12 Linux glibc 2.17+ ARM64 abi3
SHA-256 checksum
How to use checksums
612f8801ef8d38f56aa6cad782b3a09cf201e90a2d1e42e66f73f88de327e4f1
BLAKE2b-256 checksum
How to use checksums
17bffd39c3b1e71531adddb6e2702a5e9455f48b428c7576a52a24ae4b84f320
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / polspec-1.6.1-cp312-abi3-macosx_11_0_arm64.whl

Download URL polspec-1.6.1-cp312-abi3-macosx_11_0_arm64.whl
Size 6.1 MB
Tags CPython 3.12 abi3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
455552c4bdd6ee15e5f200c543f973ae4a07531684a0a96f4ceb9b5be8a4480b
BLAKE2b-256 checksum
How to use checksums
48ec89f96aece93771ec0fc6887f1ef03817f785cc2983c59f8436d73b1dddd2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Release files / polspec-1.6.1-cp312-abi3-macosx_10_12_x86_64.whl

Download URL polspec-1.6.1-cp312-abi3-macosx_10_12_x86_64.whl
Size 6.8 MB
Tags CPython 3.12 abi3 macOS 10.12+ x86-64
SHA-256 checksum
How to use checksums
0a1d6d1278752896090572b080c4ce36f7c4c63e81342a08bbec571f89423e5f
BLAKE2b-256 checksum
How to use checksums
a9cdbdcfb1287b22a5378b694683aff8e4a9935012a02e5dca3835bbc8de0137
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

Security for polspec 1.6.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for polspec-1.6.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.6.1 This release

6 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page