polspec
One schema for your Polars data. Generate it, validate it, watch it drift.
Your tests need fixtures, and your pipeline needs a contract. Written separately, they drift apart. polspec lets you write the schema once, as a Polars class, and use it for both: realistic data to test against, and validation of the data that turns up.
import datetime as dt
import polars as pl
from polspec import ColSpec, FrameSpec
class Orders(FrameSpec):
order_id = ColSpec(pl.Int64, bounds=(1, 10**9), unique=True)
status = ColSpec(pl.Enum, choices={"NEW": 0.2, "PAID": 0.5, "SHIPPED": 0.3})
total = ColSpec(pl.Float64, bounds=(0, 5_000), distribution="lognormal",
distribution_params={"mean": 4, "std": 1})
placed = ColSpec(pl.Date, bounds=(dt.date(2025, 1, 1), dt.date(2025, 12, 31)),
nullable=True, null_probability=0.05)
email = ColSpec(pl.String, format="email")
orders = Orders.generate(1_000_000, seed=42) # a million rows in ~8 ms
Orders.validate(orders) # ~26 ms; raises on any breach
┌───────────┬────────┬────────────┬────────────┬───────────────────────┐
│ order_id ┆ status ┆ total ┆ placed ┆ email │
│ i64 ┆ enum ┆ f64 ┆ date ┆ str │
╞═══════════╪════════╪════════════╪════════════╪═══════════════════════╡
│ 830916752 ┆ PAID ┆ 9.187332 ┆ 2025-12-30 ┆ dupebca@xdcbivwq.dev │
│ 366447256 ┆ PAID ┆ 19.02604 ┆ 2025-11-28 ┆ hinro@ewzc.com │
│ 922459406 ┆ PAID ┆ 336.930617 ┆ 2025-04-11 ┆ jp5uo7@mgayhywoxn.com │
└───────────┴────────┴────────────┴────────────┴───────────────────────┘
What you get
- Fast. The generator is a Rust extension that fills columns in
parallel.
validate()compiles every check on every column into one Polars aggregation, so a wide table costs about what a narrow one does. Timings here are from a laptop; more are in Comparison. - Lazy and streaming.
scan()is aLazyFrame: filter and project it like any other. The sinks write Parquet, IPC, CSV or NDJSON in batches, never holding the frame in memory, and a unique column stays unique across every batch. - Every Polars dtype. Integers to 128 bits,
Decimal, time-zonedDatetime,Enum,Categorical, andList,ArrayandStructnested to any depth (plusMapon Polars 2). Bounds, choices with weights, distributions, formats such asemailanduuid4, regex patterns, uniqueness, nulls and NaNs are all declared per column, and all of them generate. - Tables that relate. Foreign keys within and across specs, conditional rules, columns derived from others, composite keys, row order and hierarchies are generated to hold, and validated; a row count is validated. Joining two specs gives the spec of the joined frame. Checks written as Polars expressions are enforced at validation.
- Validation as data.
inspect()returns every finding at once, with the offending rows a lazy filter away, and splits a frame into what passed and what to quarantine. Declared defaults can fill what is missing first. - Learn from real data.
profile()turns a frame into a spec, finding formats, the patterns codes share, frequencies, distributions and null rates — and, withparse_text=True, the numbers, flags and dates in a file read as text, in the format they are written in.profile_all()profiles related tables at once and finds the keys between them;synthesize()andsynthesize_all()make stand-ins with the same shape, a tenth the size or ten times it, every key kept. - Drift with statistics behind it. A move is reported only when it is both significant and large, as breaking or compatible.
- Specs as files and docs. Save and load specs as YAML or Python. Render
a Markdown data dictionary or a Mermaid ER diagram. A spec pickles, and
reads as an Arrow schema:
pl.Schema(Orders). - Tests that say what broke.
assert_validfails a test with the findings, the failing rows and the seed to rerun it with. A pytest fixture gives each test frames from a seed of its own, andstrategyhands a spec to Hypothesis. - A CLI for your pipeline.
polspec validate,generate,driftandsynthesize, with exit codes for CI. - Deterministic. The same seed gives the same data on every Polars from 1.39 to 2.x, on Windows, Linux and macOS, checked case by case in CI.
A quick tour
Generate lazily
scan() returns a LazyFrame, so Polars pushes filters and projections into it:
paid = (
Orders.scan(10_000_000, seed=1)
.filter(pl.col("status") == "PAID")
.select("order_id", "total")
.collect() # ~0.06 s
)
Orders.sink_parquet("orders.parquet", 1_000_000, seed=1) # streamed in batches
Validate and quarantine
inspect() never raises: it reports every finding, with the rows behind it.
broken = orders.with_columns(
pl.when(pl.col("order_id") % 1_000 == 0).then(-1.0).otherwise("total").alias("total")
)
report = Orders.inspect(broken) # 1 finding: 1,053 totals out of bounds
clean = report.passing_rows().collect() # the other 998,947 rows
Profile real data
from polspec import profile, synthesize
spec = profile(orders.head(100_000))
spec["total"].distribution # 'lognormal', fitted
fake = synthesize(orders.head(100_000), seed=1) # a stand-in with the same shape
Fake a set of related tables
import polspec
customers = pl.DataFrame({"id": range(1, 101), "tier": ["free", "pro"] * 50})
purchases = pl.DataFrame({"id": range(1, 401), "customer_id": [i % 37 + 1 for i in range(400)]})
real = {"Customers": customers, "Purchases": purchases}
registry = polspec.profile_all(real) # Purchases.customer_id -> Customers.id
fake = polspec.synthesize_all(real, scale=0.1, seed=1)
registry.validate_all(fake) # every key held by a fake parent
Catch drift
print(Orders.drift(orders.filter(pl.col("status") != "NEW")))
Drift: 0 breaking, 2 compatible, DataFrame against 'Orders'
- [compatible] Column 'status': 1 of 3 declared value(s) never appear: ['NEW']
- [compatible] Column 'status': frequencies moved by 20.0% over 800,231 value(s): ...
From the command line
polspec schema infer orders.csv -o orders_spec.py
polspec schema infer customers.csv orders.csv -o shop.yaml # a registry, keys found
polspec validate orders_spec.py orders.csv --failing rejected.csv
polspec drift orders_spec.py orders.csv --markdown
polspec generate orders_spec.py -n 1000000 -o fixture.parquet --seed 1
Install
uv add polspec # or: pip install polspec
polspec works with Polars 1.39 and later, Polars 2 included, on Python 3.12 to 3.15 (on 3.15, Polars 2: Polars 1 does not run on it). Wheels are published for Linux (x86_64, aarch64), macOS (Intel and Apple silicon) and Windows (x86_64), so you don't need a Rust toolchain.
Stable. A public name, a spec file's format, a seed's output and the command line change only in a major release, after a deprecation. Stability says what is promised, each promise held by a test, and Known limitations what polspec does not do.
Documentation
maxwellb13.github.io/polspec: start with Getting started, find a task among the how-to guides, or look a name up in the API reference.
Contributing
Setting up a checkout, the checks CI runs and the release process are in CONTRIBUTING.md. What changed, and when, is in CHANGELOG.md.
License
MIT.
Metadata
Release files for polspec 1.6.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| polspec-1.6.0.tar.gz | 1.0 MB | With provenance | Details for polspec-1.6.0.tar.gz |
Built distributions (wheels)
| File | Actions Reset | |||
|---|---|---|---|---|
| polspec-1.6.0-cp312-abi3-win_amd64.whl | CPython 3.12 | abi3 | Windows x86-64 | Details for polspec-1.6.0-cp312-abi3-win_amd64.whl |
| polspec-1.6.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl | CPython 3.12 | abi3 | Linux glibc 2.17+ x86-64 | Details for polspec-1.6.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
| polspec-1.6.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl | CPython 3.12 | abi3 | Linux glibc 2.17+ ARM64 | Details for polspec-1.6.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
| polspec-1.6.0-cp312-abi3-macosx_11_0_arm64.whl | CPython 3.12 | abi3 | macOS 11.0+ ARM64 | Details for polspec-1.6.0-cp312-abi3-macosx_11_0_arm64.whl |
| polspec-1.6.0-cp312-abi3-macosx_10_12_x86_64.whl | CPython 3.12 | abi3 | macOS 10.12+ x86-64 | Details for polspec-1.6.0-cp312-abi3-macosx_10_12_x86_64.whl |
Total release size: 34.4 MB
Release files / polspec-1.6.0.tar.gz
| Download URL | polspec-1.6.0.tar.gz |
|---|---|
| Size | 1.0 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0aea7962af9dc2a82f39bad4ec42e0b96da47f7356d28b05b39f656cae8816db
|
|
BLAKE2b-256 checksum How to use checksums |
e4742dfade266e9598d59431da9f117ec5f696efd6152ad1fb625224bec87d34
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / polspec-1.6.0-cp312-abi3-win_amd64.whl
| Download URL | polspec-1.6.0-cp312-abi3-win_amd64.whl |
|---|---|
| Size | 7.2 MB |
| Tags | CPython 3.12 Windows x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
83bd190712d3df03f0820c34c28b38f40f5ce90dd1eca1138fecce14c0cee716
|
|
BLAKE2b-256 checksum How to use checksums |
ad49173e2776e80ac4bfce913a99222cb0199943b9b41aaadba92d93991771e6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / polspec-1.6.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | polspec-1.6.0-cp312-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 6.9 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
d4d4e7b49ce150408a976c0fef2991ad992ca8895bdd904f235c754b18cae652
|
|
BLAKE2b-256 checksum How to use checksums |
fbb9e84eec7531aec39b404edebce5b0f1ba7d74eec697f15b2f6c0290953f52
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / polspec-1.6.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | polspec-1.6.0-cp312-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 6.4 MB |
| Tags | CPython 3.12 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
1bfe04e46bc8e68130527887f84366e88d26e11f5a44d18811423f1999d06d66
|
|
BLAKE2b-256 checksum How to use checksums |
d62dafe823a9166e59509bc75c5e4a8a89b19a2605a1889e7d4ac7597970d5d6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / polspec-1.6.0-cp312-abi3-macosx_11_0_arm64.whl
| Download URL | polspec-1.6.0-cp312-abi3-macosx_11_0_arm64.whl |
|---|---|
| Size | 6.1 MB |
| Tags | CPython 3.12 abi3 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
83093a17480b2da710520c6878dd2197b412a18f218c37a4fbdcd3968320b19b
|
|
BLAKE2b-256 checksum How to use checksums |
830d02eb952ead38b0d91d4b1661465696f6b61ed5ce8bcf4f950229905e1fe4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logRelease files / polspec-1.6.0-cp312-abi3-macosx_10_12_x86_64.whl
| Download URL | polspec-1.6.0-cp312-abi3-macosx_10_12_x86_64.whl |
|---|---|
| Size | 6.8 MB |
| Tags | CPython 3.12 abi3 macOS 10.12+ x86-64 |
|
SHA-256 checksum How to use checksums |
22a2a3c890aa1197e5ae8484b0cf46d92d3774c2b6ac07e455812b4906c1abfa
|
|
BLAKE2b-256 checksum How to use checksums |
bb1141e822cc9c139964cc29da5b7f93f6d25ade7c8999d7b7ff83084c987cec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logSecurity for polspec 1.6.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 6 files in this release have been uploaded with provenance metadata.
Attestation for polspec-1.6.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 16, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.