pydantable 0.20.0

Strongly-typed DataFrames for Python, powered by Rust.

PydanTable

Typed dataframe transformations for FastAPI and Pydantic services, backed by a Rust execution core (Polars inside the native extension).

Current release: 0.20.0 · Python 3.10+

---

At a glance

• Schemas first: Pydantic field annotations define column types, nullability (T | None), and which expressions are legal. Many mistakes are caught when you build the Expr, not only when you run the query.
• Two entry styles: DataFrameModel (SQLModel-like whole-table class with a generated row model) or DataFrame[YourSchema](data) with any Pydantic BaseModel schema.
• Polars-shaped API: select, with_columns, filter, join, group_by, windows, reshape helpers — semantics are documented in the interface contract, not guaranteed identical to Polars on every edge case.
• Optional extras: pydantable[polars] for to_polars(); pydantable[arrow] for read_parquet / read_ipc, to_arrow / ato_arrow, and pa.Table / RecordBatch constructors.
• Optional façades: pydantable.pandas and pydantable.pyspark swap naming/imports; execution stays the same in-process core (not a real Spark or pandas backend).
• Service-ready: Sync and async materialization (collect, to_dict, acollect, ato_dict, …), FastAPI patterns, and trusted ingest modes for bulk JSON or Arrow.
• REPL / discovery: repr(df) on DataFrame and DataFrameModel shows the parameterized class, schema type, and column dtypes (wide tables truncate with … and N more). columns, shape, empty, dtypes, info(), and describe() (numeric, bool, str) are on the core API (see Interface contract for shape vs materialized rows). Expr and WhenChain have readable repr for debugging pipelines. Row counts in repr are omitted—use collect() / to_dict() when you need data. Jupyter / VS Code notebooks: _repr_html_() renders a bounded HTML table preview (no polars required); tune via pydantable.display or PYDANTABLE_REPR_HTML_*. Details: Execution.

---

Documentation

The canonical manual is on Read the Docs: https://pydantable.readthedocs.io/en/latest/

Topic	Read the Docs
Home / overview	Documentation home
Five-minute tour	Quickstart
Changelog & versions	Changelog · Versioning (0.x)
DataFrameModel (inputs, transforms, collisions, materialization)	DataFrameModel
Column types (scalars, structs, list[T], maps, trusted ingest)	Supported data types
FastAPI (routers, bodies, async, multipart)	FastAPI integration
Execution (collect, to_dict, to_polars, to_arrow, async, repr)	Execution
Semantics (nulls, joins, windows, reshape)	Interface contract
Roadmap (shipped 0.20.0 UX / discovery, path to v1.0.0)	Roadmap
Why not Polars alone?	Why not just use Polars?
Pandas-style API (pydantable.pandas)	Pandas UI
PySpark-style API (pydantable.pyspark)	PySpark UI · Parity matrix
Polars parity	Scorecard · Workflows · Transformation roadmap
Contributors	Developer guide
Architecture plan	Plan document
Python API (autodoc)	API reference

---

Install

pip install pydantable

Optional dependencies (same package, feature extras):

pip install 'pydantable[polars]'   # to_polars()
pip install 'pydantable[arrow]'  # read_parquet/read_ipc, to_arrow, Table/RecordBatch constructors

From a git checkout you need a Rust toolchain and a build of the extension (e.g. Maturin):

pip install .
# editable: maturin develop --manifest-path pydantable-core/Cargo.toml

Full setup, make check-full, and release notes: Developer guide.

---

Quick start

from pydantable import DataFrameModel

class User(DataFrameModel):
    id: int
    age: int | None

df = User({"id": [1, 2], "age": [20, None]})
df2 = df.with_columns(age2=df.age * 2)
df3 = df2.select("id", "age2")
df4 = df3.filter(df3.age2 > 10)

# Columnar dict (good for JSON APIs)
print(df4.to_dict())
# {'age2': [40], 'id': [1]}

# List of Pydantic row models (default collect)
for row in df4.collect():
    print(row.id, row.age2)

Materialization: collect() → list of row models; to_dict() / collect(as_lists=True) → dict[str, list]; to_polars() / to_arrow() when the matching extra is installed. Async: acollect, ato_dict, ato_polars, ato_arrow offload blocking work from the event loop (Execution, FastAPI).

Alternate import styles (same engine):

from pydantable.pandas import DataFrameModel as PandasDataFrameModel
from pydantable.pyspark import DataFrameModel as PySparkDataFrameModel
from pydantable import DataFrameModel as DefaultDataFrameModel

More examples: FastAPI, Polars-style workflows.

Validation policy: Constructors validate strictly by default. For messy row lists, ignore_errors=True plus on_validation_errors=callback receives failed rows (row_index, row, Pydantic errors). Trusted bulk paths use trusted_mode (off / shape_only / strict). Details: DataFrameModel, Supported types.

---

Expression & API surface

Typed Expr builds a Rust AST. Highlights:

• Globals in select: global_sum, global_mean, global_count, global_min, global_max, global_row_count() (row count). PySpark façade: F.count() with no argument = row count.
• Windows: row_number, rank, dense_rank, window_sum, window_mean, window_min, window_max, lag, lead with Window.partitionBy(...).orderBy(..., nulls_last=...); framed rowsBetween / rangeBetween where supported (window semantics).
• Temporal & strings: strptime, unix_timestamp, cast to date/datetime, dt_* parts, strip / lower / upper, str_replace, strip_prefix / suffix / chars, list helpers (list_len, list_get, …).
• Maps (string keys): map_len, map_get, map_contains_key, map_keys, map_values, map_entries, map_from_entries, element_at; binary_len for bytes columns.

PySpark-named wrappers: pydantable.pyspark.sql.functions mirrors much of the above (parity table).

---

Recent releases

0.20.0 — UX, discovery, docs, and display: Quickstart, Execution (materialization costs, import styles, copy-as / interchange); core columns, shape, info(), describe() (int/float/bool/str), value_counts, set_display_options / PYDANTABLE_REPR_HTML_*, _repr_mimebundle_, optional PYDANTABLE_VERBOSE_ERRORS; Expr / WhenChain repr; PySpark show() / summary(); multi-line DataFrame repr and _repr_html_. Changelog, Interface contract Introspection.

0.19.0 — Pre-1.0 documentation consolidation: Versioning (0.x), interface contract cross-links, parity/README/index refresh for the 0.19 → 1.0 path, PERFORMANCE benchmark spot-check note, release-hygiene alignment with CI; group_by tests sort output where row order is not guaranteed (stable pytest-xdist). No new Expr or PySpark façade methods.

0.18.0 — Clearer Polars error context for group_by().agg(); explicit deferral of non-string map keys (Supported types, Roadmap); parity/roadmap doc refresh (no new façade APIs); Hypothesis smoke for join / group_by.

0.17.0 — Tighter docs and tests for map_get / map_contains_key after PyArrow map<utf8, …> ingest; more pyspark.sql.functions thin wrappers (str_replace, regexp_replace, strip_*, strptime, binary_len, list_*). Non-string map keys (dict[int, T], etc.) remain future work (Roadmap Later).

0.16.x — Arrow interchange (read_parquet / read_ipc, to_arrow / ato_arrow, Table/RecordBatch constructors), FastAPI multipart and deployment docs, map-column arithmetic TypeError fix, DataFrame[Schema](pa.Table) constructor fix.

Older highlights: 0.15.0 async materialization and Arrow map ingest; 0.14.0 window null ordering and FastAPI TestClient coverage. Full history: Changelog.

---

Development

From a clone with .venv and pip install -e ".[dev]" plus a built extension:

make check-full              # Ruff, mypy, Rust fmt / clippy / tests
PYTHONPATH=python pytest -q  # integration tests (see DEVELOPER.md)

Rust tests need the Makefile PYO3_PYTHON / PYTHONPATH wiring: make rust-test. Details: Developer guide.

---

License

MIT