pydantable 0.16.1

Strongly-typed DataFrames for Python, powered by Rust.

PydanTable

Typed dataframe transformations for FastAPI and Pydantic services, backed by a Rust execution core.

Current release: 0.16.1 · Python 3.10+

---

Documentation

The full manual lives on Read the Docs:

https://pydantable.readthedocs.io/en/latest/

That site is the supported entry point for concepts, contracts, API notes, and examples. The sections below point to the same pages so you can jump straight from GitHub.

Topic	Read the Docs
Home / overview	Documentation home
DataFrameModel contract (inputs, transforms, collisions, materialization)	DataFrameModel (SQLModel-like)
Column types (scalars, structs, list[T], nullability, unsupported cases)	Supported data types
FastAPI (routers, bodies, collect, responses)	FastAPI integration
Execution model (collect, to_dict, to_polars, optional Python Polars, UI modules)	Execution (Rust engine)
Semantics (nulls, joins, ordering, reshaping, windows — Polars-style contract)	Interface contract
Roadmap (shipped through 0.16.1, planned 0.17, path to v1.0.0)	Roadmap
Why not use Polars directly?	Why not just use Polars?
Pandas-style imports (pydantable.pandas)	Pandas UI
PySpark-style imports (pydantable.pyspark)	PySpark UI
PySpark helpers & parity	PySpark interface · PySpark API parity
Polars parity (scorecard, workflows, transformation roadmap)	Parity scorecard · Polars-style workflows · Transformation parity roadmap
Contributors (build, test, benchmarks, releases)	Developer guide
Plan / vision (architecture phasing)	Plan document
Python API reference (autodoc)	API reference

For copy-paste convenience, the site base URL is:

https://pydantable.readthedocs.io/en/latest/

---

What PydanTable does

PydanTable keeps Pydantic models as the source of truth for:

• column types and nullability (Optional[T] / T | None)
• typed expressions — invalid combinations fail when the expression is built (Rust AST), not only at runtime
• schema evolution — chained transforms produce new model types with stable rules (e.g. with_columns name collisions)

The default API feels Polars-like; optional pydantable.pandas and pydantable.pyspark modules only change naming and imports — execution is always the native core. Details: Execution, Interface contract.

0.16.1 fixes Rust expression typing so invalid arithmetic on dict[str, T] map columns (for example df.m + 1) raises TypeError instead of panicking, and fixes DataFrame[Schema](pa.Table) / RecordBatch construction (correct pydantable.io imports in validate_columns_strict). 0.16.0 adds read_parquet / read_ipc, to_arrow / ato_arrow, Table / RecordBatch constructor ingest ( pyarrow; pydantable[arrow] ), and FastAPI hardening (multipart uploads, Depends executors, background tasks, HTTP status notes). 0.15.0 added async materialization (acollect, ato_dict, ato_polars, and DataFrameModel arows / ato_dicts), FastAPI async + lifespan examples, PyArrow map<utf8, …> ingest for dict[str, T] columns, PySpark trim / abs / round / floor / ceil, and removed the legacy validate_data constructor argument — use trusted_mode only on DataFrame / DataFrameModel. 0.14.0 added window orderBy(..., nulls_last=...), DtypeDriftWarning, validate_data deprecation (removed in 0.15.0), FastAPI TestClient docs/tests, and PySpark dayofmonth / lower / upper. See changelog and Roadmap.

Expression surface (current release, Rust-typed Expr):

• Globals in select: global_sum, global_mean, global_count, global_min, global_max, and global_row_count() (row count / COUNT(*)). PySpark: F.count() with no argument for row count; F.count(F.col(...)) for non-null column count.
• Windows: row_number, rank, dense_rank, window_sum, window_mean, window_min, window_max, lag, lead with Window.partitionBy(...).orderBy(..., nulls_last=...) / .spec(), plus framed windows (rowsBetween, rangeBetween) for supported operations.
• Temporal: strptime, unix_timestamp, cast from str → date / datetime (Polars parsing; use strptime for fixed formats), dt_* parts, dt_nanosecond on datetime / time.
• Maps / binary: map_len, map_get, map_contains_key, binary_len, including nested JSON-like map value dtypes with string keys.
• Map utilities: map_keys(), map_values(), map_entries(), and map_from_entries() for per-row key/value extraction and reconstruction on dict[str, T] columns.

PySpark-named helpers live under pydantable.pyspark.sql.functions. Details: Supported types, Interface contract, CHANGELOG.

---

Install

pip install pydantable

Optional Python Polars (for to_polars() only):

pip install 'pydantable[polars]'

From a git checkout, the Rust extension must be built (e.g. with maturin):

pip install .

See Developer guide — local setup for maturin develop, release builds, and CI parity.

---

Quick start

from pydantable import DataFrameModel

class User(DataFrameModel):
    id: int
    age: int | None

df = User({"id": [1, 2], "age": [20, None]})
df2 = df.with_columns(age2=df.age * 2)
df3 = df2.select("id", "age2")
df4 = df3.filter(df3.age2 > 10)

print(df4.to_dict())

Example output:

{'age2': [40], 'id': [1]}

• Materialization: collect() returns a list of Pydantic row models; to_dict() returns columnar dict[str, list]; to_arrow / ato_arrow (optional pyarrow) return a PyArrow Table. 0.15.0 acollect / ato_dict / ato_polars run the same work off the asyncio loop (ato_arrow in 0.16.0).

• Alternate UIs:

  from pydantable.pandas import DataFrameModel as PandasDataFrameModel
  from pydantable.pyspark import DataFrameModel as PySparkDataFrameModel
  from pydantable import DataFrameModel as DefaultDataFrameModel
  

More examples: FastAPI integration, Polars-style workflows.

Input quality policy (optional): constructors are strict by default, and can be switched to best-effort ingestion with ignore_errors=True plus on_validation_errors=... to receive failed rows (row_index, row, validation errors). See DataFrameModel docs.

---

Development

make check-full   # Ruff, mypy, Rust fmt/clippy/tests (see Makefile for `rust-test` env)
pytest -q         # or: pytest -n auto  with the [dev] extra

Rust + Python: see Developer guide (formatting, maturin, make rust-test for cargo test with the venv PYTHONPATH, benchmarks, contribution workflow).

---

License

MIT