pydantable 1.11.0

Strongly-typed DataFrames for Python, powered by Rust.

PydanTable

Strongly typed DataFrames for Python, powered by Rust — Pydantic schemas, Polars-backed execution in the native extension, and an API built for services (including optional FastAPI integration).

Why PydanTable

• One schema, many surfaces: define columns with Pydantic models; use DataFrameModel (SQLModel-style) or DataFrame[YourSchema].
• Typed expressions: Expr and transform chains are validated and lowered in Rust; many errors fail fast at build/plan time.
• Familiar operations: select, filter, join, group_by, windows, melt/pivot, and pandas-flavored helpers where they help.
• Flexible materialization: row models via collect() / rows(), columnar dict[str, list], or Polars/PyArrow with the right extras.
• I/O: lazy read_* / aread_*, streaming writes, NDJSON/JSON Lines, Parquet, CSV, IPC, HTTP, SQL — see the I/O overview and decision tree.
• JSON & struct columns: struct expressions, JSON encode/decode helpers, unnest/nested models — IO_JSON, SELECTORS.
• FastAPI (optional): shared executor lifespan, NDJSON streaming from astream(), OpenAPI-friendly columnar bodies, register_exception_handlers (503 / 400 / 422). Start with the golden path and FastAPI guide.

Install

pip install pydantable

Common extras:

pip install "pydantable[polars]"   # to_polars
pip install "pydantable[arrow]"    # to_arrow / Arrow constructors
pip install "pydantable[io]"       # full file I/O convenience (arrow + polars)
pip install "pydantable[sql]"      # fetch_sql / write_sql helpers
pip install "pydantable[pandas]"   # pandas-flavored façade (pandas UI doc)
pip install "pydantable[fastapi]"  # FastAPI integration (pydantable.fastapi)

Quick start

from pydantable import DataFrameModel

class User(DataFrameModel):
    id: int
    age: int | None

df = User({"id": [1, 2], "age": [20, None]})
result = (
    df.with_columns(age2=df.age * 2)
    .filter(df.age > 10)
    .select("id", "age2")
)

print(result.to_dict())
print([r.model_dump() for r in result.collect()])

Output (exact values depend on filtering; this matches scripts/verify_doc_examples.py):

{'id': [1], 'age2': [40]}
[{'id': 1, 'age2': 40}]

Core concepts

Piece	Role
DataFrameModel	Table class with annotated columns (class Orders(DataFrameModel): ...).
DataFrame[Schema]	Generic API over your own Pydantic BaseModel.
Expr	Typed expressions in with_columns, filter, etc.
Errors	Ingest issues such as column length mismatch raise ColumnLengthMismatchError (ValueError subclass) from pydantable.errors — map to HTTP 400 in FastAPI via register_exception_handlers.

Static typing

• mypy: schema-evolving return types for many chains via the bundled mypy plugin (plugins in pyproject.toml).
• Pyright / Pylance: use committed stubs under typings/; for explicit targets, as_model(...) / try_as_model(...) / assert_model(...). See TYPING.

Rich column types (Literal, ipaddress, WKB, Annotated, …) are covered in SUPPORTED_TYPES.

Materialization: collect() / rows() → row models; to_dict() → dict[str, list]; to_polars() / to_arrow() with matching extras.

I/O at a glance

• DataFrameModel / DataFrame[Schema]: lazy read_* / aread_*, export_*, write_*, SQL helpers; eager materialize_* / fetch_* patterns live on pydantable.io — pass dict[str, list] into constructors for typed frames.
• Scripts: raw helpers (ScanFileRoot, iterators) on pydantable.io for glue code.
• Large files & NDJSON patterns: IO_JSON, IO_NDJSON, EXECUTION.

Validation controls

• Strict by default on constructors.
• Optional ingest controls: trusted_mode, ignore_errors, on_validation_errors.
• Missing optional fields: fill_missing_optional (default True).

Documentation

Topic	Link
Docs home	pydantable.readthedocs.io
Map of all pages	DOCS_MAP
Quickstart	QUICKSTART
DataFrameModel	DATAFRAMEMODEL
Typing (mypy vs Pyright)	TYPING
I/O overview	IO_OVERVIEW
Pandas-like API	PANDAS_UI
FastAPI path	GOLDEN_PATH_FASTAPI → FASTAPI → FASTAPI_ENHANCEMENTS
Cookbooks	Cookbook index (FastAPI, lazy pipelines, JSON logs, …)
Example multi-router app	docs/examples/fastapi/service_layout/ in this repo
Test helpers	pydantable.testing.fastapi — see FASTAPI
Execution & async	EXECUTION · MATERIALIZATION
Behavioral contract	INTERFACE_CONTRACT
Troubleshooting	TROUBLESHOOTING
Versioning	VERSIONING
Changelog	CHANGELOG

Development

Use a virtual environment at .venv in the repo root (the Makefile defaults to .venv/bin/python). Full contributor setup, Maturin/Rust builds, and release notes: DEVELOPER.

make check-full      # ruff, mypy, pyright, typing snippet tests, Sphinx, Rust

License

MIT