pydantable 1.16.1

Strongly-typed DataFrames for Python, powered by Rust.

PydanTable

Strongly typed DataFrames for Python, powered by Rust — Pydantic schemas, Polars-backed execution in the native extension, and an API built for services (including optional FastAPI integration).

Current release: 1.16.1 — highlights in the changelog.

Why PydanTable

• One schema, many surfaces: define columns with Pydantic models; use DataFrameModel (SQLModel-style) or DataFrame[YourSchema].
• Typed expressions: Expr and transform chains are validated and lowered in Rust; many errors fail fast at build/plan time.
• Familiar operations: select, filter, join, group_by, windows, melt/pivot, and pandas-flavored helpers where they help.
• Flexible materialization: row models via collect() / rows(), columnar dict[str, list], or Polars/PyArrow with the right extras.
• I/O: lazy read_* / aread_*, streaming writes, NDJSON/JSON Lines, Parquet, CSV, IPC, HTTP, SQL (SQLModel-first fetch_sqlmodel / write_sqlmodel, explicit string SQL fetch_sql_raw / write_sql_raw, or deprecated unprefixed names) — I/O overview, IO_SQL, SQLModel roadmap, and decision tree.
• JSON & struct columns: struct expressions, JSON encode/decode helpers, unnest/nested models — IO_JSON, SELECTORS.
• FastAPI (optional): shared executor lifespan, NDJSON streaming from astream(), OpenAPI-friendly columnar bodies, register_exception_handlers (503 / 400 / 422). Start with the golden path and FastAPI guide.
• Moltres SQL engine (optional): install pydantable[moltres] for SqlDataFrame / SqlDataFrameModel backed by moltres-core’s MoltresPydantableEngine (pydantable-protocol ExecutionEngine). The goal is to keep transforms on the SQL side (plans compiled to SQL) instead of loading whole tables into Python—especially when you write results back to the same database. Guide: MOLTRES_SQL; protocol authors: Custom engine packages.

Install

pip install pydantable

Common extras:

pip install "pydantable[polars]"   # to_polars
pip install "pydantable[arrow]"    # to_arrow / Arrow constructors
pip install "pydantable[io]"       # full file I/O convenience (arrow + polars)
pip install "pydantable[sql]"      # SQLModel + SQLAlchemy: fetch_sqlmodel, write_sqlmodel, *_raw, …
pip install "pydantable[pandas]"   # pandas-flavored façade (pandas UI doc)
pip install "pydantable[fastapi]"  # FastAPI integration (pydantable.fastapi)
pip install "pydantable[moltres]"   # SqlDataFrame / SqlDataFrameModel (sqlalchemy engine)

Quick start

from pydantable import DataFrameModel

class User(DataFrameModel):
    id: int
    age: int | None

df = User({"id": [1, 2], "age": [20, None]})
result = (
    df.with_columns(age2=df.age * 2)
    .filter(df.age > 10)
    .select("id", "age2")
)

print(result.to_dict())
print([r.model_dump() for r in result.collect()])

Output (exact values depend on filtering; this matches scripts/verify_doc_examples.py):

{'id': [1], 'age2': [40]}
[{'id': 1, 'age2': 40}]

Core concepts

Piece	Role
DataFrameModel	Table class with annotated columns (class Orders(DataFrameModel): ...).
DataFrame[Schema]	Generic API over your own Pydantic BaseModel.
SqlDataFrame / SqlDataFrameModel	Same shapes with pydantable[moltres] — Moltres compiles plans to SQL so transforms can stay in the database (sql_config= / moltres_engine=); prefer when you are not round-tripping full tables through Python (e.g. write back to the same DB).
Expr	Typed expressions in with_columns, filter, etc.
Errors	Ingest issues such as column length mismatch raise ColumnLengthMismatchError (ValueError subclass) from pydantable.errors — map to HTTP 400 in FastAPI via register_exception_handlers.

Static typing

• mypy: schema-evolving return types for many chains via the bundled mypy plugin (plugins in pyproject.toml).
• Pyright / Pylance: use committed stubs under typings/; for explicit targets, as_model(...) / try_as_model(...) / assert_model(...) and typed escape hatches like agg_as_model(...) / rolling_agg_as_model(...). See TYPING.

Rich column types (Literal, ipaddress, WKB, Annotated, …) are covered in SUPPORTED_TYPES.

Materialization: collect() / rows() → row models; to_dict() → dict[str, list]; to_polars() / to_arrow() with matching extras.

I/O at a glance

• DataFrameModel / DataFrame[Schema]: lazy read_* / aread_*, export_*, write_*, and SQLModel helpers (fetch_sqlmodel, write_sqlmodel, …). For eager column loads, import materialize_*, fetch_sqlmodel, iter_sqlmodel, … from pydantable (same entrypoints as the internal pydantable.io package) and pass dict[str, list] into constructors.
• SQL details: IO_SQL (recommended APIs, *_raw, deprecations) and SQLMODEL_SQL_ROADMAP (phased migration).
• Large files & NDJSON patterns: IO_JSON, IO_NDJSON, EXECUTION.

Validation controls

• Strict by default on constructors.
• Optional ingest controls: trusted_mode, ignore_errors, on_validation_errors.
• Missing optional fields: fill_missing_optional (default True).
• Validation presets: validation_profile=... (or __pydantable__ = {"validation_profile": "..."}).
• Per-column and nested strictness: {doc}STRICTNESS (field policies + profile defaults).

Documentation

Topic	Link
Docs home	pydantable.readthedocs.io
Map of all pages	DOCS_MAP
Quickstart	QUICKSTART
DataFrameModel	DATAFRAMEMODEL
Typing (mypy vs Pyright)	TYPING
I/O overview	IO_OVERVIEW
SQL (SQLModel, raw string SQL)	IO_SQL · SQLMODEL_SQL_ROADMAP
Pandas-like API	PANDAS_UI
FastAPI path	GOLDEN_PATH_FASTAPI → FASTAPI → FASTAPI_ENHANCEMENTS
Service ergonomics (OpenAPI, aliases, redaction)	SERVICE_ERGONOMICS
Custom dtypes	CUSTOM_DTYPES
Strictness	STRICTNESS
Cookbooks	Cookbook index (FastAPI, lazy pipelines, JSON logs, …)
Example multi-router app	docs/examples/fastapi/service_layout/ in this repo
Test helpers	pydantable.testing.fastapi — see FASTAPI
Execution & async	EXECUTION · MATERIALIZATION
Behavioral contract	INTERFACE_CONTRACT
Troubleshooting	TROUBLESHOOTING
Versioning	VERSIONING
Changelog	CHANGELOG

Development

Use a virtual environment at .venv in the repo root (the Makefile defaults to .venv/bin/python). Full contributor setup, Maturin/Rust builds, and release notes: DEVELOPER.

make check-full      # ruff, ty, pyright, typing snippet tests, Sphinx, Rust

License

MIT