pydantable 1.14.0

Strongly-typed DataFrames for Python, powered by Rust.

PydanTable

Strongly typed DataFrames for Python, powered by Rust — Pydantic schemas, Polars-backed execution in the native extension, and an API built for services (including optional FastAPI integration).

Current release: 1.14.0 — highlights in the changelog.

Why PydanTable

• One schema, many surfaces: define columns with Pydantic models; use DataFrameModel (SQLModel-style) or DataFrame[YourSchema].
• Typed expressions: Expr and transform chains are validated and lowered in Rust; many errors fail fast at build/plan time.
• Familiar operations: select, filter, join, group_by, windows, melt/pivot, and pandas-flavored helpers where they help.
• Flexible materialization: row models via collect() / rows(), columnar dict[str, list], or Polars/PyArrow with the right extras.
• I/O: lazy read_* / aread_*, streaming writes, NDJSON/JSON Lines, Parquet, CSV, IPC, HTTP, SQL (SQLModel-first fetch_sqlmodel / write_sqlmodel, explicit string SQL fetch_sql_raw / write_sql_raw, or deprecated unprefixed names) — I/O overview, IO_SQL, SQLModel roadmap, and decision tree.
• JSON & struct columns: struct expressions, JSON encode/decode helpers, unnest/nested models — IO_JSON, SELECTORS.
• FastAPI (optional): shared executor lifespan, NDJSON streaming from astream(), OpenAPI-friendly columnar bodies, register_exception_handlers (503 / 400 / 422). Start with the golden path and FastAPI guide.

Install

pip install pydantable

Common extras:

pip install "pydantable[polars]"   # to_polars
pip install "pydantable[arrow]"    # to_arrow / Arrow constructors
pip install "pydantable[io]"       # full file I/O convenience (arrow + polars)
pip install "pydantable[sql]"      # SQLModel + SQLAlchemy: fetch_sqlmodel, write_sqlmodel, *_raw, …
pip install "pydantable[pandas]"   # pandas-flavored façade (pandas UI doc)
pip install "pydantable[fastapi]"  # FastAPI integration (pydantable.fastapi)

Quick start

from pydantable import DataFrameModel

class User(DataFrameModel):
    id: int
    age: int | None

df = User({"id": [1, 2], "age": [20, None]})
result = (
    df.with_columns(age2=df.age * 2)
    .filter(df.age > 10)
    .select("id", "age2")
)

print(result.to_dict())
print([r.model_dump() for r in result.collect()])

Output (exact values depend on filtering; this matches scripts/verify_doc_examples.py):

{'id': [1], 'age2': [40]}
[{'id': 1, 'age2': 40}]

Core concepts

Piece	Role
DataFrameModel	Table class with annotated columns (class Orders(DataFrameModel): ...).
DataFrame[Schema]	Generic API over your own Pydantic BaseModel.
Expr	Typed expressions in with_columns, filter, etc.
Errors	Ingest issues such as column length mismatch raise ColumnLengthMismatchError (ValueError subclass) from pydantable.errors — map to HTTP 400 in FastAPI via register_exception_handlers.

Static typing

• mypy: schema-evolving return types for many chains via the bundled mypy plugin (plugins in pyproject.toml).
• Pyright / Pylance: use committed stubs under typings/; for explicit targets, as_model(...) / try_as_model(...) / assert_model(...). See TYPING.

Rich column types (Literal, ipaddress, WKB, Annotated, …) are covered in SUPPORTED_TYPES.

Materialization: collect() / rows() → row models; to_dict() → dict[str, list]; to_polars() / to_arrow() with matching extras.

I/O at a glance

• DataFrameModel / DataFrame[Schema]: lazy read_* / aread_*, export_*, write_*, SQLModel I/O (fetch_sqlmodel, write_sqlmodel, …); eager materialize_* and SQL fetch_* / iter_* patterns live on pydantable.io — pass dict[str, list] into constructors for typed frames.
• Scripts: raw helpers (ScanFileRoot, iterators) on pydantable.io for glue code.
• SQL details: IO_SQL (recommended APIs, *_raw, deprecations) and SQLMODEL_SQL_ROADMAP (phased migration).
• Large files & NDJSON patterns: IO_JSON, IO_NDJSON, EXECUTION.

Validation controls

• Strict by default on constructors.
• Optional ingest controls: trusted_mode, ignore_errors, on_validation_errors.
• Missing optional fields: fill_missing_optional (default True).
• Validation presets: validation_profile=... (or __pydantable__ = {"validation_profile": "..."}).
• Per-column and nested strictness: {doc}STRICTNESS (field policies + profile defaults).

Documentation

Topic	Link
Docs home	pydantable.readthedocs.io
Map of all pages	DOCS_MAP
Quickstart	QUICKSTART
DataFrameModel	DATAFRAMEMODEL
Typing (mypy vs Pyright)	TYPING
I/O overview	IO_OVERVIEW
SQL (SQLModel, raw string SQL)	IO_SQL · SQLMODEL_SQL_ROADMAP
Pandas-like API	PANDAS_UI
FastAPI path	GOLDEN_PATH_FASTAPI → FASTAPI → FASTAPI_ENHANCEMENTS
Service ergonomics (OpenAPI, aliases, redaction)	SERVICE_ERGONOMICS
Custom dtypes	CUSTOM_DTYPES
Strictness	STRICTNESS
Cookbooks	Cookbook index (FastAPI, lazy pipelines, JSON logs, …)
Example multi-router app	docs/examples/fastapi/service_layout/ in this repo
Test helpers	pydantable.testing.fastapi — see FASTAPI
Execution & async	EXECUTION · MATERIALIZATION
Behavioral contract	INTERFACE_CONTRACT
Troubleshooting	TROUBLESHOOTING
Versioning	VERSIONING
Changelog	CHANGELOG

Development

Use a virtual environment at .venv in the repo root (the Makefile defaults to .venv/bin/python). Full contributor setup, Maturin/Rust builds, and release notes: DEVELOPER.

make check-full      # ruff, ty, pyright, typing snippet tests, Sphinx, Rust

License

MIT