colnade 0.8.0

A statically type-safe DataFrame abstraction layer

Colnade

A statically type-safe DataFrame abstraction layer for Python.

Colnade replaces string-based column references (pl.col("age")) with typed descriptors (Users.age), so column misspellings and type mismatches are caught by your type checker — before your code runs.

Works with ty, mypy, and pyright. No plugins, no code generation.

Installation

pip install colnade colnade-polars

Colnade requires Python 3.10+. Install the backend adapter for your engine:

Backend	Install
Polars	pip install colnade-polars
Pandas	pip install colnade-pandas
Dask	pip install colnade-dask

Quick Start

1. Define a schema

import colnade as cn

class Users(cn.Schema):
    id: cn.Column[cn.UInt64]
    name: cn.Column[cn.Utf8]
    age: cn.Column[cn.UInt64]
    score: cn.Column[cn.Float64]

2. Create or read typed data

from colnade_polars import from_rows, read_parquet

# From Python data — schema drives dtype coercion
df = from_rows(Users, [
    Users.Row(id=1, name="Alice", age=30, score=85.0),
    Users.Row(id=2, name="Bob", age=25, score=92.5),
])

# Or from files
df = read_parquet("users.parquet", Users)
# df is DataFrame[Users] — the type checker knows the schema

3. Transform with full type safety

# Column references are attributes, not strings
result = (
    df.filter(Users.age > 25)
      .sort(Users.score.desc())
      .select(Users.name, Users.score)
)

4. Bind to an output schema

class UserSummary(cn.Schema):
    name: cn.Column[cn.Utf8]
    score: cn.Column[cn.Float64]

output = result.cast_schema(UserSummary)
# output is DataFrame[UserSummary]

Safety Model

Colnade catches errors at three levels:

1. In your editor — misspelled columns, schema mismatches at function boundaries, and nullability violations are flagged by your type checker (ty, pyright, mypy) before code runs
2. At data boundaries — runtime validation ensures files and external data match your schemas (columns, types, nullability) and that expressions reference columns from the correct schema
3. On your data values — Field() constraints validate domain invariants like ranges, patterns, and uniqueness

Where static safety ends: Static checking covers column references and schema-preserving operations (filter, sort, with_columns). Schema-transforming operations (select, group_by) return DataFrame[Any] — you call cast_schema() to assert the output schema, and runtime validation (if enabled) verifies it. No type checker plugin is needed, but this means output schemas aren't inferred statically. See Type Checker Integration for the full list of what is and isn't checked.

Key Features

Type-safe column references

Column references are class attributes verified by the type checker at lint time:

Users.name   # Column[Utf8] — valid
Users.naem   # ty error: Class `Users` has no attribute `naem`

Schema-preserving operations

Operations that don't change the schema (filter, sort, limit, with_columns) preserve the type parameter:

def process(df: DataFrame[Users]) -> DataFrame[Users]:
    return df.filter(Users.age > 25).sort(Users.score.desc())

Typed expressions

Column descriptors build an expression tree with typed operators:

Users.age > 18          # Expr[Bool] — comparison
Users.score * 2         # Expr[Float64] — arithmetic
(Users.age > 18) & (Users.score > 80)  # Expr[Bool] — logical
Users.name.str_starts_with("A")        # Expr[Bool] — string method

Aggregations

class UserStats(cn.Schema):
    name: cn.Column[cn.Utf8]
    avg_score: cn.Column[cn.Float64]
    user_count: cn.Column[cn.UInt64]

result = df.group_by(Users.name).agg(
    Users.score.mean().alias(UserStats.avg_score),
    Users.id.count().alias(UserStats.user_count),
)

Conditional expressions

df.with_columns(
    cn.when(Users.age > 65).then(cn.lit("senior")).otherwise(cn.lit("standard")).alias(Users.tier)
)

Vertical concatenation

combined = cn.concat(df_jan, df_feb, df_mar)  # DataFrame[Sales]

Null handling

# Fill nulls, filter nulls, check nulls
df.with_columns(Users.score.fill_null(0.0).alias(Users.score))
df.filter(Users.score.is_not_null())
df.drop_nulls(Users.score)

Joins with typed output

joined = users.join(orders, on=Users.id == Orders.user_id)
# JoinedDataFrame[Users, Orders] — both schemas accessible

# mapped_from tells cast_schema which source column maps to each target column
class UserOrders(cn.Schema):
    user_name: cn.Column[cn.Utf8] = cn.mapped_from(Users.name)
    amount: cn.Column[cn.Float64]  # same name as Orders.amount — matched automatically

result = joined.cast_schema(UserOrders)

Schema-polymorphic utility functions

Write generic functions that work with any schema:

from colnade.schema import S

def first_n(df: cn.DataFrame[S], n: int) -> cn.DataFrame[S]:
    return df.head(n)

# Works with any schema — type preserved
users_subset: cn.DataFrame[Users] = first_n(users_df, 10)

Struct and List support

class Address(cn.Schema):
    city: cn.Column[cn.Utf8]
    zip_code: cn.Column[cn.Utf8]

class UserProfile(cn.Schema):
    name: cn.Column[cn.Utf8]
    address: cn.Column[cn.Struct[Address]]
    tags: cn.Column[cn.List[cn.Utf8]]
    tag_count: cn.Column[cn.UInt32]

# Access nested data
df.filter(UserProfile.address.field(Address.city) == "New York")
df.with_columns(UserProfile.tags.list.len().alias(UserProfile.tag_count))

Value-level constraints

import colnade as cn

class Users(cn.Schema):
    id: cn.Column[cn.UInt64] = cn.Field(unique=True)
    age: cn.Column[cn.UInt64] = cn.Field(ge=0, le=150)
    email: cn.Column[cn.Utf8] = cn.Field(pattern=r"^[^@]+@[^@]+\.[^@]+$")
    status: cn.Column[cn.Utf8] = cn.Field(isin=["active", "inactive"])

    @cn.schema_check
    def adult(cls):
        return Users.age >= 18

# Validate with df.validate() or auto-validate at the FULL level
cn.set_validation(cn.ValidationLevel.FULL)

Lazy execution

from colnade_polars import scan_parquet

lazy = scan_parquet("users.parquet", Users)
# LazyFrame[Users] — builds a query plan

result = lazy.filter(Users.age > 25).sort(Users.score.desc()).collect()
# Executes the optimized query plan

Performance

Colnade translates an expression AST into engine-native calls, adding a fixed cost per operation that doesn't grow with dataset size. For Polars and Pandas, this overhead is below the measurement noise floor — benchmarked pipelines (filter + sort + select) show no measurable difference from 100 to 1M rows. Dask graph construction adds ~200–300 us per operation, negligible compared to .compute() time.

Validation (STRUCTURAL, FULL) adds measurable cost at data boundaries and is designed for development/CI, not production hot paths. See the full benchmark results for details.

Type Checker Error Showcase

Colnade catches real errors at lint time. Here are actual error messages from ty:

Comparison with Existing Solutions

Feature	Colnade	Pandera	StaticFrame	Patito	Narwhals
Column refs checked statically	Named attrs	No	Positional types	No	No
Schema preserved through ops	Through ops¹	At boundaries²	No	No	No
Works with existing engines	Polars, Pandas, Dask	Pandas, Polars, others	Own engine	Polars only	Many engines
No plugins or code gen	Yes	Optional mypy plugin	Yes	Yes	Yes
Generic utility functions	Yes	No	No	No	No
Struct/List typed access	Yes	No	No	No	No
Lazy execution support	Yes	No	No	No	Yes
Value-level constraints	Field()	Check	No	Pydantic validators	No
Maturity / ecosystem	New (v0.7)	Mature, large community	Mature	Small	Growing fast
Engine breadth	3 backends	4+ backends	Own engine	1 backend	6+ backends
select/group_by output typing	DataFrame[Any]³	Decorator-checked	Positional types	No	No

¹ Schema-preserving ops (filter, sort, with_columns) retain DataFrame[S]. Schema-transforming ops (select, group_by) return DataFrame[Any] — use cast_schema() to bind. ² Pandera's @check_types validates schemas at function boundaries via decorator, but column references within function bodies remain unchecked strings. ³ cast_schema() re-binds at runtime. A type checker plugin could theoretically infer output schemas, but Colnade intentionally avoids plugin coupling.

See Detailed Comparisons for a fuller discussion of tradeoffs.

Documentation

Full documentation is available at colnade.com, including:

• Getting Started — installation and quick start
• User Guide — concepts, schemas, expressions, joins
• Tutorials — worked examples with real data
• API Reference — auto-generated from source

Examples

Runnable examples are in the examples/ directory:

• basic_usage.py — Schema definition, filter, select, aggregate
• null_handling.py — Nullable columns, fill_null, drop_nulls
• joins.py — Joining DataFrames, JoinedDataFrame, cast_schema
• generic_functions.py — Schema-polymorphic utility functions
• nested_types.py — Struct and List column operations
• full_pipeline.py — Complete ETL pipeline example

License

MIT