pavise 0.1.6

DataFrame validation library using Python Protocol for structural subtyping

Pavise

DataFrame validation library using Python Protocol for structural subtyping.

About the Name

A pavise was a large shield used by medieval crossbowmen, big enough to cover the entire body and provide strong protection.

Like its namesake, this library serves as a shield for your data. Whether you're working with small datasets or big data, pavise protects your code with type safety and validation.

Features

• Use Python Protocol to define DataFrame schemas
• DataFrame[Schema] type annotation for static type checking
• Structural subtyping: validate only required columns, ignore extra columns
• Covariant type parameters: DataFrame[ChildSchema] is compatible with DataFrame[ParentSchema]
• Optional runtime validation
• No inheritance required
• Support for both pandas and polars backends

Documentation

Full documentation is available at https://pavise.readthedocs.io/

Installation

# For pandas support
pip install pavise[pandas]

# For polars support
pip install pavise[polars]

# For both
pip install pavise[all]

Usage

Pandas Backend

from typing import Protocol
import pandas as pd
from pavise.pandas import DataFrame

class UserSchema(Protocol):
    name: str
    age: int

# Runtime validation when creating DataFrame[Schema]
raw_df = pd.DataFrame({'name': ['Alice', 'Bob'], 'age': [30, 17]})
validated_df = DataFrame[UserSchema](raw_df)  # Validates column types at runtime

# Type hints work with static type checkers (mypy, pyright, etc.)
def process_users(df: DataFrame[UserSchema]) -> DataFrame[UserSchema]:
    return df[df['age'] >= 18]

result = process_users(validated_df)

Polars Backend

from typing import Protocol
import polars as pl
from pavise.polars import DataFrame

class UserSchema(Protocol):
    name: str
    age: int

# Runtime validation when creating DataFrame[Schema]
raw_df = pl.DataFrame({'name': ['Alice', 'Bob'], 'age': [30, 17]})
validated_df = DataFrame[UserSchema](raw_df)  # Validates column types at runtime

# Type hints work with static type checkers (mypy, pyright, etc.)
def process_users(df: DataFrame[UserSchema]) -> DataFrame[UserSchema]:
    return df.filter(df['age'] >= 18)

result = process_users(validated_df)

Structural Subtyping

from typing import Protocol
import pandas as pd
from pavise.pandas import DataFrame

class UserSchema(Protocol):
    name: str

class UserWithEmailSchema(Protocol):
    name: str
    email: str

def process_user(df: DataFrame[UserSchema]) -> None:
    print(df['name'])

# This works! UserWithEmailSchema has all required columns of UserSchema
df = DataFrame[UserWithEmailSchema](pd.DataFrame({
    'name': ['Alice'],
    'email': ['[e-mail]']
}))
process_user(df)  # OK - covariant type parameter

Using Validators

Add validators using typing.Annotated to enforce data quality constraints:

from typing import Annotated, Protocol
import pandas as pd
from pavise.pandas import DataFrame
from pavise.validators import Range, Regex

class UserSchema(Protocol):
    name: str
    age: Annotated[int, Range(0, 150)]
    email: Annotated[str, Regex(r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$')]

# Valid data passes validation
df = pd.DataFrame({
    'name': ['Alice', 'Bob'],
    'age': [25, 30],
    'email': ['[e-mail]', '[e-mail]']
})
validated_df = DataFrame[UserSchema](df)  # OK

# Invalid data raises ValidationError
invalid_df = pd.DataFrame({
    'name': ['Charlie'],
    'age': [200],  # Exceeds maximum age
    'email': ['invalid-email']  # Invalid email format
})
DataFrame[UserSchema](invalid_df)  # ValidationError

Union Types

Union types allow columns to accept multiple different types:

from typing import Protocol, Union
import pandas as pd
from pavise.pandas import DataFrame

class MixedSchema(Protocol):
    code: Union[int, str]  # Can be int or str
    value: float

# Accept int values
df1 = pd.DataFrame({'code': [1, 2, 3], 'value': [1.0, 2.0, 3.0]})
validated1 = DataFrame[MixedSchema](df1)  # OK

# Accept str values
df2 = pd.DataFrame({'code': ['A', 'B', 'C'], 'value': [1.0, 2.0, 3.0]})
validated2 = DataFrame[MixedSchema](df2)  # OK

# Accept mixed int/str values
df3 = pd.DataFrame({'code': [1, 'B', 3, 'D'], 'value': [1.0, 2.0, 3.0, 4.0]})
validated3 = DataFrame[MixedSchema](df3)  # OK

# Union with None for nullable union types
class NullableUnionSchema(Protocol):
    code: Union[int, str, None]  # Can be int, str, or None

df4 = pd.DataFrame({'code': [1, 'B', None, 4]})
validated4 = DataFrame[NullableUnionSchema](df4)  # OK

Extra Columns are Ignored

from typing import Protocol
import pandas as pd
from pavise.pandas import DataFrame

class SimpleSchema(Protocol):
    a: int

# Extra columns are ignored during validation
df = pd.DataFrame({
    'a': [1, 2, 3],
    'b': ['x', 'y', 'z'],  # Extra column - ignored
    'c': [10.0, 20.0, 30.0]  # Extra column - ignored
})

validated = DataFrame[SimpleSchema](df)  # OK

Supported Types

Basic Types

• int - Integer values
• float - Floating point values
• str - String values
• bool - Boolean values

Date/Time Types

• datetime - Date and time values
• date - Date-only values
• timedelta - Time duration values

Generic Types

• Optional[T] - Nullable types (e.g., Optional[int], Optional[str])
• Union[T1, T2, ...] - Union types allowing multiple types (e.g., Union[int, str], Union[int, str, float])
  • Can be combined with None for nullable unions: Union[int, str, None]
• Literal[...] - Specific literal values (e.g., Literal["a", "b", "c"], Literal[1, 2, 3])
• NotRequiredColumn[T] - Optional columns (e.g., NotRequiredColumn[int], NotRequiredColumn[Optional[str]])

Backend-Specific Types

• pandas: pd.CategoricalDtype, pd.Int64Dtype, and other Extension dtypes
• polars: pl.Categorical, pl.Int64, and other polars DataTypes

Development

# Install with dev dependencies (includes both pandas and polars)
uv pip install -e ".[dev]"

# Run all tests
uv run pytest

# Run tests for specific backend
uv run pytest tests/test_pandas.py
uv run pytest tests/test_polars.py

Testing with tox

# Run tests for all Python versions and backends
tox

# Run tests for specific environment
tox -e py312-pandas    # Test pandas backend with Python 3.12
tox -e py312-polars    # Test polars backend with Python 3.12
tox -e py312-all       # Test both backends with Python 3.12

# Run linting
tox -e lint

# Run type checking
tox -e type

# Available Python versions: py39, py310, py311, py312
# Available backends: pandas, polars, all