StatGuardian

Data quality validation. 13x faster than pandera.

Stop data quality issues from reaching production. StatGuardian validates data at runtime, instantly catching schema violations, type errors, and anomalies.

30-Second Start

from statguardian import validate

# Define your schema
schema = {
    "user_id": int,
    "email": str,
    "age": {"type": int, "min": 0, "max": 150},
}

# Validate data
result = validate(df, schema)
if not result.is_valid:
    print(result.violations)

Why StatGuardian?

Real-World Use Cases

E-commerce Order Validation

schema = {
    "order_id": str,
    "amount": {"type": float, "min": 0},
    "status": {"enum": ["pending", "shipped", "delivered"]},
}
validate(orders_df, schema)

ML Feature Pipeline

schema = {
    "feature_x": {"type": float, "not_null": True},
    "feature_y": {"type": float, "mean": 0, "std": 1},
}
result = validate(features, schema)

Data Lake Monitoring

result = validate(incoming_data, schema)
if result.has_drift:
    alert("Schema changed!")

Key Capabilities

• 13x speed advantage over pandera
• Type checking with detailed error messages
• Automatic drift detection
• Anomaly detection built-in
• Supports Pandas, Polars, DuckDB with identical code
• Zero configuration—just Python

Performance

StatGuardian processes 1M rows in 0.3s (vs pandera's 4.2s).

Features

Core Validation

• Type validation (int, float, str, bool, datetime, etc.)
• Min/max constraints for numeric types
• Enum validation for categorical data
• Null/not-null constraints
• Pattern matching for strings (regex)
• Custom validation functions
• Composite constraints (multiple rules per field)

Data Quality Analysis

• Automatic drift detection (schema changes)
• Anomaly detection (outliers, unexpected values)
• Statistical profiling (mean, std, quartiles)
• Missing value reporting
• Duplicate detection

Framework Support

• Pandas DataFrames (primary target)
• Polars DataFrames (full compatibility)
• DuckDB relations (streaming support)
• NumPy arrays (optional)
• Unified API across all frameworks

Performance & Scale

• Rust core for 13x speedup
• Streaming validation (memory-efficient)
• Batch processing (optimal for large datasets)
• Zero-copy operations where possible

Requirements

• Python: 3.10+
• Core: Rust-powered validation engine (precompiled)
• Data Frameworks:
  • pandas ≥1.3.0 (primary)
  • polars ≥0.19.0 (optional)
  • duckdb ≥0.8.0 (optional)
• Optional: numpy ≥1.20.0 (for array support)
• Precompiled: Wheels for macOS, Linux, Windows (all Python 3.10-3.13)

Examples

Basic Type Validation

from statguardian import validate

# Simple schema
schema = {
    "user_id": int,
    "email": str,
    "created_at": "datetime",
}

result = validate(df, schema)
print(f"Valid: {result.is_valid}")
print(f"Violations: {result.violations}")

Constraint Validation

schema = {
    "age": {"type": int, "min": 0, "max": 150},
    "email": {"type": str, "pattern": r"^[\w\.-]+@[\w\.-]+\.\w+$"},
    "status": {"enum": ["active", "inactive", "pending"]},
    "balance": {"type": float, "min": 0},
}

result = validate(transactions, schema)
if not result.is_valid:
    for violation in result.violations:
        print(f"Row {violation['row']}: {violation['message']}")

Drift & Anomaly Detection

# Detect schema changes
result = validate(new_data, schema)
if result.has_drift:
    print(f"New columns: {result.new_fields}")
    print(f"Missing columns: {result.missing_fields}")

# Detect anomalies
if result.anomalies:
    print(f"Outlier rows: {result.anomaly_rows}")

Multi-Framework Validation

import pandas as pd
import polars as pl

# Pandas
df_pd = pd.read_csv("data.csv")
result_pd = validate(df_pd, schema)

# Polars (identical code)
df_pl = pl.read_csv("data.csv")
result_pl = validate(df_pl, schema)

# Both return same validation results

API Reference

Core Functions

• validate(data, schema) -> ValidationResult

  • Validates data against schema
  • Returns detailed violations report
  • Supports Pandas, Polars, DuckDB

• ValidationResult

  • .is_valid: Boolean flag
  • .violations: List of violations
  • .has_drift: Boolean (schema changed)
  • .anomalies: List of anomaly indices
  • .statistics: Profiling stats (count, mean, std, etc.)

Schema Constraints

• Type: "int", "float", "str", "bool", "datetime"
• Numeric: min, max, mean, std
• Categorical: enum (allowed values)
• String: pattern (regex)
• Nullability: not_null (True/False)
• Custom: custom_fn(value) -> bool

Installation

pip install statguardian
# or with uv
uv pip install statguardian

# Verify installation
statguardian --version

For development:

git clone https://github.com/Mullassery/StatGuardian
cd StatGuardian
pip install -e ".[dev]"
pytest

Documentation

• API Reference
• Examples
• Contributing

License

MIT License - See LICENSE