polaguard 0.2.0

Lightning-fast local data contract validation via Polars & Pydantic V2.

polaguard

Lightning-fast local data contract validation for CSV, Parquet, and JSON files.
Built to protect local pipelines, software loops, and CI/CD runners before data hits the cloud.

⚡ Why Polaguard?

Most data contract engines are database-centric, slow to connect, and heavy. Polaguard shifts data quality left:

• Zero Infrastructure: No cloud dependencies, database logins, or heavy configuration.
• Blazing Fast: Vectorized execution handling millions of rows in milliseconds using Polars.
• Pipeline Native: Designed to block git commits and GitHub Action pipelines via automated exit codes.

🚀 Quick Start

1. Install

pip install polaguard

2. Auto-Generate a Contract Schema

Point Polaguard at a clean file. It will automatically infer your structures, formats, uniqueness, and null distributions.

polaguard init --file data/baseline.parquet --output contract.yaml

3. Check Incoming Batches

Instantly check incoming files against your established standards:

polaguard check --file data/new_batch.csv --contract contract.yaml

4. Use the Python API

from pathlib import Path
from polaguard import validate_file

result = validate_file(Path("data/new_batch.csv"), Path("contract.yaml"))
if not result.is_valid:
  print(result.errors)

5. Check CLI Version

polaguard --version

🛠️ Automated Integrations

Pre-Commit Hooks

Catch structure breaking data changes before making a git commit. Add this to your .pre-commit-config.yaml:

repos:
  - repo: https://github.com/osadose/polaguard
    rev: v0.2.0
    hooks:
      - id: polaguard
        args: ["check", "--file", "data/raw_inputs.csv", "--contract", "contract.yaml"]

📄 Schema Configuration & Constraints

Polaguard YAML contracts support dataset-level constraints, column validations, and custom SQL expression assertions.

Dataset-level Constraints

• min_columns (integer): Minimal number of columns required.
• min_rows (integer): Minimal number of rows required.
• allow_extra_columns (boolean): Whether to fail if the dataset contains columns not defined in the contract.

Column-level Validations

Under columns.<column_name>:

• type: One of int, float, str, bool, date, datetime.
• required (boolean): Failing check if the column is absent.
• unique (boolean): Evaluates if values must be distinct.
• null_threshold (float between 0.0 and 1.0): Permissible ratio of null values (e.g. 0.2 allows up to 20% nulls).
• regex (string): For str columns, regular expression format checking.
• allowed_values (list): Defines an enum of permitted values.
• min_value / max_value (any): Upper and lower boundary limits (for numeric, date, and datetime columns).
• min_length / max_length (integer): Length limits for character strings.

Custom SQL Expressions

Define a list of arbitrary SQL checks evaluated in Polars against the dataset under expressions:

expressions:
  - "age >= 18"
  - "start_date < end_date"
  - "revenue - cost > 0"

📜 License

This project is licensed under the MIT License — see the LICENSE file for details.