dfdrift
A DataFrame schema drift detection and alerting library for pandas DataFrames.
Features
- Schema Tracking: Automatically save DataFrame schemas with location information (file:line)
- Change Detection: Detect schema changes between executions and alert when differences are found
- Configurable Storage: Support for local file storage and Google Cloud Storage with extensible interface for future cloud storage
- Configurable Alerting: Built-in stderr alerter and Slack integration with extensible interface for future integrations
Installation
# Basic installation
pip install dfdrift
# With Slack support
pip install dfdrift[slack]
# With Google Cloud Storage support
pip install dfdrift[gcs]
# With all optional features
pip install dfdrift[slack,gcs]
# Development installation
uv pip install -e .
Usage
dfdrift offers two ways to validate DataFrames:
1. Import Replacement
Simply replace your pandas import with dfdrift.pandas:
import dfdrift.pandas as pd
# Configure validation (optional - uses default settings if omitted)
pd.configure_validation()
# All DataFrame operations are automatically validated
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'age': [25, 30, 35],
'city': ['Tokyo', 'Osaka', 'Kyoto']
})
# Schema automatically saved with location info
2. Explicit Validation
import pandas as pd
import dfdrift
# Create a validator instance
validator = dfdrift.DfValidator()
# Validate a DataFrame manually
df = pd.DataFrame({
'name': ['Alice', 'Bob', 'Charlie'],
'age': [25, 30, 35],
'city': ['Tokyo', 'Osaka', 'Kyoto']
})
validator.validate(df)
Configuration
Custom Storage
Local File Storage
import dfdrift.pandas as pd
# Use custom local directory
pd.configure_validation(
storage=dfdrift.LocalFileStorage("./my_schemas")
)
Google Cloud Storage
import dfdrift.pandas as pd
# Configure GCS storage (requires: pip install dfdrift[gcs])
# Set GCS_BUCKET and optionally GCS_PREFIX environment variables
gcs_storage = dfdrift.GcsStorage() # bucket and prefix from env vars
pd.configure_validation(storage=gcs_storage)
# Or pass parameters directly
gcs_storage = dfdrift.GcsStorage(
bucket="my-dfdrift-bucket",
prefix="schemas/production" # Optional, defaults to "dfdrift"
)
pd.configure_validation(storage=gcs_storage)
GCS Authentication: Use one of the following methods:
- Set
GOOGLE_APPLICATION_CREDENTIALSenvironment variable to service account key file - Use Application Default Credentials:
gcloud auth application-default login - Use Workload Identity in GKE/Cloud Run environments
Custom Alerter
Stderr Alerter (Default)
import dfdrift.pandas as pd
# Built-in stderr alerter (default)
pd.configure_validation(alerter=dfdrift.StderrAlerter())
Slack Alerter
import dfdrift.pandas as pd
# Configure Slack alerts (requires: pip install dfdrift[slack])
# Set SLACK_BOT_TOKEN and SLACK_CHANNEL environment variables
slack_alerter = dfdrift.SlackAlerter() # Uses env vars
pd.configure_validation(alerter=slack_alerter)
# Or specify channel argument (token from env var)
slack_alerter = dfdrift.SlackAlerter(channel="#data-alerts")
pd.configure_validation(alerter=slack_alerter)
# Or pass both token and channel directly (not recommended for production)
slack_alerter = dfdrift.SlackAlerter(
channel="#data-alerts",
token="xoxb-your-bot-token"
)
pd.configure_validation(alerter=slack_alerter)
Custom Alerter
import dfdrift
# Implement your own alerter
class CustomAlerter(dfdrift.Alerter):
def alert(self, message, location_key, old_schema, new_schema):
# Send to email, webhook, etc.
pass
pd.configure_validation(alerter=CustomAlerter())
Schema Change Detection
When a DataFrame schema changes between executions, dfdrift will automatically detect and alert:
- Added columns: New columns that weren't in the previous schema
- Removed columns: Columns that existed before but are now missing
- Type changes: When a column's dtype changes (e.g., int64 → object)
- Shape changes: When the DataFrame dimensions change
Example alert output:
WARNING: DataFrame schema changed at /path/to/file.py:25. Changes: Added columns: ['new_col']; Column 'age' dtype changed: int64 → object
Location: /path/to/file.py:25
Examples
See the samples/ directory for usage examples:
samples/sample.py: Explicit validationsamples/sample_custom_path.py: Custom storage pathsamples/sample_changing_schema.py: Schema change detection demosamples/sample_pandas_import.py: Import replacement
Architecture
Storage Interface
class SchemaStorage(ABC):
def save_schema(self, location_key: str, schema: Dict[str, Any]) -> None:
pass
def load_schemas(self) -> Dict[str, Any]:
pass
Alerter Interface
class Alerter(ABC):
def alert(self, message: str, location_key: str, old_schema: Dict[str, Any], new_schema: Dict[str, Any]) -> None:
pass
Schema Format
Schemas are stored as JSON with the following structure:
{
"/path/to/file.py:line_number": {
"columns": {
"column_name": {
"dtype": "int64",
"null_count": 0,
"total_count": 100
}
},
"shape": [100, 3]
}
}
Development
Run the samples to test functionality:
# Import replacement
uv run python samples/sample_pandas_import.py # Run twice to see alerts
# Explicit validation
uv run python samples/sample.py
# Test schema change detection
uv run python samples/sample_changing_schema.py # Run twice to see alerts
Metadata
Release files for dfdrift 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| dfdrift-0.5.0.tar.gz | 16.2 kB | Without provenance | Details for dfdrift-0.5.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dfdrift-0.5.0-py3-none-any.whl | Python 3 | none | any | Details for dfdrift-0.5.0-py3-none-any.whl |
Total release size: 24.9 kB
Release files / dfdrift-0.5.0.tar.gz
| Download URL | dfdrift-0.5.0.tar.gz |
|---|---|
| Size | 16.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
fa61d58cb06d1d66c7ade6a8f1461c2da9f1c4f2c41d1c9f3c329460a9fe128f
|
|
BLAKE2b-256 checksum How to use checksums |
e954b2294647d07aa95f00ea3f4980e157857c2ca511552caa6d56f0df5b975d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.21
|
Release files / dfdrift-0.5.0-py3-none-any.whl
| Download URL | dfdrift-0.5.0-py3-none-any.whl |
|---|---|
| Size | 8.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
eb11f0df85d9be796bfc9fd067bb4ae1115ec58b6b7dfe482309a06515b98dc0
|
|
BLAKE2b-256 checksum How to use checksums |
ee5c7b10e5e8229ebb98163107b9716e3fa12bcb19794435bd96f3f5a7f3eef3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.21
|
Security for dfdrift 0.5.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.