dataprof 0.5.20

Fast, lightweight data profiling and quality assessment library

dataprof

High-Performance Data Profiler with ISO 8000/25012 Quality Metrics

Profile datasets larger than your RAM in seconds on your laptop with Rust and SIMD.

💡 Key Features

• Blazing Fast & Low Memory: Written in Rust, using optional SIMD acceleration and streaming to process huge files with < 100MB memory usage.
• Smart Adaptive Engines: Automatically chooses between Batch and Streaming engines based on your system capabilities and data size.
• ISO 8000/25012 Metrics: Robust data quality scoring alongside type distributions and value frequency.
• PII & Semantic Detection: Automatically identifies Emails, IPs, IBANs, Credit Cards, and more.
• Connects Everywhere: CSV, Parquet, JSON/L, and SQL Databases (Postgres, MySQL, SQLite).
• Privacy First: Local execution; data never leaves your machine.

🚀 Quick Start

Python (via pip)

pip install dataprof

import dataprof

# Connect, profile, and export an HTML dashboard
dataprof.profile("huge_dataset.csv").save("report.html")

CLI (via cargo - Advanced/Rust users)

cargo install dataprof

# Profile directly from your terminal
dataprof-cli report huge_data.parquet -o report.html

📊 Visual Reports

dataprof generates rich, interactive HTML reports for single files or batches, visualizing data quality distributions, anomalies, and insights.

Click to view Report Dashboards

Interactive Demo

Single File Analysis

Batch Processing Dashboard

📚 Documentation

• Python API Reference
• CLI Reference Guide
• Database Connectors

🛠️ Advanced Usage (Python)

Batch Processing

# Process an entire folder natively in parallel
result = dataprof.batch_analyze_directory("/data_folder", recursive=True)
print(f"Processed at {result.files_per_second:.1f} files/sec")

Database Integration (Async)

# Profile a SQL query directly
await dataprof.analyze_database_async(
    "postgresql://user:pass@localhost/db",
    "SELECT * FROM sales_data_2024"
)

💻 Development & Contributing

We welcome contributions! Please see CONTRIBUTING.md and our Code of Conduct.

git clone https://github.com/AndreaBozzo/dataprof.git
cd dataprof

# Default build (CSV/JSON only)
cargo build --release

# With extensive formats and DB functionality
cargo build --release --features "arrow parquet postgres mysql sqlite"

# To develop the Python API locally
maturin develop --features "python-async database postgres"

⚖️ License

Dual-licensed under either the MIT License or the Apache License, Version 2.0.