dataforge-profile 0.1.0

High-performance data profiling for pandas/pyarrow DataFrames

DataForge Profile

High-performance data profiling for Python, powered by Rust.

Installation

pip install dataforge-profile

Usage

import pandas as pd
from dataforge_profile import profile

# Profile a pandas DataFrame
df = pd.DataFrame({
    "id": range(1_000_000),
    "name": [f"user_{i}" for i in range(1_000_000)],
    "email": [f"user{i}@example.com" for i in range(1_000_000)],
    "age": [25] * 1_000_000,
})

result = profile(df)

# Access column statistics
print(result["columns"]["id"]["mean"])  # ~500000
print(result["columns"]["email"]["inferred_type"])  # "email"

Benchmark

Profiling 1M rows:

Tool	Time	Memory
ydata-profiling	~12s	~500MB
dataforge-profile	~0.14s	~50MB

~85x faster, 10x less memory

Requirements

• Python 3.9+
• pandas >= 1.5 or pyarrow >= 12.0

License

Apache 2.0