koala-diff 0.2.0

High-performance data diff tool in Rust.

Koala Diff

Blazingly Fast Data Comparison for the Modern Stack.

---

Koala Diff is the "git diff" for your data lake. It compares massive datasets (CSV, Parquet, JSON) instantly to find added, removed, and modified rows.

Built in Rust 🦀 for speed, wrapped in Python 🐍 for ease-of-use. It streams data to compare datasets larger than RAM and generates beautiful HTML reports.

🚀 Why Koala Diff?

• Zero-Copy Streaming: Compare 100GB files on a laptop without crashing RAM.
• Rust-Powered Analytics: Go beyond row counts. Track Value Variance, Null Drift, and Match Integrity per column.
• Professional Dashboards: Auto-generates premium, stakeholder-ready HTML reports with status badges and join attribution.
• Deep-Dive API: Extract mismatched records as Polars DataFrames for instant remediation.

---

📦 Installation

pip install koala-diff

⚡ Quick Start

1. Generate a "Pro" Report

from koala_diff import DataDiff, HtmlReporter

# Initialize with primary keys
differ = DataDiff(key_columns=["user_id"])

# Run comparison
result = differ.compare("source.parquet", "target.parquet")

# Generate a professional dashboard
reporter = HtmlReporter("data_quality_report.html")
reporter.generate(result)

2. Mismatch Deep-Dive

Need to fix the data? Pull the exact differences directly into Python:

# Get a Polars DataFrame of ONLY mismatched rows
mismatch_df = differ.get_mismatch_df()

# Analyze variance or push to a remediation pipeline
print(mismatch_df.head())

2. CLI Usage (Coming Soon)

koala-diff production.csv staging.csv --key user_id --output report.html

📊 Performance Benchmarks

Dataset Size	Tool	Time	Memory
10M Rows	Pandas	🐢 120s	16GB (OOM)
	Koala Diff	🚀 2.5s	250MB
100M Rows	Spark	🚜 45s	Cluster
	Koala Diff	🚀 18s	450MB

> Benchmarks run on MacBook Pro M3 Max.

🏗 Architecture

Koala Diff uses a streaming hash-join algorithm implemented in Rust:

1. Reader: Lazy Polars scan of both datasets.
2. Hasher: XXHash64 computation of row values (SIMD optimized).
3. Differ: fast set operations to classify rows as Added, Removed, or Modified.
4. Reporter: Jinja2 rendering of results.

🤝 Contributing

We welcome contributions! Whether it's a new file format reader, a performance optimization, or a documentation fix.

1. Check the Issues.
2. Read our Contribution Guide.

📄 License

MIT © 2026 godalida - KoalaDataLab