lakeflush 0.1.0

lakeflush optimizes data lakes by consolidating small files into larger bundles for big data workloads. Reduces storage overhead and improves processing efficiency.

lakeflush

lakeflush optimizes data lakes by consolidating small files into larger bundles for big data workloads. Reduces storage overhead and improves processing efficiency.

Efficiently consolidate millions of small files into larger bundles to solve common big data challenges:

✅ Reduces storage overhead – Minimize metadata bloat in HDFS/S3
✅ Boosts processing speed – Fewer files = Faster Spark/Hadoop jobs
✅ Seamless integration – Works with existing data lakes (S3, on-prem)
✅ Smart bundling – Configurable size thresholds and compression
✅ Mult-format Support – Supports Text, JSON and CSV file format

Ideal for:

• IoT sensor data
• Log file aggregation
• ML training datasets
• Data lake optimization

Works on Python 3.11+

pip install lakeflush