datapruning 2.0.0

Intelligent data pruning for ML datasets

DataPruning

Intelligent dataset pruning for ML — reduces dataset size by selecting the most informative rows.

Installation

pip install datapruning

Note: PyTorch (~2 GB) is a required dependency.

Requirements

• Python >= 3.10
• PyTorch 2.0+
• Pandas 2.0+
• NumPy 1.24+

Limits

• Minimum: 1,000 rows per dataset
• Maximum: 300,000 rows per dataset

Usage

import pandas as pd
from datapruning import Pruner

df = pd.read_csv("dataset.csv")
p = Pruner(df)
filtered = p.prune(target_col="label", keep_ratio=0.5)

print(f"Kept {len(filtered)} / {len(df)} rows")

Features

• Runs locally, no data upload
• Pre-training dataset optimization
• Pandas DataFrame input / output
• Compiled processing module

Links

• Website: datapruning.com

License

Proprietary. See LICENSE file.