DataPruning
Intelligent dataset optimization — reduces dataset size by selecting the most informative rows while preserving model quality.
Installation
pip install datapruning
For image/text multimodal support (CLIP encoding):
pip install datapruning[clip]
Quick Start
CSV / Tabular Data
import pandas as pd
from datapruning.sdk import DatasetEngine
df = pd.read_csv("dataset.csv")
engine = DatasetEngine(df, target_col="label")
# Analyze the dataset
analysis = engine.analyze()
# Get a recommendation
rec = engine.recommend()
print(f"Best method: {rec.display_name} ({rec.confidence}% confidence)")
# Optimize — keep 50% of most informative rows
result = engine.optimize(keep_ratio=0.5)
print(f"Kept {result['runtime_seconds']:.1f}s — {result['explainability']['rows_removed']} rows removed")
# Download the optimized dataset
result["optimized_df"].to_csv("optimized.csv", index=False)
Image Datasets
from datapruning.sdk import DatasetEngine
# Create engine from a folder of images
engine = DatasetEngine.from_images(
folder_path="./photos",
labels=[0, 1, 0, 1, ...], # one label per image
keep_ratio=0.5,
)
# Analyze and optimize
analysis = engine.analyze()
result = engine.optimize(keep_ratio=0.5)
result["optimized_df"].to_csv("optimized_images.csv", index=False)
Text Datasets
engine = DatasetEngine.from_texts(
texts=["label A sample", "label B sample", ...],
labels=[0, 1, 0, 1, ...],
)
result = engine.optimize(keep_ratio=0.5)
Algorithms
DataPruning selects the best algorithm automatically based on dataset characteristics:
| Method | What it does | Best for |
|---|---|---|
| Margin Optimization | Analyzes classification confidence gap via cross-validation, keeps boundary samples | General tabular data |
| Importance Scoring | Measures sample difficulty via model prediction confidence across CV folds | Quick baseline, any dataset |
| Diversity Selection | Picks the most representative subset (K-Center) | High-dimensional data |
| Noise Filtering | Drops redundant and potentially mislabeled samples via CV-based difficulty | Noisy/messy data |
| Full Pipeline | Combines all signals for maximum quality | Complex datasets |
All scoring methods use out-of-fold (OOF) model logits — each sample is scored by a model that never saw it during training, producing unbiased difficulty estimates. The optimizer also includes a "must beat random" gate that validates the selected strategy actually improves over random subsampling on an internal holdout; if not, it safely falls back to random selection.
Limits
- Minimum: 1,000 rows (tabular) or 10 images
- Maximum: 300,000 rows (tabular) or 1,000 images
- Supported image formats: JPG, PNG, BMP, GIF, TIFF, WebP
Requirements
- Python >= 3.10
- Pandas 2.0+
- NumPy 1.24+
- scikit-learn 1.3+
Optional (for image/text support):
- PyTorch 2.0+
- transformers 4.30+
- Pillow 9.0+
Features
- Runs locally — no data uploaded anywhere
- Auto-detects task type (classification/regression) and target column
- Auto-encodes categorical/string columns
- 5 optimization algorithms with automatic selection
- Preserves minority class balance
- CSV and image dataset support
- Pure Python — works on any platform
Links
- Website: datapruning.com
- PyPI: pypi.org/project/datapruning
License
Proprietary. See LICENSE file.
Metadata
Release files for datapruning 2.1.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| datapruning-2.1.4.tar.gz | 250.4 kB | Without provenance | Details for datapruning-2.1.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| datapruning-2.1.4-cp314-cp314-win_amd64.whl | CPython 3.14 | CPython 3.14 | Windows x86-64 | Details for datapruning-2.1.4-cp314-cp314-win_amd64.whl |
Total release size: 427.9 kB
Release files / datapruning-2.1.4.tar.gz
| Download URL | datapruning-2.1.4.tar.gz |
|---|---|
| Size | 250.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dbff69bc4478a47cd2dab25dd6ab688500c776eaaa2e766358c976c894e262a2
|
|
BLAKE2b-256 checksum How to use checksums |
5de6a34f98265ca7b6bf0dc28d121f253d8e29fad3a183729191c1a2d7a418a4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
Release files / datapruning-2.1.4-cp314-cp314-win_amd64.whl
| Download URL | datapruning-2.1.4-cp314-cp314-win_amd64.whl |
|---|---|
| Size | 177.5 kB |
| Tags | CPython 3.14 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
889b970300fffe2249280d29a1b92a89d89944047c62f585d3c4709c7c1a20ed
|
|
BLAKE2b-256 checksum How to use checksums |
3cbcf49b239faa8be3bb3ae2f4e7885ff1681f187181542f248a64cf7a6ea24a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.14.4
|
Security for datapruning 2.1.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.