Skip to main content

DataPruning

Intelligent dataset optimization — reduces dataset size by selecting the most informative rows while preserving model quality.

Installation

pip install datapruning

For image/text multimodal support (CLIP encoding):

pip install datapruning[clip]

Quick Start

CSV / Tabular Data

import pandas as pd
from datapruning.sdk import DatasetEngine

df = pd.read_csv("dataset.csv")
engine = DatasetEngine(df, target_col="label")

# Analyze the dataset
analysis = engine.analyze()

# Get a recommendation
rec = engine.recommend()
print(f"Best method: {rec.display_name} ({rec.confidence}% confidence)")

# Optimize — keep 50% of most informative rows
result = engine.optimize(keep_ratio=0.5)
print(f"Kept {result['runtime_seconds']:.1f}s — {result['explainability']['rows_removed']} rows removed")

# Download the optimized dataset
result["optimized_df"].to_csv("optimized.csv", index=False)

Image Datasets

from datapruning.sdk import DatasetEngine

# Create engine from a folder of images
engine = DatasetEngine.from_images(
    folder_path="./photos",
    labels=[0, 1, 0, 1, ...],  # one label per image
    keep_ratio=0.5,
)

# Analyze and optimize
analysis = engine.analyze()
result = engine.optimize(keep_ratio=0.5)
result["optimized_df"].to_csv("optimized_images.csv", index=False)

Text Datasets

engine = DatasetEngine.from_texts(
    texts=["label A sample", "label B sample", ...],
    labels=[0, 1, 0, 1, ...],
)
result = engine.optimize(keep_ratio=0.5)

Algorithms

DataPruning selects the best algorithm automatically based on dataset characteristics:

Method What it does Best for
Margin Optimization Analyzes classification confidence gap via cross-validation, keeps boundary samples General tabular data
Importance Scoring Measures sample difficulty via model prediction confidence across CV folds Quick baseline, any dataset
Diversity Selection Picks the most representative subset (K-Center) High-dimensional data
Noise Filtering Drops redundant and potentially mislabeled samples via CV-based difficulty Noisy/messy data
Full Pipeline Combines all signals for maximum quality Complex datasets

All scoring methods use out-of-fold (OOF) model logits — each sample is scored by a model that never saw it during training, producing unbiased difficulty estimates. The optimizer also includes a "must beat random" gate that validates the selected strategy actually improves over random subsampling on an internal holdout; if not, it safely falls back to random selection.

Limits

  • Minimum: 1,000 rows (tabular) or 10 images
  • Maximum: 300,000 rows (tabular) or 1,000 images
  • Supported image formats: JPG, PNG, BMP, GIF, TIFF, WebP

Requirements

  • Python >= 3.10
  • Pandas 2.0+
  • NumPy 1.24+
  • scikit-learn 1.3+

Optional (for image/text support):

  • PyTorch 2.0+
  • transformers 4.30+
  • Pillow 9.0+

Features

  • Runs locally — no data uploaded anywhere
  • Auto-detects task type (classification/regression) and target column
  • Auto-encodes categorical/string columns
  • 5 optimization algorithms with automatic selection
  • Preserves minority class balance
  • CSV and image dataset support
  • Pure Python — works on any platform

License

Proprietary. See LICENSE file.

Metadata

Release files for datapruning 2.1.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datapruning 2.1.5
File Size Uploaded Provenance Actions
datapruning-2.1.5.tar.gz 250.5 kB Without provenance Details for datapruning-2.1.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datapruning 2.1.5
File Interpreter ABI Platform Actions
datapruning-2.1.5-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for datapruning-2.1.5-cp314-cp314-win_amd64.whl

Total release size: 428.1 kB

Release files / datapruning-2.1.5.tar.gz

Download URL datapruning-2.1.5.tar.gz
Size 250.5 kB
Tags Source
SHA-256 checksum
How to use checksums
b45abe866c1beea175e7f18650157472ded77e715a93a73ac1d209cf775905b8
BLAKE2b-256 checksum
How to use checksums
fbe2cd647a4717800fa538b46663920b403491fda6f0af7ed45d67e5b9a6b05e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Release files / datapruning-2.1.5-cp314-cp314-win_amd64.whl

Download URL datapruning-2.1.5-cp314-cp314-win_amd64.whl
Size 177.6 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
0ff03913525f3a7fe351e965e7198de3cfdd3ebd37f91b1751575d3f6d711f09
BLAKE2b-256 checksum
How to use checksums
e8182fb45608f5a5bef061ad9d350b626723fc9aaa49ab2f75cd2227e4aff34e
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Security for datapruning 2.1.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page