Skip to main content

DataPruning

Intelligent dataset optimization — reduces dataset size by selecting the most informative rows while preserving model quality.

Installation

pip install datapruning

For image/text multimodal support (CLIP encoding):

pip install datapruning[clip]

Quick Start

CSV / Tabular Data

import pandas as pd
from datapruning.sdk import DatasetEngine

df = pd.read_csv("dataset.csv")
engine = DatasetEngine(df, target_col="label")

# Analyze the dataset
analysis = engine.analyze()

# Get a recommendation
rec = engine.recommend()
print(f"Best method: {rec.display_name} ({rec.confidence}% confidence)")

# Optimize — keep 50% of most informative rows
result = engine.optimize(keep_ratio=0.5)
print(f"Kept {result['runtime_seconds']:.1f}s — {result['explainability']['rows_removed']} rows removed")

# Download the optimized dataset
result["optimized_df"].to_csv("optimized.csv", index=False)

Image Datasets

from datapruning.sdk import DatasetEngine

# Create engine from a folder of images
engine = DatasetEngine.from_images(
    folder_path="./photos",
    labels=[0, 1, 0, 1, ...],  # one label per image
    keep_ratio=0.5,
)

# Analyze and optimize
analysis = engine.analyze()
result = engine.optimize(keep_ratio=0.5)
result["optimized_df"].to_csv("optimized_images.csv", index=False)

Text Datasets

engine = DatasetEngine.from_texts(
    texts=["label A sample", "label B sample", ...],
    labels=[0, 1, 0, 1, ...],
)
result = engine.optimize(keep_ratio=0.5)

Algorithms

DataPruning selects the best algorithm automatically based on dataset characteristics:

Method What it does Best for
Margin Optimization Analyzes classification confidence gap via cross-validation, keeps boundary samples General tabular data
Importance Scoring Measures sample difficulty via model prediction confidence across CV folds Quick baseline, any dataset
Diversity Selection Picks the most representative subset (K-Center) High-dimensional data
Noise Filtering Drops redundant and potentially mislabeled samples via CV-based difficulty Noisy/messy data
Full Pipeline Combines all signals for maximum quality Complex datasets

All scoring methods use out-of-fold (OOF) model logits — each sample is scored by a model that never saw it during training, producing unbiased difficulty estimates. The optimizer also includes a "must beat random" gate that validates the selected strategy actually improves over random subsampling on an internal holdout; if not, it safely falls back to random selection.

Limits

  • Minimum: 1,000 rows (tabular) or 10 images
  • Maximum: 300,000 rows (tabular) or 1,000 images
  • Supported image formats: JPG, PNG, BMP, GIF, TIFF, WebP

Requirements

  • Python >= 3.10
  • Pandas 2.0+
  • NumPy 1.24+
  • scikit-learn 1.3+

Optional (for image/text support):

  • PyTorch 2.0+
  • transformers 4.30+
  • Pillow 9.0+

Features

  • Runs locally — no data uploaded anywhere
  • Auto-detects task type (classification/regression) and target column
  • Auto-encodes categorical/string columns
  • 5 optimization algorithms with automatic selection
  • Preserves minority class balance
  • CSV and image dataset support
  • Pure Python — works on any platform

License

Proprietary. See LICENSE file.

Metadata

Release files for datapruning 2.1.6

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datapruning 2.1.6
File Size Uploaded Provenance Actions
datapruning-2.1.6.tar.gz 250.7 kB Without provenance Details for datapruning-2.1.6.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datapruning 2.1.6
File Interpreter ABI Platform Actions
datapruning-2.1.6-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for datapruning-2.1.6-cp314-cp314-win_amd64.whl

Total release size: 428.6 kB

Release files / datapruning-2.1.6.tar.gz

Download URL datapruning-2.1.6.tar.gz
Size 250.7 kB
Tags Source
SHA-256 checksum
How to use checksums
58277cfb534f5a300525ca73bb78b8fe5e54b3e02bf178b8434a84225df70673
BLAKE2b-256 checksum
How to use checksums
ed194f4c8bed5738f4e4139cb93eade9312758e220699f4df67b30b1fbe771d0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Release files / datapruning-2.1.6-cp314-cp314-win_amd64.whl

Download URL datapruning-2.1.6-cp314-cp314-win_amd64.whl
Size 177.9 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
ac0640e9dd4484039427530ede04458c90f523736e71d71e3690d1b78098a59a
BLAKE2b-256 checksum
How to use checksums
529812ecb0e3ef7c2266e6dbb14a48af5a25e00865a65ce393457d911dae316c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Security for datapruning 2.1.6

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page