Skip to main content

DataPruning

Intelligent dataset optimization — reduces dataset size by selecting the most informative rows while preserving model quality.

Installation

pip install datapruning

For image/text multimodal support (CLIP encoding):

pip install datapruning[clip]

Quick Start

CSV / Tabular Data

import pandas as pd
from datapruning.sdk import DatasetEngine

df = pd.read_csv("dataset.csv")
engine = DatasetEngine(df, target_col="label")

# Analyze the dataset
analysis = engine.analyze()

# Get a recommendation
rec = engine.recommend()
print(f"Best method: {rec.display_name} ({rec.confidence}% confidence)")

# Optimize — keep 50% of most informative rows
result = engine.optimize(keep_ratio=0.5)
print(f"Kept {result['runtime_seconds']:.1f}s — {result['explainability']['rows_removed']} rows removed")

# Download the optimized dataset
result["optimized_df"].to_csv("optimized.csv", index=False)

Image Datasets

from datapruning.sdk import DatasetEngine

# Create engine from a folder of images
engine = DatasetEngine.from_images(
    folder_path="./photos",
    labels=[0, 1, 0, 1, ...],  # one label per image
    keep_ratio=0.5,
)

# Analyze and optimize
analysis = engine.analyze()
result = engine.optimize(keep_ratio=0.5)
result["optimized_df"].to_csv("optimized_images.csv", index=False)

Text Datasets

engine = DatasetEngine.from_texts(
    texts=["label A sample", "label B sample", ...],
    labels=[0, 1, 0, 1, ...],
)
result = engine.optimize(keep_ratio=0.5)

Algorithms

DataPruning selects the best algorithm automatically based on dataset characteristics:

Method What it does Best for
Margin Optimization Analyzes classification confidence, keeps boundary samples General tabular data
Importance Scoring Measures sample difficulty via feature distances Quick baseline, any dataset
Diversity Selection Picks the most representative subset (K-Center) High-dimensional data
Noise Filtering Drops redundant and potentially mislabeled samples Noisy/messy data
Full Pipeline Combines all signals for maximum quality Complex datasets

Limits

  • Minimum: 1,000 rows (tabular) or 10 images
  • Maximum: 300,000 rows (tabular) or 1,000 images
  • Supported image formats: JPG, PNG, BMP, GIF, TIFF, WebP

Requirements

  • Python >= 3.10
  • Pandas 2.0+
  • NumPy 1.24+
  • scikit-learn 1.3+

Optional (for image/text support):

  • PyTorch 2.0+
  • transformers 4.30+
  • Pillow 9.0+

Features

  • Runs locally — no data uploaded anywhere
  • Auto-detects task type (classification/regression) and target column
  • Auto-encodes categorical/string columns
  • 5 optimization algorithms with automatic selection
  • Preserves minority class balance
  • CSV and image dataset support
  • Pure Python — works on any platform

License

Proprietary. See LICENSE file.

Metadata

Release files for datapruning 2.1.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datapruning 2.1.3
File Size Uploaded Provenance Actions
datapruning-2.1.3.tar.gz 248.3 kB Without provenance Details for datapruning-2.1.3.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datapruning 2.1.3
File Interpreter ABI Platform Actions
datapruning-2.1.3-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for datapruning-2.1.3-cp314-cp314-win_amd64.whl

Total release size: 423.5 kB

Release files / datapruning-2.1.3.tar.gz

Download URL datapruning-2.1.3.tar.gz
Size 248.3 kB
Tags Source
SHA-256 checksum
How to use checksums
1fca4cd5d8713aeacd5f30f1a030553cef84c5b80fce8b8eba73f847e18ad559
BLAKE2b-256 checksum
How to use checksums
5fb8d02e0a2a74db1cf778e34105e3adf6f31191bd4c95a2bf88c8cf00998a23
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Release files / datapruning-2.1.3-cp314-cp314-win_amd64.whl

Download URL datapruning-2.1.3-cp314-cp314-win_amd64.whl
Size 175.2 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
e7321327efe69f3ef8abe322c9c2fb17f4be09e6a06ffa4f90b654392e734c6f
BLAKE2b-256 checksum
How to use checksums
5d74cb95eaeef13aba0bba5b0b840e2aa7ce6972e673a38e0de2bb9bdeaf0029
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Security for datapruning 2.1.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page