Skip to main content

DataPruning

Intelligent dataset pruning for ML — reduces dataset size by selecting the most informative rows.

Installation

pip install datapruning

Note: PyTorch (~2 GB) is a required dependency.

Requirements

  • Python >= 3.10
  • PyTorch 2.0+
  • Pandas 2.0+
  • NumPy 1.24+

Limits

  • Minimum: 1,000 rows per dataset
  • Maximum: 300,000 rows per dataset

Usage

import pandas as pd
from datapruning import Pruner

df = pd.read_csv("dataset.csv")
p = Pruner(df)
filtered = p.prune(target_col="label", keep_ratio=0.5)

print(f"Kept {len(filtered)} / {len(df)} rows")

Features

  • Runs locally, no data upload
  • Pre-training dataset optimization
  • Pandas DataFrame input / output
  • Compiled processing module

License

Proprietary. See LICENSE file.

Metadata

Release files for datapruning 2.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datapruning 2.1.1
File Size Uploaded Provenance Actions
datapruning-2.1.1.tar.gz 246.5 kB Without provenance Details for datapruning-2.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datapruning 2.1.1
File Interpreter ABI Platform Actions
datapruning-2.1.1-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for datapruning-2.1.1-cp314-cp314-win_amd64.whl

Total release size: 420.8 kB

Release files / datapruning-2.1.1.tar.gz

Download URL datapruning-2.1.1.tar.gz
Size 246.5 kB
Tags Source
SHA-256 checksum
How to use checksums
ee8b2a86d8597bba55bc5e4074c6c242c4302af08e4179dbc32791c367e0a205
BLAKE2b-256 checksum
How to use checksums
d7f99d8e5a0ff95f3d6f721bb84ef430b22c8da00c6a671429237fb83c41f868
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Release files / datapruning-2.1.1-cp314-cp314-win_amd64.whl

Download URL datapruning-2.1.1-cp314-cp314-win_amd64.whl
Size 174.3 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
3c70751535c54e93073304e616112466eca3558e9b28edf70810108fab93c1bb
BLAKE2b-256 checksum
How to use checksums
eb35cd1789db28bd231ba6f9307c391a21101b18def5b3efff09bd0dbd39e6f9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Security for datapruning 2.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page