Skip to main content

DataPruning

Intelligent dataset pruning for ML — reduces dataset size by selecting the most informative rows.

Installation

pip install datapruning

Note: PyTorch (~2 GB) is a required dependency.

Requirements

  • Python >= 3.10
  • PyTorch 2.0+
  • Pandas 2.0+
  • NumPy 1.24+

Limits

  • Minimum: 1,000 rows per dataset
  • Maximum: 300,000 rows per dataset

Usage

import pandas as pd
from datapruning import Pruner

df = pd.read_csv("dataset.csv")
p = Pruner(df)
filtered = p.prune(target_col="label", keep_ratio=0.5)

print(f"Kept {len(filtered)} / {len(df)} rows")

Features

  • Runs locally, no data upload
  • Pre-training dataset optimization
  • Pandas DataFrame input / output
  • Compiled processing module

License

Proprietary. See LICENSE file.

Metadata

Release files for datapruning 2.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datapruning 2.1.0
File Size Uploaded Provenance Actions
datapruning-2.1.0.tar.gz 246.3 kB Without provenance Details for datapruning-2.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datapruning 2.1.0
File Interpreter ABI Platform Actions
datapruning-2.1.0-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for datapruning-2.1.0-cp314-cp314-win_amd64.whl

Total release size: 420.5 kB

Release files / datapruning-2.1.0.tar.gz

Download URL datapruning-2.1.0.tar.gz
Size 246.3 kB
Tags Source
SHA-256 checksum
How to use checksums
e9107c469a5782e5b755253ea58826e4ca062da437bd9998b7e9d49166930191
BLAKE2b-256 checksum
How to use checksums
8cc8c389822f9864df5ee0a7e01133e49b43cc421eea99a8c378b1c185f42d67
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Release files / datapruning-2.1.0-cp314-cp314-win_amd64.whl

Download URL datapruning-2.1.0-cp314-cp314-win_amd64.whl
Size 174.2 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
09aa23b7a7cd59287f062386ca5d1b0a0447c96eab6b0e9da8ab8fc2994535dd
BLAKE2b-256 checksum
How to use checksums
7a8df416c68860d827b659a041bbaec6e1074d9a8cb2a9215c5ad1363e9cd131
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Security for datapruning 2.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page