Skip to main content

DataPruning

Intelligent dataset pruning for ML — reduces dataset size by selecting the most informative rows.

Installation

pip install datapruning

Note: PyTorch (~2 GB) is a required dependency.

Requirements

  • Python >= 3.10
  • PyTorch 2.0+
  • Pandas 2.0+
  • NumPy 1.24+

Limits

  • Minimum: 1,000 rows per dataset
  • Maximum: 300,000 rows per dataset

Usage

import pandas as pd
from datapruning import Pruner

df = pd.read_csv("dataset.csv")
p = Pruner(df)
filtered = p.prune(target_col="label", keep_ratio=0.5)

print(f"Kept {len(filtered)} / {len(df)} rows")

Features

  • Runs locally, no data upload
  • Pre-training dataset optimization
  • Pandas DataFrame input / output
  • Compiled processing module

License

Proprietary. See LICENSE file.

Metadata

Release files for datapruning 2.1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datapruning 2.1.2
File Size Uploaded Provenance Actions
datapruning-2.1.2.tar.gz 246.5 kB Without provenance Details for datapruning-2.1.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for datapruning 2.1.2
File Interpreter ABI Platform Actions
datapruning-2.1.2-cp314-cp314-win_amd64.whl CPython 3.14 CPython 3.14 Windows x86-64 Details for datapruning-2.1.2-cp314-cp314-win_amd64.whl

Total release size: 420.7 kB

Release files / datapruning-2.1.2.tar.gz

Download URL datapruning-2.1.2.tar.gz
Size 246.5 kB
Tags Source
SHA-256 checksum
How to use checksums
63de6c0ee7c78df290e3388af910a8ac09c89c4edfa0d79811dda77897bdcd19
BLAKE2b-256 checksum
How to use checksums
a082c31be216fd3d9bdc5655fa1a1735322c4e0b0d41034552230968a2935834
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Release files / datapruning-2.1.2-cp314-cp314-win_amd64.whl

Download URL datapruning-2.1.2-cp314-cp314-win_amd64.whl
Size 174.3 kB
Tags CPython 3.14 Windows x86-64
SHA-256 checksum
How to use checksums
d09677e002f06234533c9d23620de834720e47bae52201b4943f7fbc07e8d180
BLAKE2b-256 checksum
How to use checksums
0b030a10d0cb1704e67133bccf1138c32fc1365241f30578bceef302050e30f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.4

Security for datapruning 2.1.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page