DPK Python Transforms

installation

The transforms are delivered as a standard pyton library available on pypi and can be installed using uv pip install: pip install uv

python -m uv pip install data-prep-toolkit-transforms[all] or python -m uv pip install data-prep-toolkit-transforms[ray, all] or python -m uv pip install data-prep-toolkit-transforms[language]

installing the python transforms will also install data-prep-toolkit

installing the ray transforms will also install data-prep-toolkit[ray]

Release notes:

1.1.1.dev1

Include all code transforms as extra [code]

1.1.1.dev0

Refactored code transforms (code_uality, code2parquet, header_cleanser, license select, proglang_select)
Added ml-filter and enrichment
renamed PDF2Parquet to Docling2Paruqet

1.0.1.dev1

Added Gneissweb transforms
fdedup fix for windows

1.0.1.dev0

PR #979 (code_profiler)

1.0.0.a6

Added Profiler
Added Resize

1.0.0.a5

Added Pii Redactor
Relax fasttext requirement >= 0.9.2

1.0.0.a4

Added missing ray implementation for lang_id, doc_quality, tokenization and filter
Added ray notebooks for lang id, Doc Quality, tokenization, and Filter

1.0.0.a3

Added code_profiler

1.0.0.a2

Relax dependencies on pandas (use latest or whatever is installed by application) Relax dependencies on requests (use latest or whatever is installed by application)

dev ray all language code images gneissweb proglang-select header-cleanser license-select code-quality code2parquet malware code-profiler repo-level-order yara folder2parquet opensearch profiler resize doc-chunk doc-quality html2parquet lang-id docling2parquet text-encoder pii-redactor similarity extreme-tokenized readability gneissweb-classification ml-filter enrichment filter doc-id hap bloom ededup fdedup tokenization web2parquet rep-removal tokenization2arrow collapse blocklist gopher-repetition-annotator fineweb-quality-annotator faces nsfw people

Metadata

Release files for data-prep-toolkit-transforms 1.1.8

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for data-prep-toolkit-transforms 1.1.8
File	Interpreter	ABI	Platform	Reset
data_prep_toolkit_transforms-1.1.8-py3-none-any.whl 9.1 MB Jun 22, 2026	Python 3	none	any	Details

Release files / data_prep_toolkit_transforms-1.1.8-py3-none-any.whl

Download URL	data_prep_toolkit_transforms-1.1.8-py3-none-any.whl
Size	9.1 MB
Tags	Python 3
SHA-256 checksum How to use checksums	`58a009d2197dffc915e022c0b27cf3614481f57dc1043c0045711886195f89e7`
BLAKE2b-256 checksum How to use checksums	`fcaf27dd9f868ba11083d65d7bf588465830145eaa4a543a99c73058db9f4356`
Upload date	Jun 22, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`twine/6.1.0 CPython/3.10.16`