pyroid

Docs | Benchmarks | Compatibility

pyroid is a Rust implementation of the Python libraries used to clean LLM training data and score model output, with the same API and the same output.

$ pip install pyroid

pyroid 1.0 is not on PyPI yet. Until it is, pip install pyroid installs the unrelated 0.7.x toolkit (see below).

In Python, change the import and keep the rest of your code:

import pyroid.ftfy as ftfy  # was: import ftfy

ftfy.fix_text("naïve résumé")  # 'naïve résumé', as ftfy returns it
ftfy.fix_text_batch(["naïve résumé", "ok"])  # ['naïve résumé', 'ok'], on all cores

On 100,000 FineWeb documents, pyroid's version of datatrove's FineWeb cleaning chain is 27× faster than datatrove on 8 processes, with the same output (AWS EC2 c7i.4xlarge, benchmarks).

Same output, checked

pyroid.ftfy is byte-identical to ftfy 6.3.1. pyroid.fasttext gives the same labels and probabilities as fastText's C++ code on the same CPU architecture. The filters and dedup make datatrove's keep/drop decisions, with its reasons and dedup files, and pyroid.trafilatura extracts the same text as Trafilatura 2.3.1 or 1.11.0, byte for byte. The metrics match rouge-score's and sacrebleu's scores to the last digit, pyroid.json_repair makes json_repair's repairs, pyroid.datasketch gives datasketch's hash values and LSH query results, and pyroid.jieba and pyroid.nltk give jieba's and NLTK's words, sentences and offsets. pyroid.warc reads the same records, documents and errors as warcio 1.8.2 and datatrove's WARC reader.

Each module is tested against the library it replaces, on pinned real data and with property-based fuzzing (parity counts). Every suite runs on Linux aarch64. The ftfy, fastText, filters, dedup, metrics and json_repair suites also ran natively on AWS EC2 x86_64 and aarch64 (c7i.4xlarge and c8g.4xlarge), and the Trafilatura pages and the warc suite on x86_64. macOS and Windows are checked in CI before each release.

Where a module meets an option or input it doesn't port, it hands the call to the real library with a pyroid.FallbackWarning, or raises. It never returns a different result silently.

pyroid's results don't depend on the number of threads or workers, or on how loaded the machine is. Stock datatrove's do: its Trafilatura step gives up on a page after 1 s, so two runs over the same WARC files can keep different documents. pyroid has no timeout.

Differences from the originals

Batch functions take a str, a list of str or an Arrow column (pyarrow, polars, Hugging Face datasets), and return text and masks in the same kind. They release the GIL and run on all cores, so you use threads instead of worker processes, and one loaded fastText model serves every thread instead of one copy per process.

Wheels are built for CPython 3.11 to 3.15 and free-threaded 3.14t and 3.15t, on Linux (glibc and musl), macOS and Windows, for x86-64 and ARM64, and need no compiler. The core has no runtime dependencies; pyroid.fasttext and pyroid.datasketch need numpy (pip install "pyroid[fasttext]", "pyroid[datasketch]").

Speed

_{The FineWeb chain without text extraction (ftfy, fastText language ID, the Gopher, C4 and FineWeb quality filters, MinHash dedup) on the first 10,000 documents of the pinned FineWeb slice, with 1 and 8 processes or threads pinned to 8 CPUs. datatrove 0.10.1 (ftfy 6.3.1, fasttext-predict 0.9.2.4, spaCy 3.8.18) on CPython 3.13.18; pyroid's native chain on free-threaded CPython 3.15.2. The output is the same document for document. On the 100k corpus the ratios are 27× and 56×.}

pyroid clean runs that chain as one command over Parquet or JSONL files:

$ pip install "pyroid[cli]"
$ pyroid clean in.parquet out.parquet --recipe fineweb   # datatrove's FineWeb pipeline

From Common Crawl WARC files to deduplicated documents, datatrove's FineWeb pipeline runs 9.5× faster with pyroid's steps in datatrove (89.4 s instead of 848.3 s on 8 workers), and pyroid clean --recipe fineweb-warc runs it 10.7× faster (79.6 s), for 171,595 pages in 8 WARC files of CC-MAIN-2026-39 on AWS EC2 c7i.4xlarge (benchmarks). Both keep the same documents, apart from the pages stock datatrove drops at its 1 s extraction timeout.

By stage, on AWS EC2 c7i.4xlarge with the 10,000-document dev slice, pyroid on one thread against the incumbent in one process:

With pyroid's stages swapped into datatrove's own executor, the chain ran 11.2× faster at 8 processes on the dev slice and 13.6× on 100k documents.

The other modules have only been measured in a Docker VM on a Mac so far, so these numbers are indicative (one thread unless stated):

pyroid.json_repair stays linear on 13 input families where json_repair takes quadratic time, so its lead grows with the input. The benchmarks page gives each row's corpus, interpreter and source.

Integrations

import pyroid

pyroid.patch("ftfy", "fasttext")  # later imports of ftfy and fasttext get pyroid's

The filters and dedup cover 4,049 of datatrove's 4,343 language codes natively, Chinese and Vietnamese included, Japanese with an installed SudachiDict dictionary (pip install sudachidict-core) and Thai with an installed PyThaiNLP (pip install pythainlp). The rest go through datatrove's own tokenizers (languages).

The 0.7.x releases

pyroid 1.0 is a rewrite. The previous general-purpose toolkit (0.1 to 0.7) is not carried forward, and the 1.0 API is incompatible with it.

• 0.7.x stays installable from PyPI: pip install "pyroid<1".
• Its source is kept on the legacy branch.
• Migrating from 0.7 lists replacements.

Development

The extension is built with maturin:

$ uv venv && source .venv/bin/activate
$ uv pip install maturin --group test
$ maturin develop --uv     # builds pyroid._pyroid into the venv (editable)
$ pytest python/tests
$ cargo test               # pure-Rust core (crates/pyroid-core)

The code is laid out as follows:

• crates/pyroid-core: pure Rust, no Python;
• crates/pyroid-py: the PyO3 bindings, built as pyroid._pyroid;
• python/pyroid: the typed Python package;
• bench/: the benchmark harness;
• parity/: differential tests against the incumbents;
• docs/: the documentation.

See CONTRIBUTING.md.

License

pyroid's own code is MIT. The ported parts keep their licences, so the package's licence expression is MIT AND Apache-2.0 AND BSD-2-Clause AND BSD-3-Clause AND ISC AND PSF-2.0 AND Unicode-3.0 AND Zlib:

• Apache-2.0: ported from ftfy, datatrove, rouge-score, NLTK (the Porter stemmer and the tokenizers), sacrebleu, Trafilatura, SudachiPy and warcio;
• MIT: ported from fastText, spaCy, indic-nlp-library, urduhack, json_repair, datasketch, jieba (with its dictionary) and libxml2 (the HTML parser);
• BSD-2-Clause: jusText (Trafilatura's paragraph classifier and its stoplists);
• BSD-3-Clause: numpy's random generator (for C4 bad words), scipy's QUADPACK integration (for MinHashLSH's parameters), tldextract and lxml;
• PSF-2.0: CPython's ipaddress (IPv6 parsing);
• ISC: anyascii's transliteration data;
• Unicode-3.0: data from the Unicode Character Database.
• Zlib: zlib-rs, the Rust crate behind pyroid.warc's gzip decoding.

See LICENSE, LICENSES/ and NOTICE, which lists the ported files. LICENSES/THIRD-PARTY-RUST.txt has the licences of the Rust crates the extension links.