dokime 0.2.0

Open-source data curation toolkit for ML. Filter, deduplicate, score, and explore your training data.

pytest for your training data.

Quick Start • Install • Docs • Contribute

---

One command to grade your ML training data. Filter junk, deduplicate, score quality, find outliers, and search — from a single pip install.

Installation

pip install dokime

For the full toolkit (embeddings, fuzzy dedup, language detection):

pip install "dokime[all]"

Pick individual extras

Extra	What it adds
dokime[embeddings]	Sentence-transformer embeddings, FAISS search, semantic dedup
dokime[dedup]	MinHash-LSH fuzzy deduplication
dokime[nlp]	Language detection (lingua)
dokime[io]	HuggingFace datasets, Pandas
dokime[explore]	Interactive web UI

Quick Start

Score your data in one command:

dokime score data.jsonl

Build a curation pipeline:

from dokime.core.pipeline import Pipeline
from dokime.core.filters import LengthFilter, WhitespaceFilter, RepetitionFilter
from dokime.quality.dedup import ExactDedup

pipeline = Pipeline("my-curation")
pipeline.add_filter(LengthFilter(min_length=50, max_length=100_000))
pipeline.add_filter(WhitespaceFilter(max_whitespace_ratio=0.4))
pipeline.add_filter(RepetitionFilter(max_repetition_ratio=0.3))
pipeline.add_filter(ExactDedup())

result = pipeline.run("data/raw.jsonl", "data/curated.parquet")
print(f"Kept {result['total_kept']:,} / {result['total_read']:,} documents")

Or from the CLI:

dokime curate data/raw.jsonl data/clean.parquet \
  --min-length 50 --max-whitespace 0.4 --dedup

What's in the box

Quality scoring — Grade your dataset A-F with per-document signals: entropy, character ratios, word count, repetition detection.

12 heuristic filters — Length, word count, whitespace, repetition, special characters, alphabetic ratio, URLs, stopwords, language, field existence, regex. All composable via code or YAML.

3 dedup strategies — Exact (SHA-256), fuzzy (MinHash-LSH), and semantic (embedding cosine similarity).

Embeddings & search — Compute sentence-transformer embeddings, FAISS-backed search, k-NN anomaly/outlier detection.

Pipeline orchestration — Chain filters, configure via YAML, streaming execution with constant memory, per-filter removal stats.

10 CLI commands — version, curate, stats, embed, search, outliers, score, push, attribute, explore.

Documentation

Full docs at dokime-ai.github.io/dokime.

Contributing

We welcome contributions. See CONTRIBUTING.md for setup instructions.

License

Apache 2.0 — see LICENSE.