winnow-ai 0.1.0

Open-source data curation toolkit for ML. Filter, deduplicate, score, and explore your training data.

The open-source workbench for ML training data.

Quick Start • Install • Features • Docs • Compare • Contribute

---

Winnow is a Python toolkit for curating ML training data. Filter junk, deduplicate, compute embeddings, find outliers, and search your dataset — from one pip install, with a CLI and a Python SDK.

Installation

pip install winnow-ai

That gives you heuristic filters, exact dedup, and the CLI. For the full toolkit:

pip install "winnow-ai[all]"    # embeddings, fuzzy dedup, language detection, HF datasets

Or pick what you need:

Extra	What it adds
winnow-ai[embeddings]	Sentence-transformer embeddings, FAISS search, semantic dedup, anomaly detection
winnow-ai[dedup]	MinHash-LSH fuzzy deduplication
winnow-ai[nlp]	Language detection (lingua)
winnow-ai[io]	HuggingFace datasets, Pandas

Quick Start

from winnow.core.pipeline import Pipeline
from winnow.core.filters import LengthFilter, WhitespaceFilter, RepetitionFilter
from winnow.quality.dedup import ExactDedup

# Build a pipeline
pipeline = Pipeline("my-curation")
pipeline.add_filter(LengthFilter(min_length=50, max_length=100_000))
pipeline.add_filter(WhitespaceFilter(max_whitespace_ratio=0.4))
pipeline.add_filter(RepetitionFilter(max_repetition_ratio=0.3))
pipeline.add_filter(ExactDedup())

# Run it
result = pipeline.run("data/raw.jsonl", "data/curated.parquet")
print(f"Kept {result['total_kept']:,} / {result['total_read']:,} documents")

Or from the command line:

winnow curate data/raw.jsonl data/clean.parquet \
  --min-length 50 --max-whitespace 0.4 --dedup

Or with a YAML config:

winnow curate data/raw.jsonl data/clean.parquet --config pipeline.yaml

See examples/quickstart.py for a full walkthrough including embeddings, search, and outlier detection.

Features

Heuristic Filters

Twelve built-in filters, all composable, all configurable via code or YAML:

Filter	What it catches
LengthFilter	Too short / too long documents
WordCountFilter	Documents outside a word-count range
LineCountFilter	Documents outside a line-count range
WhitespaceFilter	Excessive whitespace (formatting junk)
RepetitionFilter	Repeated n-grams (boilerplate, spam)
SpecialCharFilter	Special character overload (encoding artifacts)
AlphaFilter	Low alphabetic ratio (numeric spam, base64)
URLFilter	URL-heavy documents (link farms)
StopwordFilter	Missing stopwords (keyword spam, code)
LanguageFilter	Wrong language (lingua or fastText backend)
FieldExistsFilter	Missing required fields
RegexFilter	Custom pattern matching (include or exclude)

Deduplication

• Exact dedup (SHA-256) — zero dependencies, streaming
• Fuzzy dedup (MinHash-LSH) — catches near-duplicates at scale
• Semantic dedup (embedding cosine similarity) — finds paraphrases and reworded copies

Embeddings & Search

• Compute embeddings with any sentence-transformer model
• Semantic search — FAISS-backed nearest-neighbor search over your dataset
• Anomaly/outlier detection — k-NN distance scoring to surface unusual documents

Pipeline Orchestration

• Chain any number of filters in a Pipeline
• Configure via Python or YAML
• Per-filter removal stats and throughput reporting
• Reads JSONL, Parquet, CSV, and HuggingFace datasets
• Writes JSONL and Parquet

CLI

Seven commands, zero boilerplate:

winnow version     # print version
winnow curate      # run a curation pipeline
winnow stats       # dataset statistics
winnow embed       # compute embeddings
winnow search      # semantic search
winnow outliers    # find anomalous documents
winnow explore     # launch web UI (coming soon)

Comparison

Winnow is a data workbench — interactive, exploratory, designed for iteration. Pipeline engines like DataTrove and Data-Juicer are great for scheduled batch processing at massive scale. If you need to understand your data, experiment with filter thresholds, and investigate what you are keeping and discarding, Winnow is the right tool.

	Winnow	DataTrove	Data-Juicer
Interactive exploration	Yes	No	Limited
Semantic search	Yes	No	No
Outlier detection	Yes	No	No
Embedding-based dedup	Yes	MinHash only	MinHash only
CLI + Python SDK	Both	Python only	Both
YAML config	Yes	No (Python)	Yes
HuggingFace datasets	Yes	Yes	Yes
Spark/distributed	Roadmap	Yes	Yes
Web UI	Coming soon	No	Yes

Examples

• examples/quickstart.py — Full workflow: load, filter, dedup, embed, search, outliers
• examples/basic_pipeline.yaml — Simple YAML config
• examples/advanced_pipeline.yaml — All available filters

Documentation

Full docs at winnow-ai.github.io/winnow (coming soon).

Contributing

We welcome contributions. See CONTRIBUTING.md for setup instructions.

License

Apache 2.0 — see LICENSE.