cureta

Unified Data Curation Framework — enrich text datasets at scale with a composable pipeline of feature-extraction Taggers.

Install

pip install cureta              # core: heuristic taggers, in-process tagging
pip install "cureta[ray]"       # + distributed pipelines (the `cureta run` CLI)
pip install "cureta[ray,llm]"   # + LLM / GPU taggers

Requires: Python ≥ 3.10

Developing on cureta itself? See the contributor setup guide for an editable install from source.

Quick example

Cureta runs a pipeline — a list of taggers that you define. Create one as pipelines/my_pipeline/pipeline_config.yaml:

pipeline_name: "my_pipeline"

tagger_stages:
  - tagger_name: "cureta_id"     # stable per-document content hash
  - tagger_name: "num_words"     # word count
  - tagger_name: "line_stats"    # per-line statistics

Tag a single string in-process with quick_tag:

from cureta import quick_tag

tags = quick_tag(
    "The quick brown fox jumps over the lazy dog.",
    pipeline_config="pipelines/my_pipeline/pipeline_config.yaml",
)
# {'cureta_id': {'cureta_id': 'ef537f2...'}, 'num_words': {'num_words': 10}, ...}

Or run the same pipeline over a whole dataset from the command line:

# Create a small sample dataset (JSONL with a "text" field per line)
python -c "
import json, pathlib
rows = [{'text': f'This is sample document number {i}.'} for i in range(200)]
pathlib.Path('sample.jsonl').write_text('\n'.join(json.dumps(r) for r in rows))
print('Created sample.jsonl with 200 rows')
"

# Resolves pipelines/my_pipeline/pipeline_config.yaml in the current directory
cureta run --pipeline my_pipeline --dataset ./sample.jsonl --limit 100

For HuggingFace datasets see docs/how_to/load_huggingface_data.md

Documentation

Examples

Key concepts

• Document — canonical data envelope with raw_content, document_id, metadata, and tags
• Tagger — atomic feature-extraction unit; subclass CPUTagger or GPUTagger, implement process_document (Tier 1) or run (Tier 2)
• Pipeline — orchestrator that loads config, resolves dependencies, and dispatches taggers

45 built-in taggers ship with the library across three categories:

Heuristic taggers (CPU, no model weights) — word count, line/paragraph/word-length statistics, Gopher/C4/FineWeb quality filters, compression ratio, vocabulary diversity, MinHash/SimHash/exact dedup signals, PII regex, HTML density, Markdown structure, content type, URL parsing, domain blocklist, date extraction, and readability metrics.

Lightweight ML taggers — GlotLID language ID (1665 languages), paragraph-level language ID, FastText quality scoring, programming language detection, and Microsoft Presidio PII detection.

GPU taggers — FineWeb educational quality (0–5 score), domain classification (26 domains), multi-label toxicity, dense sentence embeddings, LLM-as-judge scoring, and perplexity.

See Reference: Taggers for the full list with output schemas.

Contributing

See CONTRIBUTING.md and docs/contributing/setup.md.

License

MIT