autocurate

Curation is a compute multiplier: train on the right subset of your data and you reach the same quality with less compute. autocurate does it in one call. Point it at a table of training data. It detects what kind of data it is, scores every row with the signals that matter for that kind, applies the strongest established public filter, and hands you the subset to train on as a view: read straight from the table without copying it, in a globally shuffled order. Seven kinds of data are routed from the schema: web text, noisy labels, image-text pairs, SFT chats, preference pairs, RL prompts and robot episodes. Nothing in a default path calls an LLM. Give it your evaluation and it tunes the recipe; add agent=True and an LLM agent surveys the literature and writes recipes. A tuned recipe is applied only if it beats the default on your held-out metric. This is research code, and APIs may change.

Install

pip install -e ".[all]"     # torch, transformers, open_clip, optuna (TPE search), anthropic (opt-in agent)
pip install fasttext        # for the web-text default (DCLM); without it, text falls back to FineWeb-Edu and says so

One verb, three levels

import autocurate as ac

ds = ac.load("data/*.parquet", name="pool", uri="./db")      # parquet, pandas, pyarrow, hf://... into a Lance table

def train_eval(recipe, ids):                                 # your proxy training; higher is better
    model = train(ds.view(ids, columns=["text"]))
    return {"score": evaluate(model), "heldout": evaluate(model)}   # use a held-out set the search never sees

cur = ac.curate(ds)                                          # 1. default: domain recipe, no LLM, no training
cur = ac.curate(ds, eval=train_eval, budget=8)               # 2. tuned: TPE on your eval, guarded by the default
cur = ac.curate(ds, eval=train_eval, budget=8, agent=True)   # 3. agent: literature signals + LLM recipes, guarded
print(cur, cur.report["level"], cur.report["guard"])         # what ran, what was picked and why
loader = cur.to_torch(columns=["text"], batch_size=256)      # zero-copy, globally shuffled batches

ac.diagnose(ds) runs pre-flight checks before you spend compute. Level 2 is TPE by default (no LLM); level 3 is opt-in and the only path that calls an LLM (a logged-in claude CLI, ANTHROPIC_API_KEY, OPENAI_API_KEY or the codex CLI). Without eval=, agent=True only runs signal discovery and applies the default. Every signal is a column, cataloged with its citations and provenance. Tables live on local disk or an object store (S3).

Defaults

Each is a compute multiplier against training on the whole pool at matched quality, read off measured curves and never extrapolated (gross; DataComp-medium is the 80.2M-pair pool). An agent-searched recipe scored above DFN top 30% there on all 3 seeds (3.90×, p = 0.051). Net of curation compute, and against the strongest public filter, see results. Fallbacks when a model cannot run are logged in cur.report.

Docs

pip install -e ".[docs]" && mkdocs serve, or read them in docs/: quickstart · which level do I need? · the guard · guides · agent mode · API · results · benchmarks · contributing.

License

Apache-2.0.