dropoutt

Pre-flight checks for LLM training data. Point it at a folder. It tells you what is wrong before you burn a training run finding out.

pip install dropoutt
dropoutt scan ./data

No model, no config, no flags required. Add --model or --target to unlock more checks and CI gating. Skipped checks always name the one flag that unlocks them.

What it is

A local CLI that:

1. Scans training datasets for structural bugs (empty loss masks, broken roles, truncation that kills the answer, contamination, PII, language damage).
2. Fingerprints the corpus so two datasets can be compared without shipping records.
3. Maps your data onto a frozen atlas — a latent coordinate system built from public datasets — so you see what you cover, what you miss, and what sits off the map. That is dropoutt atlas, its own command since 1.3.

It runs on your CPU. One install brings everything — there are no extras to choose between, and nothing in the dependency list needs a compiler.

Install

Requires Python 3.10+.

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install dropoutt

That is the whole install. Every dependency publishes a wheel for every Python and platform dropoutt supports, so nothing is ever compiled at install time — which is what a user on Windows and CPython 3.14 hit when the language-detection dependency had no wheel and pip fell through to needing Visual C++ Build Tools.

There used to be extras ([atlas], [parquet], [lid], [all]). They are gone. Each one was a way to end up with a dropoutt that silently could not read your Parquet, and pip install 'dropoutt[all]' still works — pip warns about the unknown extra and installs everything, which is what you wanted anyway.

If dropoutt is not on PATH (module systems, batch schedulers, some Windows setups): python -m dropoutt does the same thing.

Supported inputs: JSON, JSONL/NDJSON, TXT, Markdown, CSV/TSV, Parquet, Arrow, Feather, ORC, .mds (MosaicML Streaming) and .tar (WebDataset). Text formats may be gzip / bzip2 / xz / zstd compressed.

Works on macOS, Linux, and Windows. Cache defaults to ~/.cache/dropoutt, or %LOCALAPPDATA%\dropoutt on Windows. Override with DROPOUTT_CACHE. See docs/portability.md for offline / HPC use.

Quick start

dropoutt scan ./my-corpus
# writes .dropoutt/{report.html, report.md, report.json, findings.jsonl, fingerprint.json}

dropoutt scan ./my-corpus --model qwen3 --seq-len 4096 --target sft
# unlocks token/mask checks and exit code 10 on blocking findings

dropoutt atlas ./my-corpus      # where that corpus sits on the map
# writes .dropoutt/{atlas.html, atlas.md, atlas.json}

dropoutt checks                 # live catalog
dropoutt checks T0-MASK-001     # one check in detail
dropoutt fetch                  # pre-download everything --offline needs

The whole report prints in the terminal, and the same content is written to report.html, report.md and report.json — the page, the pasteable version and the machine-readable one all say the same things. --brief prints the verdict and one line per finding instead; --quiet prints nothing and writes the files.

The report is one self-contained file: no CDN, no web fonts, no network, opens from file://. A scan opens it for you when there is a desktop to open it on, and quietly does not when there is not — over SSH, in CI, under a batch scheduler, or with output redirected. --no-open or DROPOUTT_OPEN=0 turns that off; DROPOUTT_OPEN=1 forces it, which is what you want with X11 forwarding.

Anything above 24 MB is scanned across processes, and the pool is sized against the machine this process can actually use — CPU affinity, any cgroup quota, hyperthread topology and free memory, not os.cpu_count(). Measured: 480,000 records across eight datasets, 377 MB, in 38 seconds on a 14-core laptop. The result does not depend on how many cores you have: same findings, same examples, same fingerprint id on one core or on sixteen. Cap it with -j or DROPOUTT_WORKERS if you are sharing a node.

New here? docs/getting-started.md.

What it catches

Bugs that waste a whole training run without appearing in the logs:

• Records that train nothing — empty loss masks from role-name mismatches (from: "gpt" vs role: "assistant").
• Truncation that removes the answer — including cases where the entire assistant span falls beyond --seq-len.
• Benchmark contamination — Tülu 3 rule against bundled hashed 8-gram indices.
• Files that are not training data — agent session logs and telemetry that look like chat.
• Directional overlap — a small set wholly contained in a large one.
• Language damage — e.g. Turkish that lost its diacritics (degil mi). (Coverage shape — specialised vs broad, missing subject areas, and regions of near-identical writing that shingle dedup cannot see — comes from dropoutt atlas, below.)

dropoutt atlas — the coverage map

dropoutt atlas ./my-corpus

The atlas is a frozen topical map compressed from public datasets: one product, atlas-v3, with 4,096 cells over 256 subject areas in 128 dimensions, fitted once on 163 million records from 244 public sources — 213 GB of text, roughly 85 to 100 billion tokens depending on the tokenizer, in 95 languages. There is nothing to choose and nothing to configure; the command places on it in a terminal and in CI alike. Frozen is the point — a coverage plot that fits UMAP or k-means on the sample in front of it gives the next folder a new projection, so its neighbourhoods mean something different and two runs cannot be compared. Here the bins already exist, and a run only decides which of them your records fall into.

It is a separate command from 1.3, and was a section of the scan report before that. Two reasons. It answers a different question — where the corpus sits, not what is wrong with it, and coverage is right or wrong only against a goal the tool has not been told. And it costs differently: placement runs every sampled record through a neural encoder that has to be downloaded once, which is the one part of a scan whose cost had nothing to do with which checks were enabled.

It writes atlas.html, atlas.md and atlas.json, and reports:

--sampling is the resolution knob: omit it for the default of 500,000 records, pass a count, or pass 0 for every record. A count larger than the corpus is the same as 0. Nothing here can fail a build.

Every cell and subject area on atlas-v3 carries a hand-written name, shown as a caption and never as a finding: no record is tested against a name, and renaming a cell changes no assignment. What the map is trusted for is geometry. Details, and what the map is built from: docs/atlas.md.

Exit codes

Check catalog

Identifiers are T{tier}-{GROUP}-{nnn} and are never renumbered. Mute by id in dropoutt.toml. Full narrative: docs/checks.md. Live list: dropoutt checks.

Tier 0 — structural (CPU)

Tier 1 — statistical

Every finding in this release is labelled unverified: no calibration corpus yet links acting on a finding to a measured change in model quality.

Progressive disclosure

Documentation

• Getting started
• CLI reference
• Check catalog
• Fingerprint
• Atlas
• Configuration
• Portability / offline
• Limitations
• Design rules

What it will not do

• Fail a run whose purpose you never declared (--target).
• Tell you to delete data it has not measured for downstream effect.
• Write raw PII values into reports (matches are masked).
• Extract text from PDFs — point it at extracted text instead.

Develop

pip install -e '.[dev]'
pytest -q
ruff check .
python -m build && twine check dist/*

Lint rules live in pyproject.toml, and every rule that is switched off says why. Two conventions are worth knowing before reading the source:

• Imports go inside functions wherever the import is expensive or optional. dropoutt --help should not pay for numpy, tokenizers and the atlas.
• Every fast path has a slow one beside it. tests/test_fastpaths.py checks the vectorised implementations against the obvious ones they replaced. The contamination hashes in particular are frozen — the shipped .idx files are tables of exactly those numbers, and no benchmark text exists anywhere to recompute them from.

Licence

dropoutt is Apache-2.0.