trainproof 0.3.0

A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.

trainproof

A deterministic linter for ML training runs. Point it at your dataset, your tokenizer, or your first-epoch logs — it returns a PASS / WARN / FAIL verdict with named findings and cited evidence, before you burn days of GPU time on a run that was doomed at step 50.

No ML judging ML. No invented "confidence 97%". Every rule is a deterministic threshold in one auditable module, and every finding cites the numbers that triggered it.

pip install trainproof

See a verdict in 60 seconds

This repo ships the real logs of five QLoRA fine-tuning runs (Qwen2.5-3B, RTX 5080 — see the gallery below). Judge one right now:

trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf

[FAIL] Critical checks failed:
  [FAIL] Loss curve is diverging.
         Evidence: End loss 7.492 vs Min loss 1.398
  [WARN] Gradient norm spikes detected.
         Evidence: Max gn 2649.75 > 10.0x median (0.55)

Why this exists

The author lost a real 11-hour fine-tune to a failure nothing warned about. Pointed retroactively at that run's 1MB Coqui Trainer log (2,501 logged steps), trainproof's verdict: FAIL — diverging. The loss reached its minimum at 82% of the run and ended 1.9x above it. Translation: the final two hours of GPU time made the model measurably worse, and the checkpoint worth keeping had already existed for hours. No tool in the stack said a word.

The fault-injection gallery

To validate the rules, the same QLoRA fine-tune (Qwen2.5-3B-Instruct, 4-bit, LoRA r=16, 300 steps on Alpaca-cleaned) was run five times — once healthy, four times with exactly one knob deliberately broken. Real runs, real logs, all shipped in examples/gallery/:

Run	Sabotage	Verdict	Key evidence
healthy	none	PASS	loss 1.52 → 0.94, stable gradients
lr_hot	LR x100 (2e-2)	FAIL	diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55
lr_zero	LR = 0	FAIL	dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps
fp16_nan	fp16 + hot LR, no clipping	FAIL	diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed)
bad_labels	labels shuffled per-sequence	WARN only (single-run) — caught by trainproof compare (v0.3)	grad spike 23.3 vs median 1.09

The honest finding: loss curves cannot see corrupted data

The bad_labels run — whose shuffled labels make real learning impossible — reduced its loss by 62% (18.9 → 5.75). The model was genuinely learning: not the task, but the marginal token statistics of the garbage. From its own loss curve, that is indistinguishable from healthy training (neural networks famously fit random labels). No single-run, loss-only rule can catch this class of failure — its real signature is relative: a loss floor ~6x higher than a known-good run of the same task (5.59 vs 0.94).

That finding sets the roadmap: v0.3 is trainproof compare <run> <baseline> — deterministic ratio rules against the healthy baseline you already have. See ROADMAP.md. (The gallery also improved v0.2 itself: the dead-run rule exists because lr_zero initially escaped with only a WARN.)

The three commands

# 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
#    duplicates, text-vs-audio duration mismatches
trainproof data /path/to/dataset_or_manifest.jsonl

# 2. Tokenizer preflight: vocabulary coverage, OOV rate, sequence blowouts,
#    suspicious splits on numbers/dates
trainproof tokenizer my_tokenizer.model transcripts.txt

# 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
#    LR sanity, throughput — from log files, any framework
trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready

# 4. Compare against a baseline
#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json

========================================
TRAINPROOF VERDICT
========================================
[FAIL] Critical checks failed:
  [FAIL] loss floor ratio exceeded limit
         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
  [FAIL] end loss ratio exceeded limit
         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
========================================

Each command prints the verdict, writes a self-contained HTML report, and sets the process exit code — so it works as a CI gate out of the box.

Supported log formats

• HuggingFace Trainer (trainer_state.json)
• Coqui Trainer text logs (ANSI-colored trainer_0_log.txt)
• Generic JSONL / CSV (columns: step, loss, lr, grad_norm, time — all optional)

Auto-detected; override with --format hf|coqui|jsonl|csv. TensorBoard event files are planned (Lightning console captures are TTY dumps, not logs, and will not be supported).

Philosophy

1. Deterministic. A rule fires or it doesn't. Thresholds live in one module, commented, tunable.
2. Evidence-cited. Every finding names the steps and values that triggered it.
3. Honest about limits. What the tool cannot detect is documented in the README, not discovered by the user in production.

Family

trainproof judges training runs. Its sibling ttsproof judges TTS model outputs (structural audio checks, equivalence-aware WER/CER, published method with DOI) — and trainproof builds on it for the speech dataset checks.

Author

Panagiotis (Panos) Gkilis — portfolio · bedvibe.studio

MIT license.