gradsnitch

it tells on your training run

A linter/diagnoser for ML training runs. Hook your loop with one line and get plain-language verdicts on why it broke — not just charts.

Tracking (W&B / TensorBoard) shows you the loss curve. Snitch reads it and tells you what went wrong and what to try. Every finding cites concrete evidence and stays silent unless the signature is unambiguous — a wrong diagnosis is worse than none.

[GS001] [ERROR] Loss became NaN/Inf
  evidence: train_loss non-finite first at step 7 (lr=0.55)
  likely:   LR too high, fp16/bf16 overflow, or a bad batch.
  try:      Lower LR, add grad clipping, check inputs, or use bf16.

⚠️ Early project, not production-hardened. Thresholds are tuned on the bundled real-run rigs; weird curves (RL/GAN/restarts) may still surprise it. Found a false positive? That is the most useful issue you can file.

Install

pip install gradsnitch            # core (numpy + pandas)
pip install "gradsnitch[torch]"   # + torch, for watch()/framework adapters

Use

Raw PyTorch loop — one line:

import gradsnitch

mon = gradsnitch.watch(model, optimizer, check_every=50)  # prints errors live
for step in range(steps):
    loss = loss_fn(model(x), y)
    loss.backward()
    mon.log(step, loss.item(), val_loss=val)   # grabs grad_norm + lr for you
    optimizer.step(); optimizer.zero_grad()

mon.report()                                    # verdicts at the end

Framework callbacks (verified against real transformers / lightning / keras):

from gradsnitch import integrations
Trainer(..., callbacks=[integrations.hf()])         # HuggingFace
Trainer(callbacks=[integrations.lightning()])       # PyTorch Lightning
model.fit(..., callbacks=[integrations.keras()])    # Keras

Already have a run? Lint any export — column names are auto-normalized (train/loss, learning_rate, global_step, … all map):

for finding in gradsnitch.lint_csv("wandb_export.csv"):
    print(finding)

Options: mute={"GS003"} suppresses a rule by id; on_alert=integrations.wandb_alert pushes verdicts into W&B (Slack/email) on a live run.

What it catches today

Rule IDs are stable — suppressions and config pin to them, so they are never renumbered.

How it's built

• One engine, thin adapters. Pure detector functions over a metrics DataFrame; Monitor is the only sink; each framework adapter is a ~10-line extractor over a shared _feed + normalize() alias table. Torch is optional (duck-typed); each framework imported lazily.
• The harness is the moat. tests/test_real_runs.py trains tiny real torch models that break through real mechanisms (LR 1e4→NaN, 8-pt set→overfit, frozen→plateau, corrupted batch→spike, deep-sigmoid→vanishing, tiny/big LR→update ratio off, half-length LR schedule→dead second half) plus negative rigs that must stay silent (converged val, terminal spike, short noisy learner, momentum→decaying oscillation, a correct full-length decay). 25 rigs; correctness here is emergent across adapters, so this is where the value lives.

What could be added (roadmap, not done)

• More detectors: dead-ReLU / saturation from activation hooks (the metrics-only cousin — grad_norm collapse — already ships as GS007).
• v2 flagship — Cockpit's Alpha (α): a principled "LR too high/low" verdict from the loss curvature along the update direction. Needs per-sample grads, so it's intrusive (breaks log-only/torch-optional) — a separate opt-in mode.
• Gradient-noise / batch-size test (Cockpit/McCandlish).
• Rule-catalog docs page + richer mute/config (Cleanlab-style).
• Run history — compare a run to your last N (Aim-style store).

Adding a rule or an adapter: CONTRIBUTING.md. Design notes: PLAN.md.