Skip to main content

trainlint

Agent skills and an offline checker for ML bugs that do not crash.

ci license: MIT

Cross-validated accuracy on random labels: 85% with a leak, trainlint pointing at it, 48% after the fix

The 5,000 features and the labels are random, so no model can beat 50%. Picking features before cross-validating reports 85%. The GIF is rendered from the real output of the commands shown; assets/make_assets.py rebuilds it.

Install

uvx trainlint check .

or pipx install trainlint / pip install trainlint. It needs Python 3.10 or newer, has no dependencies, and never touches the network or a GPU.

Quickstart

uvx trainlint check train.py     # find the bugs in a script, a folder, or a repo
uvx trainlint skills install     # teach your coding agent to avoid them

skills install copies four skills into ./.claude/skills. Use --agent agents for ./.agents/skills, --global for your user folder, or --dest DIR for any other place.

More ways to run the checker:

trainlint check src/ --ignore TL005   # skip a rule
trainlint check . --format json       # for scripts and CI
trainlint rules                       # list every rule

Exit code 0 means no findings, 1 means findings, 2 means a usage error. To silence a finding on one line:

scaler.fit(X_all)  # trainlint: ignore[TL002] fitting on the pooled set is intended here

In GitHub Actions:

- run: uvx trainlint check .

Why it exists

ML code fails quietly. A scaler fitted before the split, a model left in eval mode, or a softmax in front of CrossEntropyLoss still trains and still prints a number. Coding agents and people both write these mistakes, and general linters do not know what the code is for. trainlint is a small offline checker for the common cases, plus skills that tell an agent how to avoid them in the first place.

What it does

  • Twelve rules, TL001 to TL012, listed below and explained with examples in docs/rules.md.
  • Each finding shows the line, a pointer into the code, and a one-line fix.
  • Pure standard library. It parses the code and never runs it.
  • Text or JSON output, rule selection, and per-line ignore comments.
  • Four Agent Skills that cover the same ground and run the checker when it is installed.
Rule Flags
TL001 Scaler, encoder, selector or resampler fitted on all rows before a split or cross-validation
TL002 Transform or model fitted on a validation or test split
TL003 model.eval() inside a training loop with no model.train() to switch back
TL004 Evaluation under no_grad in a script that never calls model.eval()
TL005 Training script that never sets a random seed
TL006 float16 autocast without a GradScaler, or a scaled loss stepped without scaler.step
TL007 Script that resumes from checkpoints but saves no optimizer state
TL008 softmax or sigmoid at the end of a model trained with a logits loss
TL009 Gradients never cleared, or cleared between backward() and step()
TL010 Loss added to a running total or list as a tensor, keeping every graph alive
TL011 y_true and y_pred swapped, or class labels passed to roc_auc_score
TL012 Validation or test DataLoader with drop_last=True
Skill Covers
ml-data-leakage Splits, pipelines, grouped and time-ordered data, resampling, the test set
ml-training-loop Modes, gradient order, loss inputs, shapes, seeds, mixed precision
ml-metrics-eval Argument order, scores versus labels, averaging, imbalance, variance
ml-checkpoint-resume What a checkpoint must hold, best versus last, verifying a resume

How far to trust it

The checks parse each file with Python's ast module and match calls by name. There is no type inference and no cross-file analysis, so:

  • Splits are recognized by variable names such as X_train, val_loader and y_test.
  • A bug whose cause sits in another file is not seen.
  • A heuristic can be wrong. When it is, add an ignore comment and open an issue.

python -m bench runs the checker over a corpus of 18 buggy scripts, one or two per rule, each with a fixed twin:

rule    buggy files  caught   false alarms on fixed twins
TL001             3       3                             0
TL002             1       1                             0
TL003             1       1                             0
TL004             1       1                             0
TL005             1       1                             0
TL006             2       2                             0
TL007             1       1                             0
TL008             2       2                             0
TL009             2       2                             0
TL010             1       1                             0
TL011             2       2                             0
TL012             1       1                             0
total            18      18                             0

I wrote that corpus together with the rules, so it guards against regressions and says nothing about accuracy on code I have not seen.

For a check on someone else's code, I ran it over all 93 Python files of pytorch/examples at commit acc295d and read every finding:

git clone https://github.com/pytorch/examples && cd examples
git checkout acc295dc7b90714f1bf47f06004fc19a7fe235c4
trainlint check .

It reports 21 findings. By my reading, 19 are real: 15 scripts never set a seed (one declares a --seed option and never uses it), 3 tensor-parallel scripts never clear gradients, and one script resumes from a snapshot without saving optimizer state. Two more are correct as written but harmless, because those models have no dropout or batch norm to leave in training mode. The first run also flagged three distributed-autograd scripts for missing zero_grad(), which was a false positive that is now fixed. This is one repository of tutorial code and I did the reviewing, so treat it as a spot check. It tells you nothing about what the rules miss.

Compared with other tools

Tool What it covers Where trainlint differs
ruff, pylint, flake8 General Python style and bugs None of their rules know about splits, eval mode or metrics. Use them alongside trainlint.
TorchFix Removed and deprecated PyTorch functions, requires_grad typos, unsafe torch.load Its README lists no rules for training logic. trainlint does not check API deprecations.
mllint Repository-level quality of ML projects: version control, dependencies, code quality It looks at project hygiene. trainlint reads training and evaluation code.
Deepchecks Checks that run on real datasets and models, including drift Deepchecks needs your data and model loaded. trainlint reads source only and runs nothing.
General skill collections Engineering process for coding agents They are not specific to ML. trainlint's skills target training and evaluation code and come with a checker.

Roadmap

  • Jupyter notebook support
  • SARIF output for GitHub code scanning
  • Settings in pyproject.toml
  • More rules: training loaders without shuffling, retain_graph in a loop, normalization on a whole DataFrame before the split
  • A benchmark that measures how often an agent writes these bugs with and without the skills (needs an API key, so it will be optional)

Issues labeled good first issue are small and have a clear definition of done.

Contributing

False positives and missed bugs are the most useful reports. See CONTRIBUTING.md for setup and for how to add a rule.

uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check .
uv run python -m bench

License

MIT

If trainlint saved you an afternoon, a star on the repository helps other people find it.

Metadata

Release files for trainlint 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for trainlint 0.1.1
File Size Uploaded Provenance Actions
trainlint-0.1.1.tar.gz 222.5 kB Without provenance Details for trainlint-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for trainlint 0.1.1
File Interpreter ABI Platform Actions
trainlint-0.1.1-py3-none-any.whl Python 3 none any Details for trainlint-0.1.1-py3-none-any.whl

Total release size: 253.8 kB

Release files / trainlint-0.1.1.tar.gz

Download URL trainlint-0.1.1.tar.gz
Size 222.5 kB
Tags Source
SHA-256 checksum
How to use checksums
ce483d123c03c12e862c338e3aea0e612e4a2f90c4604f81db1ea8b827c58cf6
BLAKE2b-256 checksum
How to use checksums
ea729f293ec9462af88e96654c4da118dde1c6106c2eab37f4bbafabcf26f8c8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.1 {"installer":{"name":"uv","version":"0.13.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / trainlint-0.1.1-py3-none-any.whl

Download URL trainlint-0.1.1-py3-none-any.whl
Size 31.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3b71675059ee1a7783dce7178cc78acd3d5dcdd24b56c112054b2c2ae16e20ef
BLAKE2b-256 checksum
How to use checksums
d54f917ac5de5678416ec28f7518dbb0d91aa89e69a1a418c2a2bfe40482b8a1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.1 {"installer":{"name":"uv","version":"0.13.1","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":null,"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for trainlint 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page