🩺 runmedic

A doctor for your training runs

Point it at your logs. Get a diagnosis with evidence and the fix to try.

English · Русский

_{A real PyTorch run with a broken LR schedule. runmedic finds the NaN, the gradient blow-up before it, and the schedule that caused it.}

Why

Loss went NaN overnight. Val loss has been creeping up for two days. The GPU sits at 40%. All the curves are in TensorBoard or W&B, but reading them still takes experience and time.

runmedic reads the same logs and tells you:

• 🔎 what is wrong, with the step where it started
• 📊 why it thinks so, with the numbers from your run
• 🛠️ what to change, as concrete code-level fixes

It also ships as a skill for coding agents (Claude Code, Codex and others), so the agent can run the diagnosis, open your training code, find the line responsible and propose a fix.

Quick start

pip install runmedic
runmedic diagnose runs/        # a log dir, a file, mlflow.db or a W&B run URL

No install needed with uv: uvx runmedic diagnose runs/

How it works

flowchart LR
    A["Your logs<br/>TensorBoard · MLflow · W&B<br/>Lightning · HF Trainer · stdout"] --> B["runmedic<br/>13 checks"]
    B --> C["Report<br/>evidence + fixes"]
    B --> D["JSON"]
    D --> E["Coding agent<br/>reads your code,<br/>proposes a diff"]
    E --> F["Next run<br/>runmedic confirms the fix"]

What it catches

A healthy run gets a short "No problems found". The thresholds are tuned to stay quiet unless something is clearly off.

More examples

Overfitting on a tiny noisy dataset:

A slow __getitem__ and a checkpoint saved every 50 steps:

All four demo runs are in examples/logs, produced by examples/broken_training.py.

Supported logs

Metric names are recognized automatically. runmedic series PATH shows how they were read, --train-loss NAME and friends override it.

Commands

runmedic diagnose runs/exp1                      # readable report
runmedic diagnose runs/ --json                   # for agents and scripts
runmedic diagnose mlflow.db --run my-run         # pick runs by name
runmedic diagnose train.log --fail-on critical   # exit code 1 for CI
runmedic series runs/exp1                        # what was found and how it was read

Agent skill

Send this to your agent and it installs everything:

Install the runmedic skill following these instructions: https://raw.githubusercontent.com/vincere-mori/runmedic/main/INSTALL.md

Then ask it things like "why did my training blow up?" or "why is the GPU at 40%?". The skill walks the agent through the whole loop: find the logs, diagnose, read the code, explain the cause, propose a minimal diff, verify on the next run. It comes with a playbook of fixes per problem.

/plugin marketplace add vincere-mori/runmedic
/plugin install runmedic@runmedic

git clone --depth 1 https://github.com/vincere-mori/runmedic.git
mkdir -p ~/.agents/skills && cp -r runmedic/plugins/runmedic/skills/runmedic ~/.agents/skills/

Under the hood

Contributing

The most useful thing you can send is a run that runmedic got wrong: open an issue with the logs (or a CSV of the metrics) and what the real problem was. New checks and log formats are welcome too. Each check needs a test with a curve that triggers it and one that must stay quiet.

pip install -e . pytest tensorboard ruff
pytest && ruff check src tests examples && ruff format --check src tests examples

License

MIT