PulseGrad (pug)

Live training telemetry and CLI for PyTorch.

Training instability — loss spikes, dead adapters, vanishing gradients, precision underflow — shows up first in the gradient/weight/activation stream, not the loss curve. By the time loss visibly degrades you have often lost minutes to hours of compute. PulseGrad instruments the optimizer step boundary directly and turns that stream into localized, typed findings rather than raw numbers a human has to interpret.

Status: early development. The collector core (hooks, stat reducers, local SQLite store, CLI) is implemented. Read the roadmap before depending on this in production.

Install

pip install pulsegrad

PyTorch is a peer dependency — PulseGrad never installs or pins it, so your existing CUDA/ROCm/CPU build is left exactly as it is. Bring your own torch.

For Hugging Face integrations (transformers, peft):

pip install "pulsegrad[hf]"

Quickstart

import pulsegrad as pug

pug.init()  # one line, before training starts — that's all

for batch in loader:  # a completely vanilla loop, no pulsegrad calls
    loss = model(batch).loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Model, optimizer, steps, loss, learning rate, and end-of-run status (completed/crashed, even on unhandled exceptions) are discovered and recorded automatically — no per-step calls, no close().

When auto-discovery cannot identify the right model — and it says so loudly when that happens — pulsegrad.watch(model, optimizer) is the explicit form. Both return the same Session.

Then, from a shell:

pug ls        # runs recorded so far
pug summary   # one-screen report on the latest run

What the collector does

• Zero-touch capture — gradients read at the optimizer-step boundary (post-accumulation, semantically final), weights post-step, activation stats via sampled forward hooks. Per-module rows go to a local SQLite store behind a non-blocking writer thread.
• Multi-model training (GAN, DPO, distillation) — call watch() once per model; each gets its own run. Losses are scoped to the correct run by walking the loss tensor's autograd graph.
• DDP-aware — under torchrun, only rank 0 records, since post-allreduce gradients and weights are identical across ranks. Other ranks get an inert session, so the same script runs unmodified. PULSEGRAD_ALL_RANKS=1 opts into per-rank runs.
• Post-hoc CLI summary — pug summary prints status, loss trajectory, NaN/Inf health with the first offending step and modules, top grad-norm modules, dead activations, and throughput.
• Kill-switch — PULSEGRAD_DISABLED=1 makes the whole collector a no-op.

import pulsegrad resolves its public API lazily and does not import torch, so adding it to a script costs approximately nothing until you call init().

Known limitations

• Multi-evaluation optimizer closures aren't supported yet. Optimizers whose step(closure) calls the closure more than once per step — torch.optim.LBFGS is the common case — desync PulseGrad's internal stats pairing and silently record no data for the affected steps. Training itself is unaffected; you just won't see anything in pug ls for that run. Tracked for a fix; everything else in this document is unaffected.

Documentation

• Researcher guide — init()/ watch() usage, per-training-type guides, detector catalog.
• Repository — design docs, benchmarks, and roadmap.

License

Apache-2.0. See LICENSE.