ormas-monitor

Catch the layer that breaks while your model trains, and get proof.

ormas-monitor watches a PyTorch training run layer by layer. Each layer is compared with its own history, so the monitor names the layer that failed. When a channel can never recover, it says so with a certificate, not a guess. It can repair those channels if you ask, and it writes a tamper-evident record of everything it saw.

Everything runs on your machine. The core makes no network calls and never reads your data; only the demo downloads CIFAR-10, and only sinks you add (W&B, MLflow, a webhook) send anything.

Why use it

• Alerts you can automate. Other tools say a layer looks dead. This one issues a certificate only when the channel provably cannot recover, so it is safe to drive repairs and release gates (check --fail-on warning) without a human in the loop.
• Find dead weight you already ship. In 19 of 21 torchvision ImageNet models that use BatchNorm → ReLU, it found channels decayed to ~0, up to 9.9% of the checked channels in DenseNet-169. Switching them off left the answers unchanged; switching off the same number of live channels changed them on 98–100% of images. In VGG-19-BN, cutting them out saved 18.6% of the compute and ran 12–18% faster on CPU, with identical answers (details).
• A signed record of what happened in training, plus a one-page summary for non-engineers.
• Nothing to trust us with: open source, local, never reads your data.

See it end to end: five-minute tutorial with real output. More detail: Why use it.

Three ways in

1. Check a saved model (no code changes; a 98 MB ResNet-50 file takes under a second)

pip install "ormas-monitor[safetensors]"
ormas-monitor check model.safetensors                  # or .pt / .pth / .bin / .ckpt, or a Hugging Face folder
ormas-monitor check finetuned.pt --against base.pt     # what changed, and what died
ormas-monitor check model.pt --fail-on warning         # non-zero exit for a CI gate

2. Two lines in a training loop, or one callback

from ormas_monitor import Monitor

mon = Monitor(model, optimizer)        # repair=True to re-initialise stuck channels
for x, y in loader:
    loss = loss_fn(model(x), y)
    optimizer.zero_grad()
    loss.backward()
    mon.step(loss)                     # after backward, before optimizer.step()
    optimizer.step()
mon.close()                            # writes the record and reports to ormas_run/

A new Monitor refuses to overwrite an existing record in its run directory (FileExistsError); pass another out= or overwrite=True.

Using Lightning or the Hugging Face Trainer? Add one callback instead, ormas_monitor.integrations.lightning.OrmasMonitorCallback or ormas_monitor.integrations.huggingface.OrmasMonitorCallback: see Lightning and Hugging Face.

3. Watch it catch a broken layer (about a minute on an Apple M-series Mac, after the download)

pip install "ormas-monitor[demo]"
ormas-monitor demo        # CIFAR-10: two layers are zeroed; watch-only vs repair (downloads CIFAR-10 once)

Or open the real reports without installing anything: sample_reports/.

What it found (verified)

All numbers: one seed (0), one Apple M-series Mac (training on its GPU via MPS, torch 2.11), CIFAR-10 unless stated. Exact accuracies vary with torch version and hardware (a fresh install with torch 2.14.1 got 61.96% / 10.00% / 62.67%, 191 repaired, on the quick demo); the qualitative result holds. Not yet measured on NVIDIA/Linux, multi-GPU or at scale.

Details and exact reproduce commands: docs/results.md.

The certificate, in one paragraph

In a Conv/Linear → BatchNorm → ReLU channel, if the incoming filter or the BatchNorm scale is exactly zero and the BatchNorm shift is ≤ 0, the ReLU is closed for every input. Every gradient into the channel is then exactly zero, so SGD or AdamW can never revive it (Theorem 2 of the DeepMath 2026 paper below). The monitor issues the certificate only when, checked at run time: the BatchNorm reads the layer's output and a ReLU module is its only consumer (from the autograd graph of the first sampled step); the channel's gradient is exactly zero at that check; the optimizer is SGD or one of the Adam family, holds no momentum for the channel, and its weight decay cannot lift the shift above zero; and BatchNorm is in training mode. Otherwise it records why it refused. Conditions and known gaps.

What it is not

• Not ORMAS. The ORMAS training architecture is not in this package; this tool needs no architecture change.
• Not compliance. The record is evidence for your technical documentation, not a certificate of compliance with the EU AI Act or anything else.
• No proof for transformers or LLMs. The certificate needs BN→ReLU. Transformers get alarms, loss checks and the record.
• Never reads your data. It cannot find mislabelled examples.
• No uploads. Nothing leaves your machine unless you add a sink that sends data (W&B, MLflow, a webhook).
• Not the first to notice dead units. ReDo (Sokar et al., ICML 2023) and network slimming came first. We claim the certificate and the packaging.
• Not yet tested on NVIDIA/Linux, multi-GPU or at scale.

Full list: docs/limitations.md.

Install

pip install ormas-monitor                     # core: depends only on PyTorch
pip install "ormas-monitor[safetensors]"      # read .safetensors files
pip install "ormas-monitor[signing]"          # Ed25519 signing of the record
pip install "ormas-monitor[lightning]"        # Lightning callback
pip install "ormas-monitor[huggingface]"      # Hugging Face Trainer callback
pip install "ormas-monitor[tensorboard]"      # also: [wandb], [mlflow]
pip install "ormas-monitor[demo]"             # torchvision, for `ormas-monitor demo`
pip install "ormas-monitor[all]"

Python 3.9+, PyTorch 2.3+ (declared). Run so far on Python 3.13 with PyTorch 2.11 (macOS); the CI matrix (Linux, macOS, Windows; Python 3.9 and 3.12; CPU) is set up but has not run yet. Status: OPEN.

Documentation

Quickstart · Concepts · Guides · CLI reference · Python API · For leaders · Security and privacy · FAQ

Hosted: https://oxiedo.github.io/ormas-monitor/

Work with us

ormas-monitor is free and stays free. Oxiedo is looking for its first design partners: teams who train their own models, especially vision models, and want help getting more out of their runs.

• What you get: setup help, a review of your reports, early access to team features, and a say in what we build next.
• What we ask: a 30-minute call and honest feedback.

Email raadh@oxiedo.com (subject: "design partner"), or read Work with us.

Not technical? Start with For leaders: what it is in plain words, and what you would receive.

Citation

If you use the dead-channel certificate, please cite the paper (CITATION.cff):

@inproceedings{raadh2026selfrepair,
  author    = {Raadh, Rokib Al Dhin},
  title     = {Self-Repair as a Bounded Disturbance: Input-to-State Stability of Neural Network Training Dynamics},
  booktitle = {DeepMath 2026},
  address   = {The Ohio State University, Columbus, OH},
  year      = {2026},
  note      = {Poster}
}

Contact

Built by Rokib Al Dhin Raadh at Oxiedo. Questions, bugs, or a run you'd like a second pair of eyes on: raadh@oxiedo.com. Security issues: see SECURITY.md.

Licensed under Apache-2.0.