Quick Start

pip install arc-training

import arc

controller = arc.Controller(model, optimizer)

for batch in dataloader:
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()

    action = controller.step(loss)
    if action.should_skip:
        continue          # ARC already rolled back; skip this update

    optimizer.step()

controller.summary()      # end-of-run report: failures, rollbacks, LR changes

Call controller.step(loss) after loss.backward() and before optimizer.step(). No changes to your model, loss, or optimizer.

git clone https://github.com/a-kaushik2209/ARC.git
cd ARC
pip install -e .

pip install "arc-training[hf]"          # Hugging Face Trainer callback
pip install "arc-training[lightning]"   # Lightning callback
pip install "arc-training[dev]"

ARC never changes a healthy step. It doesn't clip gradients unless you ask (ControllerConfig(enable_baseline_clipping=True)).

controller = arc.Controller(model, optimizer, scaler=scaler)

scaler.scale(loss).backward()
action = controller.step(loss)      # ARC unscales; don't call scaler.unscale_()
if not action.should_skip:
    scaler.step(optimizer)
    scaler.update()

from arc.api.integrations import HuggingFaceCallback, LightningCallback

Trainer(..., callbacks=[HuggingFaceCallback()])      # transformers
L.Trainer(..., callbacks=[LightningCallback()])      # lightning

Full guide: pyarc.pages.dev/docs.html

The Problem

Training neural networks is fragile. A single NaN gradient at hour 47 of a 48-hour run can destroy days of compute. Engineers waste enormous time adding manual checkpointing and babysitting long runs.

• NaN explosions: the loss becomes NaN or Inf with no warning and corrupts the run.
• Gradient collapse: vanishing or exploding gradients stop learning without any alert.
• Wasted compute: every failure you can't recover from burns GPU-hours you already paid for.
• Manual monitoring: engineers watch runs overnight and write their own recovery scripts and checkpointing logic.

How It Works

Controller.step(loss)
    |
    +-- warm-up: auto-calibrate thresholds (first 50 steps; NaN/Inf still caught)
    +-- NaN / Inf loss                    immediate
    +-- loss explosion                    absolute threshold + 10x spike vs recent mean
    +-- NaN / Inf weights, weight health  every 10 steps: sparsity, magnitude collapse
    +-- NaN / Inf gradients               true scale (GradScaler-aware)
    +-- gradient explosion                norm threshold (clipping only if enabled)
    +-- Lyapunov divergence               predictive, every 5 steps
    +-- trend drift                       OLS slope over a sliding window
    +-- DDP: every rank agrees            one all-reduce, so replicas never diverge
    |
    +--[FAILURE]--> restore snapshot (model + optimizer + RNG), skip the step
    |               reduce LR x0.5 (undone after 500 healthy steps)
    |               5 rollbacks within a few steps -> escape: oldest snapshot,
    |                   perturb weights, 2 extra LR cuts, skip 5 steps
    |               3 escapes with no progress -> unrecoverable (warns once)
    |
    +--[HEALTHY]--> snapshot copied async after optimizer.step(),
                    kept once the next step passes; adaptive frequency

Capabilities

Examples for CNNs, YOLO, Transformers, LLM fine-tuning, PINNs, RL, and more: pyarc.pages.dev/examples.html

Configuration

from arc import Controller, ControllerConfig

config = ControllerConfig(
    checkpoint_frequency=20,
    calibration_steps=50,
    lr_reduction_factor=0.5,
    enable_lyapunov=True,
    enable_trend=True,
    gradient_accumulation_steps=4,   # only check full accumulation steps
    verbose=False,
)
controller = Controller(model, optimizer, scheduler=scheduler, config=config)

All defaults are listed in the configuration reference.

Architecture

arc/
  core/            Controller (canonical API), recovery ledger
  signals/         Lyapunov, trend, gradient, weight, RL + rollout health
  features/        Feature extraction and buffering
  prediction/      Failure predictor, remediation policy engine
  intervention/    Rollback, OOM, hardware, DDP, model heuristics
  checkpointing/   Adaptive + signed checkpoints, DR drill
  distributed/     DDP rollback, straggler detection
  evaluation/      Reliability metrics, run comparison, chargeback
  security/        SDC detection, health baseline, adversarial
  api/             Alert router, enterprise report, callbacks
  introspection/   Hessian + Fisher analysis (experimental)
  physics/         PINN stabilization
  uncertainty/     Conformal prediction
  learning/        Failure meta-model, EWC

• One canonical entry point: Controller.step(loss) runs every detector in order and returns a StepAction with the health status, the failure type, and what ARC did about it.
• Toggleable signals: the Lyapunov, trend, gradient, and weight-health monitors switch on or off independently in ControllerConfig, without touching recovery logic.
• Auditable by default: every intervention lands in a hash-chained ledger you can export, verify, compare across runs, and roll up per team.
• Minimal dependencies: PyTorch, NumPy, and SciPy. Optional integrations for Lightning, Hugging Face, and Weights & Biases.

Works with any architecture: CNN / ResNet, Transformers, YOLO / ViT, PINNs, GPT / LLMs, Diffusion UNet, RNN / LSTM, GANs.

Validated on GPT-2 Medium, Large, and XL (1.5B), a 2.6B-parameter custom transformer, BERT, ResNet-18/50, VGG11, ViT-Base, DINOv2, YOLOv11, Llama-style models, and Stable Diffusion UNet.

ARC Lens (VS Code)

ARC Lens is a self-contained VS Code extension for real-time PyTorch training visualization: live losses, learning rates, gradient norms, GPU memory, and layer effective rank, plus anomaly detection and a bundled offline recovery agent. No data leaves your machine.

Benchmarks

Every number comes from a script in experiments/ that you can re-run.

Fault-injection recovery (ARC 5.2.0)

experiments/recovery_benchmark.py, 5 seeds, CPU. Each trial trains the same MLP on the same data twice, once unprotected and once with arc.Controller, and injects the same fault at step 60 of 150. Both runs are judged by the same checks at the end: no exception, every weight finite, and held-out loss no more than 2x its pre-fault value.

Under persistent NaN data, rollback can't fix the input. In all 5 trials ARC kept the model intact and flagged the run unrecoverable instead of looping. In 20 transient trials ARC rolled back for a fault that unprotected Adam would have absorbed (for example a x1e10 loss spike). That cost a few steps but never left a worse model. The control group is small; false-positive rates on full-size training recipes have not been measured yet.

Results from ARC 5.1 scripts (pending re-measurement)

An October 2026 audit found that the 5.1 stress suites (Nightmare, Comprehensive, Silent Failure) never applied their crash checks to the ARC-protected run, so their "100% recovered" figures meant "didn't raise an exception". Those suites are superseded by the benchmark above. The tables below come from 5.1 code (some from legacy APIs). They are kept for reference until they're re-run on 5.2.

Baseline comparison

Billion-scale rollback (NVIDIA H200)

A 40 to 50 step fine-tune with one NaN loss injected at step 30, using the legacy WeightRollback. The unprotected run is stopped at the NaN by the script, so this shows that rollback fits and runs at 2.65B parameters, not a recovery rate.

Real-dataset recovery

Failure prediction (research)

Classifiers trained and tested on simulated training curves. No trained predictor ships with the package, so FailurePredictor reports risk_level="unknown" unless you load your own model.

Overhead (GPU)

Measured with experiments/controller_overhead.py: GPT-2 (124M, random init), batch 4, sequence 128, AdamW, 400 steps in interleaved blocks of 50 (median step time).

Reproduce: python experiments/controller_overhead.py --steps 400 --batch 4 --seq 128

Full results, protocols, and ablations: pyarc.pages.dev/benchmarks.html

Known Limitations

Community

The Discord server is the primary hub for contributor discussions, development updates, research discussions, feature proposals, and debugging support. Contributors, researchers, students, and developers of all experience levels are welcome.

Contributing

1. Open an issue describing the feature, fix, or improvement.
2. Wait for the issue to be assigned to you before starting work. PRs without prior discussion or assignment may be closed.
3. Branch, make a focused change, and open a PR that references the issue.

Full workflow and standards: CONTRIBUTING.md · Release history: CHANGELOG.md

Citation

@article{kaushik2026arc,
  title   = {ARC: Autonomous Recovery Controller for Fault-Tolerant Neural Network Training},
  author  = {Kaushik, Aryan},
  year    = {2026},
  note    = {Maharaja Agrasen Institute of Technology, New Delhi}
}