Quick Start

pip install arc-training

import arc

controller = arc.Controller(model, optimizer)

for batch in dataloader:
    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()

    action = controller.step(loss)
    if action.should_skip:
        continue          # ARC already rolled back; skip this update

    optimizer.step()

controller.summary()      # end-of-run report: failures, rollbacks, LR changes

Call controller.step(loss) after loss.backward() and before optimizer.step(). No changes to your model, loss, or optimizer.

git clone https://github.com/a-kaushik2209/ARC.git
cd ARC
pip install -e .

pip install arc-training[lightning]
pip install arc-training[full]
pip install arc-training[dev]

ARC clips gradients by default. On healthy steps ARC clips the gradient norm to gradient_clip_norm=1.0. If your loop already clips, set enable_baseline_clipping=False in ControllerConfig.

Full guide: pyarc.pages.dev/docs.html

The Problem

Training neural networks is fragile. A single NaN gradient at hour 47 of a 48-hour run can destroy days of compute. Engineers waste enormous time adding manual checkpointing and babysitting long runs.

• NaN explosions - loss becomes NaN or Inf with no warning, corrupting the run irreversibly.
• Gradient collapse - vanishing or exploding gradients silently halt learning, wasting compute without any alert.
• Wasted compute - cloud GPU spending exceeds $30B annually. Every unrecoverable failure burns money directly.
• Manual monitoring - engineers watch training overnight, writing custom recovery scripts and checkpointing logic.

How It Works

Controller.step(loss)
    |
    +-- warm-up: auto-calibrate thresholds (first 50 steps; NaN/Inf still caught)
    +-- NaN / Inf loss                    immediate
    +-- loss explosion                    absolute threshold + 10x spike vs recent mean
    +-- NaN / Inf weights, weight health  every 10 steps: sparsity, magnitude collapse
    +-- gradient explosion                norm threshold, then baseline clipping
    +-- Lyapunov divergence               predictive, every 5 steps
    +-- trend drift                       OLS slope over a sliding window
    |
    +--[FAILURE]--> restore checkpoint (model + optimizer + RNG)
    |               reduce LR x0.5 (re-applied after every restore)
    |               5 rollbacks within a few steps -> escape: oldest checkpoint,
    |                   perturb weights, 2 extra LR cuts, skip 5 steps
    |               3 escapes with no progress -> unrecoverable (warns once)
    |
    +--[HEALTHY]--> update history, adaptive checkpoint

Capabilities

Examples for CNNs, YOLO, Transformers, LLM fine-tuning, PINNs, RL, and more: pyarc.pages.dev/examples.html

Configuration

from arc import Controller, ControllerConfig

config = ControllerConfig(
    checkpoint_frequency=20,
    calibration_steps=50,
    lr_reduction_factor=0.5,
    enable_lyapunov=True,
    enable_trend=True,
    gradient_accumulation_steps=4,   # only check full accumulation steps
    verbose=False,
)
controller = Controller(model, optimizer, scheduler=scheduler, config=config)

All defaults are listed in the configuration reference.

Architecture

arc/
  core/            Controller (canonical API), recovery ledger
  signals/         Lyapunov, trend, gradient, weight, RL + rollout health
  features/        Feature extraction and buffering
  prediction/      Failure predictor, remediation policy engine
  intervention/    Rollback, OOM, hardware, DDP, model heuristics
  checkpointing/   Adaptive + signed checkpoints, DR drill
  distributed/     DDP rollback, straggler detection
  evaluation/      Reliability metrics, run comparison, chargeback
  security/        SDC detection, health baseline, adversarial
  api/             Alert router, enterprise report, callbacks
  introspection/   Hessian + Fisher analysis (experimental)
  physics/         PINN stabilization
  uncertainty/     Conformal prediction
  learning/        Failure meta-model, EWC

• One canonical entry point - Controller.step(loss) runs every detector in order and returns a StepAction with health status, failure type, and what ARC did about it.
• Toggleable signals - Lyapunov, trend, gradient, and weight-health monitors switch on or off independently in ControllerConfig, without touching recovery logic.
• Auditable by default - every intervention lands in a hash-chained ledger you can export, verify, diff across runs, and roll up per team.
• Minimal dependencies - requires PyTorch, NumPy, and SciPy. Optional integrations for Lightning, Hugging Face, and Weights & Biases.

Works with any architecture: CNN / ResNet, Transformers, YOLO / ViT, PINNs, GPT / LLMs, Diffusion UNet, RNN / LSTM, GANs.

Validated on GPT-2 Medium, Large, and XL (1.5B), a 2.6B-parameter custom transformer, BERT, ResNet-18/50, VGG11, ViT-Base, DINOv2, YOLOv11, Llama-style models, and Stable Diffusion UNet.

ARC Lens (VS Code)

ARC Lens is a self-contained VS Code extension for real-time PyTorch training visualization: live losses, learning rates, gradient norms, GPU memory, and layer effective rank, plus anomaly detection and a bundled offline recovery agent. No data leaves your machine.

Benchmarks

Every number comes from a reproducible script in experiments/ or from the ARC research paper.

Recovery across test suites

Baseline comparison

Billion-scale validation (NVIDIA H200)

NaN loss injected at step 30.

Real-dataset recovery

Failure prediction

Overhead (CPU)

End-to-end GPU overhead on a Tesla T4 is 40.7% (GPT-2, 124M) and 33.4% (GPT-2 Medium, 355M). Relative overhead falls as models grow.

Full results, protocols, and ablations: pyarc.pages.dev/benchmarks.html

Known Limitations

Community

The Discord server is the primary hub for contributor discussions, development updates, research discussions, feature proposals, and debugging support. Contributors, researchers, students, and developers of all experience levels are welcome.

Contributing

1. Open an issue describing the feature, fix, or improvement.
2. Wait for the issue to be assigned to you before starting work. PRs without prior discussion or assignment may be closed.
3. Branch, make a focused change, and open a PR that references the issue.

Full workflow and standards: CONTRIBUTING.md · Release history: CHANGELOG.md

Citation

@article{kaushik2026arc,
  title   = {ARC: Autonomous Recovery Controller for Fault-Tolerant Neural Network Training},
  author  = {Kaushik, Aryan},
  year    = {2026},
  note    = {Maharaja Agrasen Institute of Technology, New Delhi}
}