lm-resiliency includes two runtime safeguards to an existing training stack:

Why LLM Resiliency

• SDC-safe recovery-verified checkpoint: SCOUT certifies recovery checkpoints and excludes candidates affected by recurring SDC, preventing corrupted state from being selected during recovery.
• Localize faulty components at runtime: SCOUT identifies affected ranks, GPUs, nodes, communication endpoints, or peer groups, including communication hangs while the training job is blocked.
• Minimize rollback: GEMINI saves complete training state to CPU memory at high frequency with no measurable training-throughput loss, reducing lost computation after a failure.
• Retrieve checkpoints quickly: GEMINI restores nearby state from memory, a surviving peer, or node-local storage, minimizing checkpoint retrieval time and global-storage reads.
• Keep protection lightweight: SCOUT incurs less than 1% amortized overhead during training for runtime failure localization.
• No framework fork: lm-resiliency integrates with PyTorch, TorchTitan, Megatron Core, and DeepSpeed through automatic adapters and one public entry point.
• No training-loop rewrite: lm-resiliency attaches hooks at framework initialization and leaves the existing training loop unchanged.
• Bring your own launcher: Users can integrate lm-resiliency with torchrun, Slurm, Kubernetes, or custom managers through platform-neutral APIs.

Quick Start

Install

git clone https://github.com/LMResiliency/lm-resiliency.git
cd lm-resiliency
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e ".[megatron]"

Use .[torchtitan], .[deepspeed], .[all], or the core package without an extra for other environments.

Add resiliency to Megatron Core

Attach resiliency after Megatron creates its model chunks, optimizer, and scheduler. Then resume the existing train() loop from the recovered iteration.

from megatron.training import get_args

from lm_resiliency import enable_resiliency


def attach_resiliency(model_chunks, optimizer, scheduler):
    resiliency = enable_resiliency(
        model_chunks,
        optimizer,
        opt_param_scheduler=scheduler,
        interval=10,
    )
    get_args().iteration = resiliency.step_count
    return resiliency


# Call attach_resiliency(...) after Megatron setup, then enter train() unchanged.

See the Megatron Core production-loop example for a complete tiny-GPT job.

Add resiliency to TorchTitan

Pass the initialized TorchTitan Trainer before entering its existing training loop. The adapter discovers the model, optimizer, scheduler, dataloader, topology, and checkpoint state.

from torchtitan.train import Trainer
from lm_resiliency import enable_resiliency


class MyTrainer(Trainer):
    def train(self):
        enable_resiliency(self, interval=10)
        super().train()

See the TorchTitan production-loop example for a complete Llama debug-model job.

Framework support

The package-root enable_resiliency entry point selects dense or expert replay peers from framework topology metadata. See the API guide for framework invocation, configuration, recovery, callbacks, and lifecycle management. See the compatibility policy for supported and tested versions. See the PyTorch production-loop example for a complete DDP causal-LM job. See the DeepSpeed production-loop example for a complete ZeRO-2 causal-LM job.

Manager Integration

External launchers and cluster managers can consume normalized SCOUT reports and checkpoint RecoveryDecision records, preserve GEMINI state before worker replacement, and coordinate checkpoint transfer through platform-neutral APIs. Launcher-specific retry, placement, and replacement policy remains external. See Manager Integration in the API guide.

Documentation

Development

python -m pip install -e ".[dev]"
CUDA_VISIBLE_DEVICES="" python -m pytest -q

GPU and distributed tests are opt-in and document their required torchrun command in each test file. See the test guide for the integration and validation layout.

Contributing

Contributions to code, tests, documentation, and framework integrations are welcome. See CONTRIBUTING.md for setup, required checks, GPU validation, and pull-request expectations. Report security vulnerabilities privately according to SECURITY.md.

Citation

If you use lm-resiliency in your research, please cite the relevant papers.

GEMINI

GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints

@inproceedings{gemini-sosp23,
  title = {{GEMINI}: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints},
  author = {Wang, Zhuang and Jia, Zhen and Zheng, Shuai and Zhang, Zhen and Fu, Xinwei and Ng, T. S. Eugene and Wang, Yida},
  booktitle = {Proceedings of the 29th ACM Symposium on Operating Systems Principles},
  year = {2023},
}

SCOUT

SCOUT: Symmetric Consensus Outlier Detection for Failure Localization in LLM Pre-Training

@misc{wang2026scout,
  title = {{SCOUT}: Symmetric Consensus Outlier Detection for Failure Localization in {LLM} Pre-Training},
  author = {Wang, Zhuang},
  year = {2026},
  url = {https://arxiv.org/abs/2608.11034}
}

License

Licensed under the BSD-3-Clause License.