Skip to main content

torchft

Easy Per Step Fault Tolerance for PyTorch

| Documentation | Poster | Design Doc |

PyPI - Stable Version PyPI - Nightly Version


This repository implements techniques for doing a per-step fault tolerance so you can keep training if errors occur without interrupting the entire training job.

This is based on the large scale training techniques presented at PyTorch Conference 2024.

Overview

torchft is designed to provide the primitives required to implement fault tolerance in any application/train script as well as the primitives needed to implement custom fault tolerance strategies.

Out of the box, torchft provides the following algorithms:

  • Fault Tolerant DDP
  • Fault Tolerant HSDP: fault tolerance across the replicated dimension with any mix of FSDP/TP/etc across the other dimensions.
  • LocalSGD
  • DiLoCo

To implement these, torchft provides some key reusable components:

  1. Coordination primitives that can determine which workers are healthy via heartbeating on a per-step basis
  2. Fault tolerant ProcessGroup implementations that report errors sanely and be reinitialized gracefully.
  3. Checkpoint transports that can be used to do live recovery from a healthy peer when doing scale up operations.

The following component diagram shows the high level components and how they relate to each other:

Component Diagram

See torchft's documentation for more details.

Examples

torchtitan (Fault Tolerant HSDP)

torchtitan provides an out of the box fault tolerant HSDP training loop built on top of torchft that can be used to train models such as Llama 3 70B.

It also serves as a good example of how you can integrate torchft into your own training script for use with HSDP.

See torchtitan's documentation for end to end usage.

Fault Tolerant DDP

We have a minimal DDP train loop that highlights all of the key components in torchft.

See train_ddp.py for more info.

DiLoCo

LocalSGD and DiLoCo are currently experimental.

See the diloco_train_loop/local_sgd_train_loop tests for an example on how to integrate these algorithms into your training loop.

Design

torchft is designed to allow for fault tolerance when using training with replicated weights such as in DDP or HSDP (FSDP with DDP).

See the design doc for the most detailed explanation.

Lighthouse

torchft implements a lighthouse server that coordinates across the different replica groups and then a per replica group manager and fault tolerance library that can be used in a standard PyTorch training loop.

This allows for membership changes at the training step granularity which can greatly improve efficiency by avoiding stopping the world training on errors.

Lighthouse Diagram

Fault Tolerant HSDP Algorithm

torchft provides an implementation of a fault tolerant HSDP/DDP algorithm. The following diagram shows the high level operations that need to happen in the train loop to ensure everything stays consistent during a healing operation.

HSDP Diagram

See the design doc linked above for more details.

Installing from PyPI

Install the latest stable release:

pip install torchft

Nightly builds are available at https://pypi.org/project/torchft-nightly/

To install a nightly build:

pip install torchft-nightly

If you want all development dependencies you can install:

pip install torchft-nightly[dev]

Installing from Source

Prerequisites

Before proceeding, ensure you have the following installed:

  • Rust (with necessary dependencies)
  • protobuf-compiler and the corresponding development package for Protobuf.
  • PyTorch 2.7 RC+ or Nightly

Note that the Rust versions available in many conda environments may be outdated. To install the latest version of Rust, we recommend downloading it directly from the official website as shown in the below command:

curl --proto '=https' --tlsv1.2 https://sh.rustup.rs -sSf | sh

To install the required packages on a Debian-based system (such as Ubuntu) using apt, run:

sudo apt install protobuf-compiler libprotobuf-dev

or for a Red Hat-based system, run:

sudo dnf install protobuf-compiler protobuf-devel

Installation

pip install .

This uses pyo3+maturin to build the package, you'll need maturin installed.

If the installation command fails to invoke cargo update due to an inability to fetch the manifest, it may be caused by the proxy, proxySSLCert, and proxySSLKey settings in your .gitconfig file affecting the cargo command. To resolve this issue, try temporarily removing these fields from your .gitconfig before running the installation command.

To install in editable mode w/ the Rust extensions and development dependencies, you can use the normal pip install command:

pip install -e '.[dev]'

Usage

Lighthouse

The lighthouse is used for fault tolerance across replicated workers (DDP/FSDP) when using synchronous training.

You can start a lighthouse server by running:

RUST_BACKTRACE=1 torchft_lighthouse --min_replicas 1 --quorum_tick_ms 100 --join_timeout_ms 10000

Example Training Loop (DDP)

See train_ddp.py for the full example.

Invoke with:

TORCHFT_LIGHTHOUSE=http://localhost:29510 torchrun --master_port 29501 --nnodes 1 --nproc_per_node 1 train_ddp.py

train.py:

from torchft import Manager, DistributedDataParallel, Optimizer, ProcessGroupGloo

manager = Manager(
    pg=ProcessGroupGloo(),
    load_state_dict=...,
    state_dict=...,
)

m = nn.Linear(2, 3)
m = DistributedDataParallel(manager, m)
optimizer = Optimizer(manager, optim.AdamW(m.parameters()))

for i in range(1000):
    batch = torch.rand(2, 2, device=device)

    optimizer.zero_grad()

    out = m(batch)
    loss = out.sum()

    loss.backward()

    optimizer.step()

Running DDP

After starting the lighthouse server by running:

RUST_BACKTRACE=1 torchft_lighthouse --min_replicas 1 --quorum_tick_ms 100 --join_timeout_ms 10000

A test DDP script can be launched with torchX with:

torchx run

Or Diloco with:

USE_STREAMING=True torchx run ./torchft/torchx.py:hsdp --script='train_diloco.py'

See .torchxconfig, torchx.py and the torchX documentation to understand how DDP is being ran.

torchx.py could also launch HSDP jobs when workers_per_replica is set > 1, if the training script supports it. For an example HSDP training implementation with torchFT enabled, see torchtitan.

Alternatively, to test on a node with two GPUs, you can launch two replica groups running train_ddp.py by:

On shell 1 (one replica groups starts initial training):

export REPLICA_GROUP_ID=0
export NUM_REPLICA_GROUPS=2

CUDA_VISIBLE_DEVICES=0 TORCHFT_LIGHTHOUSE=http://localhost:29510 torchrun --master_port=29600 --nnodes=1 --nproc_per_node=1 -- train_ddp.py

On shell 2 (a second replica group joins):

export REPLICA_GROUP_ID=1
export NUM_REPLICA_GROUPS=2

CUDA_VISIBLE_DEVICES=1 TORCHFT_LIGHTHOUSE=http://localhost:29510 torchrun --master_port=29601 --nnodes=1 --nproc_per_node=1 -- train_ddp.py

By observing the outputs from both shells, you should observe process group reconfiguration and live checkpoint recovery.

Example Parameter Server

torchft has a fault tolerant parameter server implementation built on it's reconfigurable ProcessGroups. This does not require/use a Lighthouse server.

See parameter_server_test.py for an example.

Contributing

We welcome PRs! See the CONTRIBUTING file.

License

torchft is BSD 3-Clause licensed. See LICENSE for more details.

Metadata

Release files for torchft-nightly 2026.11.4

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for torchft-nightly 2026.11.4
File Actions
torchft_nightly-2026.11.4-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.15 CPython 3.15 Linux glibc 2.17+ x86-64 Details for torchft_nightly-2026.11.4-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
torchft_nightly-2026.11.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.14 CPython 3.14 Linux glibc 2.17+ x86-64 Details for torchft_nightly-2026.11.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
torchft_nightly-2026.11.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.17+ x86-64 Details for torchft_nightly-2026.11.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
torchft_nightly-2026.11.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.17+ x86-64 Details for torchft_nightly-2026.11.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
torchft_nightly-2026.11.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.17+ x86-64 Details for torchft_nightly-2026.11.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
torchft_nightly-2026.11.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl CPython 3.10 CPython 3.10 Linux glibc 2.17+ x86-64 Details for torchft_nightly-2026.11.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Total release size: 15.0 MB

Release files / torchft_nightly-2026.11.4-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL torchft_nightly-2026.11.4-cp315-cp315-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.5 MB
Tags CPython 3.15 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
ea7156b4b8765e1ab74b1e7c47138ca9d80fb0d7b4697bfeaba189f03d490719
BLAKE2b-256 checksum
How to use checksums
4f551e68bd754e5d8ab523dd084b6815bf27d6f84ff4b5038e02804f7d6868fe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / torchft_nightly-2026.11.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL torchft_nightly-2026.11.4-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.5 MB
Tags CPython 3.14 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
276b818245a365ae11ba993684cb08a1b4276c1fa5dcadb365a9421cd5b7ba83
BLAKE2b-256 checksum
How to use checksums
bde34201132d3e4ce8158a75c5ec70cd64c214558b35671e5534c58a19e4c8c0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / torchft_nightly-2026.11.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL torchft_nightly-2026.11.4-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.5 MB
Tags CPython 3.13 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
64f2ae82e988aeae589e4ff6b0cae42dfef815e2d8463d38b7c528bbb1d270e2
BLAKE2b-256 checksum
How to use checksums
0e7674f9e58ac3454d9e38798da544e284912dffb15d7784c1f6c3585774c534
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / torchft_nightly-2026.11.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL torchft_nightly-2026.11.4-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.5 MB
Tags CPython 3.12 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
41e34f555c07beb2b2451da1ccead6a7cb40eadbab828c8919bd05337fadbcf2
BLAKE2b-256 checksum
How to use checksums
a6cfcb26fac9edda486616e52340c776f33c42ff17b421979fa2271ce9eb931a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / torchft_nightly-2026.11.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL torchft_nightly-2026.11.4-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.5 MB
Tags CPython 3.11 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
f224f17f94eabafaca5bcf49519412f1d036679a2e6df0a15a74d2fa9e73bb67
BLAKE2b-256 checksum
How to use checksums
82692426f826cf03320e4e0e4aee809a452fcc22884d4e1eb25afdf978061657
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Release files / torchft_nightly-2026.11.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl

Download URL torchft_nightly-2026.11.4-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Size 2.5 MB
Tags CPython 3.10 Linux glibc 2.17+ x86-64
SHA-256 checksum
How to use checksums
36f0d93ab9f487cfd0d5c19704bfec9f6dca2e8b355ccabc9175aac0f8d54826
BLAKE2b-256 checksum
How to use checksums
02f2c6bb180d7fa0af8ea143410f3d38612c5b2868608edfc4f0204d4afc79b8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.10.22

Security for torchft-nightly 2026.11.4

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

2026.11.4 This release

6 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page