RoundPipe 0.1.1

High Performance · Easy to Use · Built for Gaming GPUs

Documentation · 中文文档 · Benchmarks · Examples

RoundPipe is a large DNN training framework that lets you train huge models on consumer-grade GPUs. On a single 24 GB GPU, you can full fine-tune 32B-parameter models, LoRA fine-tune up to 235B, and handle 64K+ token sequences, with throughput approaching datacenter-class hardware.

Highlights

• Train bigger than ever: Full fine-tune 32B models or LoRA fine-tune up to 235B on a single 24 GB GPU. Up to 7× longer sequence length than PyTorch FSDP.
• High performance: Push a 4090 close to A800 NVLINK-class throughput. Up to 6× faster than FSDP Offload in typical workloads.
• Linear multi-GPU scaling: Scale to multiple GPUs within a node without rewriting your training loop. Throughput grows linearly while max sequence length per GPU stays unchanged.
• Feels like PyTorch: Sequential programming interface with a low learning curve. Works well in Jupyter Notebook for rapid iteration.
• General by design: No constraints on layer structure, training flow, or parameter update strategy.
• Portable across accelerators: Pure PyTorch implementation. Runs on Nvidia, AMD, and Ascend platforms.

Benchmarks

All benchmarks below are measured on a single node with 8 GPUs. "OOM" means the framework cannot fit the model under that configuration.

Maximum Input Sequence Length

Training Throughput (tokens/s)

Multi-GPU Scaling (8× RTX 4090)

Max sequence length per GPU stays constant across all GPU counts (73 K, 49 K, 28 K, and 31 K respectively).

Cross-Platform

Quick Start

Installation

pip install roundpipe

Requirements: Python ≥ 3.8, PyTorch ≥ 2.4

Examples

See the example/ directory. More examples and tutorials will be added soon.

Documentation

Full documentation is available at itcarrot.github.io/RoundPipe.

中文文档请访问 itcarrot.github.io/RoundPipe/index.zh.html。

License

RoundPipe is licensed under the LGPL-3.0.