The d9d Project
d9d is a distributed training framework built on top of modern PyTorch. It aims to be hackable, modular, and efficient, designed to scale from single-GPU debugging to massive clusters running 6D parallelism.
Installation
Install it with your package manager:
pip install d9d
poetry add d9d
uv add d9d
Extras
d9d[aim]: Aim experiment tracker integration.d9d[visualization]: Plotly, for plotting learning rate schedules.d9d[linear-attention]: Flash Linear Attention kernels for linear attention.d9d[backend-sdpa-flash-attention-2]: FlashAttention 2 SDPA backend kernels. You may need to build FlashAttention 2 from source if no prebuilt wheel matches your setup.d9d[backend-sdpa-flash-attention-4]: FlashAttention 4 SDPA backend kernels.d9d[moe]: Mixture-of-Experts (MoE) GPU kernels. You must build and install DeepEP and grouped-gemm manually first.d9d[cce]: Fused cross-entropy kernels. You must install Cut Cross Entropy from its repository first.
Examples
- Qwen3-MoE Pretraining: Causal LM pretraining of a Qwen3-MoE model.
About
Why Another Framework?
Distributed training frameworks such as Megatron-LM are monolithic: you run a script from the command line to train one of a set of predefined models, using predefined regimes. These systems are hard to hack and to integrate into new research workflows. They aim to be a complete end-to-end solution, which limits flexibility for experiment-driven research.
Writing your own distributed training solution from scratch is also hard. You must implement many low-level components that are the same across setups, such as distributed checkpoints and synchronization. You must also fix common performance bottlenecks yourself.
d9d fills the gap between monolithic frameworks and homebrew setups. It gives you modular building blocks for distributed training.
What d9d Is and Isn't
In terms of core concept:
- IS a pluggable framework for implementing distributed training regimes for your deep learning models.
- IS built on clear interfaces and building blocks that may be composed and implemented in your own way.
- IS NOT an all-in-one CLI platform for setting up pre-training and post-training like torchtitan, Megatron-LM, or torchforge.
In terms of codebase and engineering:
- IS built on a strong engineering foundation: we enforce strict type checking and linting to catch errors before execution.
- IS reliable: a suite of over 900 tests covers unit logic, integration flows, and end-to-end distributed scenarios.
- IS eager to use performance hacks (like DeepEP or custom kernels) if they improve MFU, even if they aren't PyTorch-native.
- IS NOT for legacy setups: we do not maintain backward compatibility with older PyTorch versions or hardware. We prefer simplicity and modern APIs (like
DTensor).
Key Philosophies
To balance hackability and performance, d9d follows these design principles:
- Composition over Monoliths: We avoid "God Classes" like
DistributedDataParallelorParallelDimsthat own the entire execution loop. Instead, we provide composable and extendable APIs, such as per-layer horizontal parallelism strategies (parallelize_replicate,parallelize_expert_parallel, ...). - White-Box Modeling: We encourage standard PyTorch code. Models are not wrapped in metadata specifications. They are standard
nn.Modules that implement small protocols. - Pragmatic Efficiency: We prefer native PyTorch, but we integrate non-native solutions if they improve MFU. For example, our MoE uses DeepEP communications, reindexing kernels from Megatron-LM, and grouped GEMM kernels.
- Graph-Based State Management: Our I/O system treats model checkpoints as directed acyclic graphs. You can transform architectures (e.g. merge
q,k,vintoqkv) on the fly while streaming from disk, without loading the whole checkpoint into memory. - DTensors: Distributed parameters must be
torch.distributed.tensor.DTensors. DTensors know their topology, which makes checkpointing simpler. We use modern PyTorch APIs (DeviceMesh) wherever we can.
Community and Support
Join our community if you need help, want to discuss large-scale training strategies, or want to contribute to d9d.
- 💬 Discord: Join the d9d server for real-time chat, support, and framework hacking.
- 🐛 Issues: For bug reports and feature requests, please use the GitHub Issue Tracker.
1 maintainer
Metadata
1 maintainer
Release files for d9d 0.22.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| d9d-0.22.0.tar.gz | 201.1 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| d9d-0.22.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 530.9 kB
Release files / d9d-0.22.0.tar.gz
| Download URL | d9d-0.22.0.tar.gz |
|---|---|
| Size | 201.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d78966c2008c74457ce94965d631e44ee280439865dd19f69dc1812dbf58c0bc
|
|
BLAKE2b-256 checksum How to use checksums |
7ab75c1cf7a59c1a66e8631ed1cabcde0d256054eae6ceefaba793d57920872a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 4, 2026.
Transparency logRelease files / d9d-0.22.0-py3-none-any.whl
| Download URL | d9d-0.22.0-py3-none-any.whl |
|---|---|
| Size | 329.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
255bcefbd106a34bbead63fef07f9bb802c494b8636527cf9f858fc961cca051
|
|
BLAKE2b-256 checksum How to use checksums |
85339f358435ec86ae061315375d8a10dc740789acd8ff33dd922734c106bd86
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 4, 2026.
Transparency log
