The d9d Project
d9d is a distributed training framework built on top of PyTorch 2.0. It aims to be hackable, modular, and efficient, designed to scale from single-GPU debugging to massive clusters running 6D-Parallelism.
Installation
Just use your favourite package manager:
pip install d9d
poetry add d9d
uv add d9d
Extras
d9d[aim]: Aim experiment tracker integration.d9d[visualization]: Plotting libraries required to some advanced visualization functionality.d9d[linear-attention]: Efficient Linear Attention kernels.d9d[backend-sdpa-flash-attention-2]: FlashAttention 2 SDPA backend kernels.d9d[backend-sdpa-flash-attention-4]: FlashAttention 4 SDPA backend kernels.d9d[moe]: Efficient Mixture of Experts GPU kernels. You should build and install some dependencies manually before installation: DeepEP, grouped-gemm.d9d[cce]: Efficient Fused Cross-Entropy kernels. You should build and install some dependencies manually before installation: Cut Cross Entropy.
Examples
- Qwen3-MoE Pretraining: an example showing causal LM pretraining for the Qwen3-MoE model.
About
Why another framework?
Distributed training frameworks such as Megatron-LM are monolithic in the way you run a script from the command line to train any of a set of predefined models, using predefined regimes. While powerful, these systems can be difficult to hack and integrate into novel research workflows. Their focus is often on providing a complete, end-to-end solution, which can limit flexibility for experimentally-driven research.
Conversely, creating your own distributed training solution from scratch is tricky. You have to implement many low-level components (like distributed checkpoints and synchronization) that are identical across setups, and manually tackle common performance bottlenecks.
d9d was designed to fill the gap between monolithic frameworks and homebrew setups, providing a modular yet effective solution for distributed training.
What d9d is and isn't
In terms of core concept:
- IS a pluggable framework for implementing distributed training regimes for your deep learning models.
- IS built on clear interfaces and building blocks that may be composed and implemented in your own way.
- IS NOT an all-in-one CLI platform for setting up pre-training and post-training like torchtitan, Megatron-LM, or torchforge.
In terms of codebase & engineering:
- IS built on a strong engineering foundation: We enforce strict type-checking and rigorous linting to catch errors before execution.
- IS reliable: The framework is backed by a suite of over 450 tests, covering unit logic, integration flows, and End-to-End distributed scenarios.
- IS eager to use performance hacks (like DeepEp or custom kernels) if they improve MFU, even if they aren't PyTorch-native.
- IS NOT for legacy setups: We do not maintain backward compatibility with older PyTorch versions or hardware. We prioritize simplicity and modern APIs (like
DTensor).
Key Philosophies
To achieve the balance between hackability and performance, d9d adheres to specific design principles:
- Composition over Monoliths: We avoid "God Classes" like
DistributedDataParallelorParallelDimsthat assume ownership of the entire execution loop. Instead, we provide composable and extendable APIs. For instance, specific horizontal parallelism strategies for specific layers (parallelize_replicate,parallelize_expert_parallel, ...). - White-Box Modelling: We encourage standard PyTorch code. Models are not wrapped in obscure metadata specifications; they are standard
nn.Modules that implement lightweight protocols. - Pragmatic Efficiency: While we prefer native PyTorch, we are eager to integrate non-native solutions if they improve MFU. For example, we implement MoE using DeepEp communications, reindexing kernels from Megatron-LM, and efficient grouped-GEMM implementations.
- Graph-Based State Management: Our IO system treats model checkpoints as directed acyclic graphs. This allows you to transform architectures (e.g., merging
q,k,vintoqkv) on-the-fly while streaming from disk, without massive memory overhead. - DTensors: We mandate that distributed parameters be represented as
torch.distributed.tensor.DTensor. This simplifies checkpointing by making them topology-aware automatically. We leverage modern PyTorch 2.0 APIs (DeviceMesh) as much as possible.
Community & Support
If you need help, want to discuss large-scale training strategies, or want to contribute to d9d, come join our community!
- 💬 Discord: Join the d9d server for real-time chat, support, and framework hacking.
- 🐛 Issues: For bug reports and feature requests, please use the GitHub Issue Tracker.
1 maintainer
Metadata
1 maintainer
Release files for d9d 0.20.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| d9d-0.20.0.tar.gz | 207.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| d9d-0.20.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 540.0 kB
Release files / d9d-0.20.0.tar.gz
| Download URL | d9d-0.20.0.tar.gz |
|---|---|
| Size | 207.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e3d9e2c38cfe6bcc27b2f935e15fb9fdce0d23e52e1bc5283d0594f3172831f6
|
|
BLAKE2b-256 checksum How to use checksums |
16603c6483a17ac9b8f2944e178838a511f201af970521d491696c9e4dcf89e2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency logRelease files / d9d-0.20.0-py3-none-any.whl
| Download URL | d9d-0.20.0-py3-none-any.whl |
|---|---|
| Size | 333.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
fbb11f506dee91a59768f54dc5b7ad43f1a2e30d4a37d519242f5e5ed31031ea
|
|
BLAKE2b-256 checksum How to use checksums |
be265e488915ca871850c33e5801f9b9f544d946f6395272dc1b1ddbb2b5ca0b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 2, 2026.
Transparency log
