distensor 0.5.0

Decentralized distributed training with explicit 4D parallelism control

Decentralized distributed training with explicit 4D parallelism control.

disTensor trains large models across heterogeneous consumer compute. You declare the parallelism strategy — data, tensor, pipeline, and (soon) context — and disTensor handles orchestration, synchronization, and fault tolerance while keeping familiar PyTorch semantics.

Status: alpha. APIs and wire formats may change between minor releases.

Why disTensor

• Explicit 4D parallelism. No auto-magic sharding — you define StageMapping and ShardingSpec, the system executes them predictably.
• Driver-mediated execution. Your process owns the training loop and data pipeline. The hub coordinates topology but stays off the critical path.
• Control/data plane split. gRPC for orchestration (:50051), direct P2P for tensor transfers. Workers sync without round-tripping through the hub.
• Heterogeneous compute. CUDA, Apple Silicon (MPS/MLX), and CPU backends with auto-detected capabilities.
• Cloud or standalone. Run a local hub for experiments, or point at the managed cloud for persistence, auth, billing, and a web console.

Installation

# Trainer SDK (PyPI)
pip install distensor

# Compute provider — macOS Apple Silicon (one-liner installer)
DISTENSOR_API_KEY=dt_p_... curl -fsSL https://api.distensor.com/install/macos.sh | bash

# Provider node app, install manually
pip install distensor-server distensor-tui

# Development (editable, from source)
git clone https://github.com/don-arash/distensor
cd distensor && make dev

Role-specific extras: distensor[worker], distensor[worker-cuda], distensor[worker-mlx], distensor[cloud], distensor[dev].

Quick Start

import distensor as ds
from distensor.core import ParallelConfig
from distensor.nn import DistributedModel, StageMapping, ShardingSpec, Shard, Replicate

ds.init(mode="cloud", api_key="dt_u_...")

# Inspect available compute before committing
mesh = ds.query_mesh()
print(f"Workers online: {mesh.available_workers}")

# Declare parallelism explicitly
parallel = ParallelConfig(data_parallel=4, tensor_parallel=2, pipeline_parallel=2)

stages = StageMapping([
    ["model.embed"],
    ["model.layers.[0-5]"],
    ["model.layers.[6-11]"],
    ["model.norm", "lm_head"],
])

tp_spec = ShardingSpec({
    "model.layers.*.self_attn.q_proj.weight": Shard(dim=0),
    "model.layers.*.self_attn.o_proj.weight": Shard(dim=1),
    "model.layers.*.input_layernorm.weight": Replicate(),
})

model = DistributedModel(my_module, stage_mapping=stages, tp_spec=tp_spec)
trainer = ds.Trainer(model, parallel=parallel)
trainer.fit(data_loader, epochs=1)

See the docs for full training examples, mesh planning, and the cloud API reference.

Architecture

disTensor uses a hub-and-spoke topology. The hub coordinates, workers compute and sync directly, and the cloud layer adds persistence and a web surface on top.

Component	Package	Description
Hub	distensor.hub	Control plane — node / job / topology registries, gRPC services (mesh, job, training, checkpoint, sync)
Cloud API	distensor.cloud	Production HTTP layer — FastAPI + Postgres + Redis + Prometheus. Auth, billing, worker registry, telemetry
Node	distensor.node	Compute worker daemon — model shard execution, P2P data plane, CUDA / MPS / CPU backends
Trainer	distensor.training	Orchestration — Trainer.fit(), pipeline scheduling (GPipe, 1F1B), shard loading, execution plans
Sync	distensor.sync	4D parallel coordination — TP AllReduce, PP transfers, DP gradient sync, worker topology
Optimizer	distensor.optim	Distributed optimizer — ZeRO-3, gradient sync with bucketing, LR schedulers, distributed state dict
Console	ui/	React web UI for trainers, providers, and admins
Node App	distributions/	Provider surface — distensor-server supervisor + distensor TUI dashboard

See Architecture.md for diagrams, execution flows, and design principles.

Parallelism

Mode		Strategy
Data Parallel	DP	Replicate model across workers, split data batches
Tensor Parallel	TP	Shard individual layers across workers
Pipeline Parallel	PP	Split model into sequential stages across workers
Context Parallel	CP	Split sequence dimension across workers (coming soon)

Development

make dev                  # Full bootstrap (uv sync + proto generation)
make test                 # Run full test suite
make lint                 # Ruff linting
make format               # Auto-format

make cloud-up             # Start Postgres, Redis, MinIO, Prometheus
make cloud-api            # Run HTTP API server (:8000)
make cloud-hub            # Run gRPC hub server (:50051)

make console-dev          # Vite dev server (:5173, proxies /api to :8000)

make sandbox-up           # Docker sandbox (coordinator + 8 workers)
make sandbox-cloud-up     # Full cloud stack in Docker

make help                 # All available targets

Contributions welcome — see CONTRIBUTING.md. Commit messages follow Conventional Commits; releases are cut by commitizen and published to PyPI via GitHub Actions Trusted Publishing on v* tags.

Community & Support

• Documentation: distensor.com/docs
• Issues & bugs: GitHub Issues
• Discussions: GitHub Discussions
• Changelog: CHANGELOG.md

License

Apache 2.0 — see LICENSE.