distensor 0.1.4

Decentralized distributed training with explicit 4D parallelism control

disTensor

Decentralized distributed training with explicit 4D parallelism control.

What is disTensor?

disTensor is a distributed training platform that lets you train large models across heterogeneous consumer compute. You define parallelism strategy explicitly — data, tensor, pipeline, and context parallel dimensions — and disTensor handles orchestration, synchronization, and fault tolerance while preserving familiar PyTorch semantics.

The system uses a driver-mediated architecture: your process owns the training loop and data pipeline, the hub coordinates topology, and workers execute computation and sync directly via P2P. The control plane (gRPC) handles orchestration; the data plane (P2P) handles tensor transfers on the critical path.

Installation

# Trainer SDK
pip install distensor

# Compute provider — macOS Apple Silicon
DISTENSOR_API_KEY=dt_p_... curl -fsSL https://api.distensor.com/install/macos.sh | bash

# Development (editable, from source)
make dev

Quick Start

import distensor as ds

# Local mode — no hub, no GPUs, simulated mesh
ds.init(mode="local")

mesh = ds.query_mesh()
print(f"Available workers: {mesh.available_workers}")

# Cloud mode — requires API key
ds.reset()
ds.init(mode="cloud", api_key="dt_u_...")

Architecture

disTensor uses a hub-and-spoke model where the hub coordinates topology but stays off the critical training path. Workers sync directly via P2P for low-latency tensor transfers. The cloud API layer adds persistence, auth, billing, and a web console on top.

Component	Package	Description
Hub	distensor.hub	Central control plane — node/job/topology registries, gRPC services (mesh, job, training, checkpoint, sync)
Cloud API	distensor.cloud	Production HTTP layer — FastAPI, PostgreSQL, Redis, Prometheus. Auth, billing, worker registry, telemetry
Node	distensor.node	Compute worker daemon — model shard execution, P2P data plane, CUDA/MPS/CPU backends
Trainer	distensor.training	Orchestration — Trainer.fit(), pipeline scheduling (GPipe, 1F1B), shard loading, execution plans
Sync	distensor.sync	4D parallel coordination — TP AllReduce, PP transfers, DP gradient sync, worker topology
Optimizer	distensor.optim	Distributed optimizer — ZeRO-3, gradient sync with bucketing, LR schedulers, state dict support
Console	ui/	React web UI for trainers, providers, and admins
Node App	distributions/	Provider surface — distensor-server supervisor + distensor TUI dashboard

See Architecture.md for diagrams, execution flows, and design principles. Full documentation at distensor.com/docs.

Parallelism

Mode	Abbreviation	Strategy
Data Parallel	DP	Replicate model across workers, split data batches
Tensor Parallel	TP	Shard individual layers across workers
Pipeline Parallel	PP	Split model into sequential stages across workers
Context Parallel	CP	Split sequence dimension across workers (coming soon)

Development

make dev                  # Full bootstrap (uv sync + proto generation)
make test                 # Run full test suite
make lint                 # Ruff linting
make format               # Auto-format

make cloud-up             # Start Postgres, Redis, MinIO, Prometheus
make cloud-api            # Run HTTP API server (:8000)
make cloud-hub            # Run gRPC hub server (:50051)

make console-dev          # Vite dev server (:5173, proxies /api to :8000)

make sandbox-up           # Docker sandbox (coordinator + 8 workers)
make sandbox-cloud-up     # Full cloud stack in Docker

make help                 # All available targets

Links

• Documentation
• Architecture
• Contributing
• License — Apache 2.0