yaarbal 0.1.7

BitTorrent-inspired peer-to-peer distributed LLM inference

Yaarbal

Yaarbal is a BitTorrent-inspired peer-to-peer system for running LLM inference across distributed nodes. Each peer hosts a complete model; a tracker brokers peer discovery; clients route inference to available seeders. Hosting earns credits, querying spends them — no central GPU pool, just peers.

Status: v2.1 milestone in progress. Core whole-model P2P inference, credit economy, browser chat UI, and the yaarbal CLI are working. Treat the public reference network as demo-grade — production hardening (TLS, automatic failover, DHT) is on the roadmap.

Visit yaarbal.app for the public reference network.

---

Installation

pip install yaarbal
yaarbal --version

Requires Python 3.11 or newer. The wheel pulls in PyTorch, Transformers, Flask, and the rest of the runtime stack — no manual model setup needed.

For a source/development install:

git clone https://github.com/hannanabdul55/distributed-full-gpt-inference.git
cd distributed-full-gpt-inference
python -m venv venv && source venv/bin/activate
pip install -e ".[dev]"

See docs/distribution/ for Docker and other distribution paths.

---

Quick Start

The fastest path is single-node mode — one machine, one model, no tracker. Best for trying the system or running a local assistant.

1. Start a seeder

yaarbal seed --model qwen2.5-0.5b-instruct --port 7001

Loads the full model weights and serves inference on port 7001.

2. Chat with it

In a second terminal:

YAARBAL_CHAT_BACKEND=single \
YAARBAL_SINGLE_NODE_URL=http://127.0.0.1:7001 \
yaarbal chat --model qwen2.5-0.5b-instruct

Type messages and press Enter. Use /health, /metrics, /models, /reset, or /quit inside the REPL.

For the full quickstart — including the distributed swarm path, Docker profiles, and the browser UI — see QUICKSTART.md.

---

How It Works

   prompt
     │
     ▼
  ┌─────────┐         ┌────────────┐
  │ Client  │ ──────▶ │  Tracker   │   discover seeders for model X
  │ (CLI /  │ ◀────── │            │
  │ web UI) │         └────────────┘
  └────┬────┘                │
       │ POST /inference     ▼
       ▼               ┌──────────┐
  ┌─────────┐  ◀────── │  Seeder  │   hosts complete model X
  │  text   │          │  (full   │   earns credits per request
  │ output  │          │   model) │
  └─────────┘          └──────────┘

• Tracker — peer discovery, swarm management, credit settlement.
• Seeder — loads the full model weights and serves POST /inference. One seeder = one model on one host.
• Client / leecher — discovers a seeder via the tracker, routes the prompt, returns text.

The whole-model design (post-2026-04-12 pivot) means each peer hosts an entire model rather than a layer-shard. Layer-split inference was removed in Phases 51–52; whole-model is the only paradigm.

For the protocol design, credit economy, and trust tiers, see PROJECT_OVERVIEW.md and docs/architecture/.

---

Common Commands

# CLI
yaarbal seed --model <name> --port 7001        # host a model
yaarbal run <model> "<prompt>"                  # one-shot inference via the swarm
yaarbal chat --model <name>                     # interactive REPL
yaarbal tracker 8000                            # host a tracker
yaarbal ui                                      # browser chat UI on localhost:5173
yaarbal status                                  # credits, hosted models, peers
yaarbal --list-models                           # show all 30+ supported models

# Docker
docker compose --profile single-node up single-node    # tracker + one seeder
docker compose up                                       # tracker only
docker compose --profile query run --rm leecher         # one-shot inference

# Tests
make test                                       # full suite
make test-unit                                  # unit tests only
make test-integration                           # integration tests
python -m pytest tests/unit/test_tracker.py -v  # single file

# Memory / benchmarks
make verify-memory

Environment variables use the canonical YAARBAL_* prefix. Legacy DGPT_* aliases still work in v2.1 but emit a one-time deprecation warning.

---

Supported Models

30+ models are registered out of the box, including:

Family	Variants	Sizes
GPT-2	5 variants	82M – 1.5B
LLaMA	3.2-1B, 3.2-3B, 3.1-8B, 3.3-70B	1B – 70B
Mistral / Mixtral	7B, 8x7B (MoE)	7B – 46.7B
Phi	3-Mini (4K/128K), 3-Medium	3.8B – 14B
Qwen 2.5	7 sizes	0.5B – 72B
DeepSeek	V3 (671B MoE), R1-Distill-Qwen-1.5B	1.5B – 671B
MiniGPT	tiny test model	<100M

LLaMA 8B and larger require a Hugging Face access token (HF_TOKEN).

See MODELS.md for the full catalog with hardware requirements and per-model notes.

---

Browser UI

yaarbal ui

Opens the chat UI at http://127.0.0.1:5173. Paste either a seeder URL or a tracker URL into the Server URL field — the SettingsDialog auto-detects the server type via GET /health. Configure model, backend, and connection in the Connection panel.

The UI is a Vite + React + TypeScript app under web/. Run npm install && npm run dev from web/ for local development.

---

Configuration

Configure Yaarbal via environment variables (canonical YAARBAL_* prefix). See .env.example and config.py for the complete list. Common ones:

Variable	Purpose
YAARBAL_MODEL	Default model name
YAARBAL_TRACKER_URL	Tracker URL (default: https://tracker.yaarbal.app)
YAARBAL_CHAT_BACKEND	single (direct seeder) or tracker
YAARBAL_SINGLE_NODE_URL	Seeder URL when CHAT_BACKEND=single
YAARBAL_AUTH_USER / YAARBAL_AUTH_PASS	Optional HTTP Basic Auth
HF_TOKEN / HUGGINGFACE_HUB_TOKEN	Required for gated models (LLaMA 8B+)

Full reference: docs/configuration/overview.md.

---

Documentation

Doc	What it covers
QUICKSTART.md	Two-minute guide: single-node, swarm, Docker, browser UI
PROJECT_OVERVIEW.md	Engineering deep-dive: protocol, credit economy, code layout
MODELS.md	Catalog of 30+ models with hardware requirements
KNOWN_LIMITATIONS.md	Open issues (P0–P3), security caveats, roadmap items
CHANGELOG.md	Version history
docs/architecture/	Protocol design, BitTorrent mechanisms, topology
docs/api/	HTTP API reference (tracker + seeder)
docs/configuration/	Environment variables and tuning
docs/deployment/	Production deployment guides
docs/guides/	Getting started, development, operations
docs/research/	Academic paper draft and related work

---

Known Limitations

• No TLS — tensor data is plaintext over HTTP unless fronted by a TLS-terminating proxy.
• HTTP Basic Auth only — optional, off by default; not a substitute for TLS.
• Central tracker is a single point of failure — DHT is on the roadmap, not yet implemented.
• Sequential per-request latency — single-request latency does not benefit from peer parallelism (only throughput does).
• Gated models require HF auth — LLaMA 8B+ needs HF_TOKEN.

Full list with severity and tracking: KNOWN_LIMITATIONS.md.

---

License

Apache License 2.0. See LICENSE.