peerai 0.2.0

PeerAI — P2P GPU sharing for AI inference. Contribute your GPU, earn revenue. Use spot inference at a fraction of cloud cost.

PeerAI

Peer-to-peer GPU sharing for AI inference — OpenAI-compatible, zero friction.

PeerAI turns idle GPUs into a shared AI inference network. Contributors earn credits by sharing compute; consumers get cheap inference through a standard OpenAI-compatible API. No code changes required — just point your existing OpenAI SDK at PeerAI.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="your-key")

response = client.chat.completions.create(
    model="llama3:8b",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)

# Streaming works too
for chunk in client.chat.completions.create(
    model="llama3:8b",
    messages=[{"role": "user", "content": "Tell me a joke"}],
    stream=True,
):
    print(chunk.choices[0].delta.content or "", end="")

How It Works

Your App (OpenAI SDK / curl / any HTTP client)
    → PeerAI Gateway (OpenAI-compatible API)
        → Smart Router (model, latency, load, reputation, price-aware)
            → Peer Node A (RTX 4090, Ollama)     ← direct P2P
            → Peer Node B (M3 Max, LM Studio)    ← WebSocket relay
            → Peer Node C (A100, vLLM)            ← gateway proxy

Contribute idle GPU time → earn credits. Consume inference → spend credits or pay market price.

Quick Start

1. Install

pip install -e ".[dev]"

2. Start the Gateway

peerai gateway --port 8000

The gateway starts with an in-memory SQLite backend by default. Visit http://localhost:8000/dashboard for the web UI.

3. Start a Node (contributor)

In a separate terminal, with Ollama running locally:

peerai serve --backend-url http://localhost:11434/v1 --gateway-url ws://localhost:8000/ws/node

The node agent auto-discovers your models and registers with the gateway.

4. Send Requests

# Chat completion
peerai chat llama3:8b "What is the meaning of life?"

# Streaming
peerai chat llama3:8b "Write a haiku" --stream

# Or use curl
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3:8b", "messages": [{"role": "user", "content": "Hello"}]}'

# List models
peerai models list

Features

Feature	Description
OpenAI-compatible API	Drop-in replacement — chat completions, completions, embeddings, model listing
Multi-node routing	7 strategies: round_robin, least_connections, least_latency, best_reputation, random, cheapest, best_value
Auto failover	Retries on up to 2 other nodes if one fails, with mid-stream resume
Credit metering	Token-based billing, contributor earnings (70% split), SQLite or PostgreSQL ledger
Authentication	API keys (SHA-256 hashed), node tokens, admin roles
Rate limiting	Global RPM limits + per-user per-model RPM/TPM/TPH/TPD budgets
Node reputation	Composite scoring: success rate, latency, uptime, disconnect history
GPU marketplace	Dynamic pricing, capacity listings, bidding, price-aware routing
Batch inference	Priority queuing, semaphore throttling, webhooks, pause/resume/retry
Model management	Auto-pull on demand, remote delete, pull progress tracking (Ollama)
Direct P2P	HMAC-signed tokens, connectivity probing, three modes: direct/relay/gateway
Multi-gateway federation	Redis pub/sub sync across gateway instances
Job checkpointing	Mid-stream failover with resume, preemption protocol, spot instance model
Model catalog	Rich metadata, load state tracking, per-model analytics, hot/idle discovery
Security hardening	Audit logging, login throttling, key rotation/expiration, security headers, CORS
Observability	Metrics endpoint, latency percentiles, per-node/model analytics
Web dashboard	Tabbed UI: Overview, Catalog, Marketplace, Nodes, Metrics — auto-refreshing
Python SDK	Sync and async clients mirroring OpenAI SDK structure
CLI	28 subcommands across 6 groups (models, batch, nodes, credits, market, chat)

Architecture

┌─────────────────────────────────────────────────┐
│                  PeerAI Gateway                  │
│                                                  │
│  ┌──────────┐ ┌────────┐ ┌──────────────────┐  │
│  │  Routes   │ │ Router │ │   Marketplace    │  │
│  │(OpenAI)  │ │(7 strats)│ │(pricing, bids)  │  │
│  └─────┬────┘ └────┬───┘ └────────┬─────────┘  │
│        │           │              │              │
│  ┌─────┴───────────┴──────────────┴──────────┐  │
│  │              Request Broker                │  │
│  │      (asyncio.Queue / Redis pub/sub)       │  │
│  └────────────────────┬───────────────────────┘  │
│                       │                          │
│  ┌────────────────────┴───────────────────────┐  │
│  │            Node Registry                   │  │
│  │   (reputation, connectivity, federation)   │  │
│  └────────────────────┬───────────────────────┘  │
└───────────────────────┼──────────────────────────┘
                        │ WebSocket (persistent)
        ┌───────────────┼───────────────┐
        ▼               ▼               ▼
   ┌─────────┐    ┌─────────┐    ┌─────────┐
   │  Node A  │    │  Node B  │    │  Node C  │
   │ (Ollama) │    │(LM Studio)│   │ (vLLM)  │
   └─────────┘    └─────────┘    └─────────┘

See docs/ARCHITECTURE.md for detailed design documentation.

Documentation

Document	Description
API Reference	Complete HTTP API documentation (42 endpoints)
Python SDK	PeerAI Python SDK — sync and async clients
CLI Reference	All CLI commands and options
Deployment Guide	Local dev, Docker, production setup
Architecture	System design, data flow, protocols
Project Plan	Roadmap and competitive analysis

Configuration

All settings are configurable via environment variables (PEERAI_ prefix) or a .env file:

Variable	Default	Description
PEERAI_DATABASE_URL	sqlite:///peerai.db	Database URL (SQLite or PostgreSQL)
PEERAI_REDIS_URL	""	Redis URL (enables Redis broker + federation)
PEERAI_PORT	8000	Gateway port
PEERAI_ROUTING_STRATEGY	least_connections	Routing strategy
PEERAI_ENABLE_CREDITS	true	Enable credit metering
PEERAI_ENABLE_AUTH	true	Enable API key authentication
PEERAI_ENABLE_RATE_LIMIT	true	Enable global rate limiting
PEERAI_ENABLE_BATCH	true	Enable batch inference
PEERAI_ENABLE_AUTO_PULL	false	Auto-pull models when no node has them
PEERAI_ENABLE_DIRECT_CONNECT	false	Enable direct P2P connections
PEERAI_ENABLE_FEDERATION	true	Enable multi-gateway federation
PEERAI_CORS_ORIGINS	""	Comma-separated allowed CORS origins
PEERAI_API_KEY_MAX_AGE_DAYS	0	API key expiration (0 = never)
PEERAI_LOGIN_THROTTLE_ENABLED	true	IP-based login throttling

See docs/DEPLOYMENT.md for the full configuration reference.

Development

# Install with dev deps
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Start gateway (dev mode, no auth)
peerai gateway --no-auth --no-credits

# Start node
peerai serve --backend-url http://localhost:11434/v1

Project Structure

src/peerai/
├── cli/             # CLI (Typer) — 28 subcommands
├── config.py        # Settings (pydantic-settings, PEERAI_* env vars)
├── gateway/         # FastAPI gateway
│   ├── app.py           # App factory
│   ├── routes.py        # OpenAI-compatible HTTP API
│   ├── ws_handler.py    # WebSocket node handler
│   ├── registry.py      # Node registry + reputation
│   ├── broker.py        # Request broker (asyncio.Queue)
│   ├── redis_broker.py  # Request broker (Redis pub/sub)
│   ├── federation.py    # Multi-gateway sync
│   ├── marketplace.py   # GPU marketplace engine
│   ├── model_catalog.py # Model metadata + analytics
│   ├── model_rate_limit.py  # Per-model rate limiting
│   ├── batch_store.py   # Batch job store
│   ├── batch_scheduler.py   # Background batch processor
│   ├── dispatch.py      # Shared inference dispatcher
│   ├── checkpoint.py    # Job checkpointing for failover
│   ├── pull_tracker.py  # Model pull job tracker
│   ├── direct_auth.py   # HMAC token signing for P2P
│   ├── connectivity.py  # Node reachability probing
│   ├── auth.py          # API key auth (SQLite)
│   ├── pg_auth.py       # API key auth (PostgreSQL)
│   ├── audit.py         # Security audit logging + login throttler
│   ├── security.py      # Security headers middleware + key expiration
│   └── static/          # Web dashboard
├── node/            # Node agent
│   ├── agent.py         # Single-gateway agent
│   ├── federated_agent.py   # Multi-gateway agent
│   ├── model_ops.py     # Model pull/delete mixin
│   └── direct_server.py # Direct P2P HTTP server
├── relay/           # Relay service for NAT traversal
│   ├── server.py        # Relay HTTP/WS server
│   └── tunnel.py        # Node-side tunnel client
├── router/          # Request routing
│   └── selector.py      # 7 routing strategies + reputation
├── models/          # Pydantic models
│   ├── api.py           # OpenAI-compatible schemas
│   ├── node.py          # Node models
│   ├── protocol.py      # WebSocket message protocol
│   ├── marketplace.py   # Marketplace models
│   ├── batch.py         # Batch job models
│   ├── pull.py          # Model pull/delete models
│   └── direct_connect.py    # P2P connection models
├── sdk/             # Python SDK
│   ├── __init__.py      # PeerAI, AsyncPeerAI exports
│   └── client.py        # Sync + async clients
└── db/              # Database layer
    ├── ledger.py        # Credit ledger (SQLite)
    └── pg_ledger.py     # Credit ledger (PostgreSQL)

License

TBD