llm-relay 0.7.0

Unified LLM usage management — API proxy, session diagnostics, multi-CLI orchestration.

llm-relay

Unified LLM usage management — API proxy, session diagnostics, multi-CLI orchestration.

한국어 | llms.txt

Why

This project started from a need to escape deep vendor lock-in with a single AI coding tool. After investigating hidden behaviors in Claude Code — silent token inflation, false rate limits, context stripping, and opaque feature flags — it became clear that relying on one vendor's black box was a risk. llm-relay was built to take back visibility and control: monitor what's actually happening, diagnose problems independently, and orchestrate across multiple CLI tools (Claude Code, Codex, Gemini) so no single provider becomes a single point of failure.

Features

• Proxy: Transparent API proxy with cache/token monitoring and 12-strategy pruning
• Detect: 7 detectors (orphan, stuck, synthetic, bloat, cache, resume, microcompact)
• Recover: Session recovery and doctor (7 health checks)
• Guard: 4-tier threshold daemon with dual-zone classification
• Cost: Per-1% cost calculation and rate-limit header analysis
• Orch: Multi-CLI orchestration (Claude Code, Codex CLI, Gemini CLI)
• Display: Multi-CLI session monitor with context composition pie chart, connection type badges (SSH/tmux/tailscale/mosh), and provider liveness detection
• History: Proxy-level conversation capture with delta/full storage, compaction detection, and web replay viewer
• Composition: Real-time context window analysis — classifies content into 6 categories (user/assistant/tool_use/tool_result/thinking/system) with SNR metrics and duplicate read tracking
• TUI: llm-relay top — btop-style terminal monitor with Rich Live (works over SSH, no browser needed)
• i18n: Browser locale detection with en/ko support; server-side override via LLM_RELAY_LANG
• MCP: 8 tools via stdio transport (cli_delegate, cli_status, cli_probe, orch_delegate, orch_history, relay_stats, session_turns, session_history)

Install

# CLI only (diagnostics, recovery, orchestration)
pip install llm-relay

# With Rich TUI (llm-relay top)
pip install llm-relay[cli]

# With proxy + web dashboard
pip install llm-relay[proxy]

# With MCP server (Python 3.10+)
pip install llm-relay[mcp]

# Everything
pip install llm-relay[all]

Quick Start

CLI diagnostics (no server needed)

llm-relay scan              # Session health check (7 detectors)
llm-relay doctor            # Configuration health check (7 checks)
llm-relay recover           # Extract session context for resumption
llm-relay top               # Live terminal monitor (btop-style TUI)

Web dashboard

# Option 1: Direct
pip install llm-relay[proxy]
uvicorn llm_relay.proxy.proxy:app --host 0.0.0.0 --port 8083

# Option 2: Docker
cp .env.example .env        # Edit as needed
docker compose up -d

Then open:

• /dashboard/ — CLI status, cost, delegation history, Turn Monitor (alive sessions only; ?include_dead=1 to bypass)
• /display/ — Turn counter with context composition pie chart, connection type badges, CC/Codex/Gemini session cards
• /history/ — Session conversation replay with compaction timeline

MCP server

llm-relay-mcp               # stdio transport, 8 tools

API proxy for Claude Code

# Set in Claude Code
export ANTHROPIC_BASE_URL=http://localhost:8080

CLI Status

CLI	Status
Claude Code	Fully supported
OpenAI Codex	Fully supported
Gemini CLI	Display supported, oauth-personal has known 403 server-side bug (#25425)

Requirements

• Python >= 3.9
• MCP tools require Python >= 3.10

License

MIT

Ecosystem

Part of the QuartzUnit open-source ecosystem.