LLM Orchestra
Orchestrate ensembles of specialized models — local and cloud — to do real analytical work. Coordination over scale.
A decent laptop can run multiple small language models simultaneously. What's missing is the coordination layer — the system that decomposes problems, routes them to specialized agents, manages dependencies between them, and synthesizes results. LLM Orchestra provides that layer: independent agents run in parallel, dependent agents wait for what they need, and script agents handle deterministic work alongside LLM agents. Mix expensive cloud models with free local models, and put cost only where it matters.
What it does
- Multi-agent ensembles — DAGs of LLM and script agents with dependencies, fan-out, guards, bounded loops, and dynamic dispatch
- Agentic serving —
llm-orc serveexposes OpenAI-compatible endpoints so coding tools (OpenCode, Aider, Cline) can use composed ensembles as their model backend, with build deliverables verified by an accept gate before they ship - Model profiles — named model+provider shortcuts; back any tier with your own provider via untracked local overrides
- Hybrid local/cloud — local GGUF models through a llama-server router that llm-orc owns, plus Claude, Gemini, and any OpenAI-compatible server (vLLM, LM Studio, OpenRouter), with cost and usage tracking
- Scripts and artifacts — script agents with JSON I/O, timestamped execution artifacts, an ensemble library, and an MCP server
Install
# Homebrew (macOS)
brew tap mrilikecoding/llm-orchestra && brew install llm-orchestra
# pip (all platforms)
pip install llm-orchestra
llm-orc --version
Quick start
# Configure providers (keys are encrypted at rest); skip for local-only use
llm-orc auth setup
# Initialize project config (.llm-orc/ with ensembles/, scripts/, config.yaml)
llm-orc config init
# Run an ensemble
llm-orc list-ensembles
cat code.py | llm-orc invoke code-review
An ensemble is a YAML file: agents, their models, and who depends on whom.
name: code-review
agents:
- name: security-reviewer
model_profile: free-local
system_prompt: "You are a security analyst. Identify vulnerabilities."
- name: senior-reviewer
model_profile: default-claude
depends_on: [security-reviewer]
system_prompt: "Synthesize the analysis into actionable recommendations."
The full command and configuration reference — output formats, config management, script agents, the ensemble library, the MCP server, profile and fallback schemas — lives in docs/cli-reference.md.
Agentic serving
Point an agentic coding tool at llm-orc and use composed ensembles as its model backend:
llm-orc serve # OpenAI-compatible /v1/models + /v1/chat/completions
// e.g. ~/.config/opencode/opencode.json
{
"provider": {
"llm-orc": {
"npm": "@ai-sdk/openai-compatible",
"options": { "baseURL": "http://localhost:8765/v1" },
"models": { "agentic": {} }
}
}
}
Every turn runs one declarative serving ensemble: classify the request, route it to a capability seat, verify the deliverable (build turns run a test-writer → code-writer → sandboxed executor → adequacy judge pipeline with a bounded retry round), and emit it through the client's own tool surface. Conversation context threads from the client-sent history; the builder never grades itself. Runs entirely on local models by default.
Architecture and operator guide: docs/serving.md · Staged path to full model parity: docs/serving-roadmap.md
Documentation
| Doc | What's in it |
|---|---|
| docs/cli-reference.md | Complete CLI and configuration reference |
| docs/serving.md | Agentic serving architecture and operator guide |
| docs/serving-roadmap.md | Staged roadmap toward full model parity |
| docs/architecture.md | System architecture and design principles |
| docs/adrs/ | Architecture decision records (serving ADRs in docs/adrs/serving/) |
| docs/ensemble_vs_single_agent_analysis.md | Multi-agent vs single-agent research findings |
Philosophy
Coordination over scale. Process over generation.
Generation was never the bottleneck — evaluation is. An ensemble of smaller, specialized models, each owning a bounded task, produces structured output designed for human evaluation: a security reviewer finds vulnerabilities, a synthesis agent integrates findings, and the human evaluates a structured analysis rather than raw generation. The research evidence supports the bet: orchestrated ensembles of open-source models have matched or exceeded frontier model performance on established benchmarks, and cascade routing replicates frontier quality at a fraction of the cost.
Running models locally is a practical choice, not an ideological one: no per-query billing, data stays on your hardware, and the local/cloud mix puts spend where it matters.
Development
uv sync --dev
make test # pytest with coverage gate
make lint # mypy, ruff, complexity, security checks
License
AGPL-3.0-or-later — see LICENSE.
1 maintainer
Metadata
1 maintainer
Release files for llm-orchestra 0.20.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| llm_orchestra-0.20.6.tar.gz | 3.3 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| llm_orchestra-0.20.6-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.7 MB
Release files / llm_orchestra-0.20.6.tar.gz
| Download URL | llm_orchestra-0.20.6.tar.gz |
|---|---|
| Size | 3.3 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4cc550f03c22cf7b1f6d6a0a3f99366ef6d5d624fb06fc4cbbf61c918c0f75bf
|
|
BLAKE2b-256 checksum How to use checksums |
c113411fa6d4517333093491b787118f07468fb4bf3b80fd777d5541532d1e10
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / llm_orchestra-0.20.6-py3-none-any.whl
| Download URL | llm_orchestra-0.20.6-py3-none-any.whl |
|---|---|
| Size | 451.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
382ee9b128ab3a78239127a113101a8f63f35cc36fab29bf9b0bd2f1565142a9
|
|
BLAKE2b-256 checksum How to use checksums |
b0745e0e61fe522b25692c70baf05163a564e9571a15d9af44b78540f7d5f404
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Security for llm-orchestra 0.20.6
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
