openadapt-consilium 0.4.0

Multi-LLM council for consensus-driven AI responses

Consilium

Multi-LLM council for consensus-driven AI responses.

Consilium queries multiple LLMs in parallel, has each model review the others' responses, then synthesizes the best answer through a chairman model. Inspired by Karpathy's llm-council.

How It Works

┌─────────────────────────────────────────────────────────┐
│                    STAGE 1: QUERY                       │
│                                                         │
│   ┌─────────┐   ┌──────────┐   ┌─────────────────┐     │
│   │ GPT-5.2 │   │ Claude   │   │ Gemini 3.1 Pro  │     │
│   │         │   │ Sonnet   │   │                 │     │
│   └────┬────┘   └────┬─────┘   └────────┬────────┘     │
│        │             │                  │               │
│        ▼             ▼                  ▼               │
│   Response A    Response B         Response C           │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│                  STAGE 2: REVIEW                        │
│                                                         │
│   Each model reviews all anonymized responses.          │
│   Ranks them best → worst with reasoning.               │
└─────────────────────┬───────────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────────┐
│                STAGE 3: SYNTHESIS                       │
│                                                         │
│   Chairman model synthesizes the best answer from       │
│   all responses + reviews.                              │
│                                                         │
│              ┌──────────────────┐                       │
│              │  Final Answer    │                       │
│              └──────────────────┘                       │
└─────────────────────────────────────────────────────────┘

Installation

pip install openadapt-consilium

Or for development:

git clone https://github.com/OpenAdaptAI/openadapt-consilium.git
cd openadapt-consilium
pip install -e ".[dev]"

Quick Start

Set your API keys:

export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
export GOOGLE_API_KEY="AI..."

Python API

from consilium import Council

council = Council()
result = council.ask("What are the key differences between REST and GraphQL?")

print(result.final_answer)
print(result.cost_summary())

CLI

consilium "What are the key differences between REST and GraphQL?"

API Reference

Council

The main orchestrator class.

from consilium import Council

council = Council(
    models=["gpt-5.2", "claude-sonnet-4-6", "gemini-3.1-pro"],
    chairman="claude-sonnet-4-6",
    max_workers=8,
)

Parameters:

Parameter	Type	Default	Description
models	list[str]	["gpt-5.2", "claude-sonnet-4-6", "gemini-3.1-pro"]	Models to query in Stage 1
chairman	str	"claude-sonnet-4-6"	Model for Stage 3 synthesis
max_workers	int	8	Max parallel threads

council.ask()

result = council.ask(
    "Your question here",
    images=[open("screenshot.png", "rb").read()],  # optional
    budget=0.50,        # max USD spend
    system="Be concise",  # system prompt for all models
    skip_review=False,  # skip Stages 2-3
    json_schema={...},  # request JSON output
)

Returns: CouncilResult

Attribute	Type	Description
result.final_answer	str	Synthesized best answer
result.individual_responses	list	Each model's Stage 1 response
result.reviews	list	Each model's Stage 2 review
result.total_cost	float	Total estimated cost in USD
result.total_latency_seconds	float	Wall-clock time
result.cost_summary()	str	Pretty-printed cost breakdown

Agent SDK

For AI agents and automated pipelines, use the dict-based interface:

from consilium import council_query

result = council_query(
    "Analyze this screenshot and list the UI elements",
    images=[screenshot_bytes],
    models=["gpt-5.2", "claude-sonnet-4-6"],
    budget=0.25,
    skip_review=True,  # fast mode: Stage 1 only
)

print(result["final_answer"])
print(result["cost"]["total_usd"])

Returns: JSON-serializable dict with keys:

{
  "final_answer": "...",
  "individual_responses": [
    {
      "model": "openai/gpt-5.2",
      "text": "...",
      "latency_seconds": 2.1,
      "input_tokens": 1500,
      "output_tokens": 400,
      "cost_usd": 0.007
    }
  ],
  "reviews": [...],
  "cost": {
    "breakdown": {"gpt-5.2": 0.007, "claude-sonnet-4-6": 0.012},
    "total_usd": 0.019,
    "total_input_tokens": 3000,
    "total_output_tokens": 800
  },
  "total_latency_seconds": 3.2
}

CLI Reference

consilium "prompt" [OPTIONS]

Options:
  --models TEXT      Comma-separated model IDs (default: gpt-5.2,claude-sonnet-4-6,gemini-3.1-pro)
  --chairman TEXT    Chairman model for synthesis (default: claude-sonnet-4-6)
  --image PATH      Image file to include (repeatable)
  --budget FLOAT    Max spend in USD
  --no-review       Skip Stages 2-3 (faster, cheaper)
  --system TEXT     System prompt for all models
  --json            Output raw JSON

Examples:

# Full 3-stage pipeline
consilium "Compare Python and Rust for CLI tools"

# Fast mode (Stage 1 only)
consilium "Summarize this" --no-review

# With screenshot
consilium "What's on this screen?" --image screenshot.png

# Budget-limited
consilium "Write a haiku about AI" --budget 0.10

# JSON output for piping
consilium "List 3 colors" --json | jq '.final_answer'

# Custom models
consilium "Hello" --models gpt-5.2,gemini-3.1-pro --chairman gpt-5.2

Model Support

Consilium supports any model from these providers:

Provider	Models	Env Var
OpenAI	gpt-5.2, gpt-5.2-pro, gpt-5, gpt-5-mini, o3, o4-mini	OPENAI_API_KEY
Anthropic	claude-opus-4-6, claude-sonnet-4-6, claude-haiku-4-5, claude-sonnet-4-5	ANTHROPIC_API_KEY
Google	gemini-3.1-pro, gemini-3-flash, gemini-2.5-pro, gemini-2.5-flash	GOOGLE_API_KEY

Use any model with the provider/model format:

council = Council(models=["openai/gpt-5.2", "anthropic/claude-sonnet-4-6"])

Budget Control

Consilium tracks costs in real-time and can halt the pipeline when a budget is exceeded:

result = council.ask("Expensive question", budget=0.10)

# If Stage 1 costs > $0.10, Stages 2-3 are automatically skipped
# The best Stage 1 response is returned as the final answer

Error Handling

Individual model failures don't crash the council — failed responses are marked with [ERROR: ...] and the remaining models continue:

result = council.ask("Test")
for r in result.individual_responses:
    if r.text.startswith("[ERROR:"):
        print(f"{r.model} failed: {r.text}")

Development

git clone https://github.com/OpenAdaptAI/openadapt-consilium.git
cd openadapt-consilium
pip install -e ".[dev]"
pytest

License

MIT