the-llm-council 0.4.9

Multi-LLM Council Framework for adversarial debate, cross-validation, and structured decision-making

The LLM Council

$ council run architect "Design a mass hallucination prevention system"

                    ╔══════════════════════════════════════════════════════════╗
                    ║             ⚖️  THE LLM COUNCIL CONVENES  ⚖️              ║
                    ╚══════════════════════════════════════════════════════════╝

      ┌─────────────────┐      ┌─────────────────┐      ┌─────────────────┐
      │  ┌───────────┐  │      │  ┌───────────┐  │      │  ┌───────────┐  │
      │  │ ╭───────╮ │  │      │  │ ╭───────╮ │  │      │  │ ╭───────╮ │  │
      │  │ │GPT5.2│ │  │      │  │ │ CLAUDE│ │  │      │  │ │GEMINI │ │  │
      │  │ ╰───────╯ │  │      │  │ ╰───────╯ │  │      │  │ ╰───────╯ │  │
      │  │   ◉ ◉     │  │      │  │   ◉ ◉     │  │      │  │   ◉ ◉     │  │
      │  │    ⌣      │  │      │  │    ▽      │  │      │  │    ○      │  │
      │  └───────────┘  │      │  └───────────┘  │      │  └───────────┘  │
      │    JUDGE #1     │      │    JUDGE #2     │      │    JUDGE #3     │
      └────────┬────────┘      └────────┬────────┘      └────────┬────────┘
               │                        │                        │
               │ "I propose we use      │ "Actually, I must      │ "Interesting, but
               │  a vector database..." │  respectfully disagree" │  what about...?"
               │                        │                        │
               └────────────────────────┼────────────────────────┘
                                        ▼
                         ┌──────────────────────────────┐
                         │     🔥 ADVERSARIAL DEBATE 🔥   │
                         │                              │
                         │  GPT5.2: "Your approach has  │
                         │          a cold start issue" │
                         │                              │
                         │  CLAUDE: "Fair, but yours    │
                         │          doesn't scale"      │
                         │                              │
                         │  GEMINI: "Both valid. What   │
                         │          if we combine..."   │
                         └──────────────┬───────────────┘
                                        ▼
                         ┌──────────────────────────────┐
                         │      ✅ VERDICT REACHED ✅     │
                         │                              │
                         │   Synthesized best ideas     │
                         │   Schema-validated output    │
                         │   Confidence: 94%            │
                         └──────────────────────────────┘

[Council] Task completed in 45.2s | 3 judges | 2 debate rounds | Cost: $0.12

A Multi-LLM Council Framework that orchestrates multiple LLM backends to enable adversarial debate, cross-validation, and structured decision-making.

Why Use a Council?

Single-model outputs have blind spots. By running multiple models in parallel and having them critique each other, the council:

• Catches errors that any single model might miss
• Reduces hallucination through cross-validation
• Produces higher-quality outputs via adversarial refinement
• Validates structure with JSON schema enforcement and retry logic

Features

Feature	Description
Multi-Model Council	Run Claude, GPT-4, and Gemini in parallel via single OpenRouter key
Adversarial Critique	Built-in critique phase identifies weaknesses and blind spots
Schema Validation	JSON schema validation with automatic retry for structured outputs
Provider Agnostic	Swap between OpenRouter, direct APIs, or CLI-based providers
Health Checks	Preflight provider health checks with latency tracking
Graceful Degradation	Automatic retry, fallback, and skip strategies for failures
Artifact Store	Persistent storage of drafts with tiered summarization
Secret-Safe Logging	Redaction pipeline prevents credential leakage

Requirements

Requirement	Details
Python	3.10, 3.11, or 3.12
OS	macOS, Linux, Windows (native or WSL)
API Key	At least one provider key (see below)

Supported Providers

Provider	Environment Variable	Notes
OpenRouter	OPENROUTER_API_KEY	Recommended - single key for all models
OpenAI	OPENAI_API_KEY	Direct GPT access
Anthropic	ANTHROPIC_API_KEY	Direct Claude access
Google	GOOGLE_API_KEY or GEMINI_API_KEY	Direct Gemini access
Vertex AI	GOOGLE_CLOUD_PROJECT or ANTHROPIC_VERTEX_PROJECT_ID + ADC	Enterprise GCP - Gemini + Claude

Installation

pip install the-llm-council

With specific providers:

# OpenRouter (recommended - single API key for all models)
pip install the-llm-council

# Direct APIs
pip install the-llm-council[anthropic,openai,google]

# Vertex AI (Enterprise GCP)
pip install the-llm-council[vertex]

# All providers
pip install the-llm-council[all]

# Development
pip install the-llm-council[dev]

Agent Skills (Claude Code, OpenAI Codex, Cursor, etc.)

The LLM Council is available as an Agent Skill following the open Agent Skills standard. This works across Claude Code, OpenAI Codex, Cursor, VS Code, and other skill-compatible agents.

Claude Code

# Step 1: Add the repo as a marketplace
/plugin marketplace add sherifkozman/the-llm-council

# Step 2: Install the plugin
/plugin install llm-council@the-llm-council

Once installed, the council skill is auto-invoked when relevant, or use the /council command:

/council implementer "Build a login page with OAuth"

OpenAI Codex

# Copy skills directory to Codex skills location
cp -r skills/council ~/.codex/skills/

Other Agents (Cursor, VS Code, GitHub, etc.)

Copy the skills/council/ directory to your agent's skills folder. The skill follows the open Agent Skills spec and works with any compatible agent.

Quick Start

CLI Usage

# Set your API key
export OPENROUTER_API_KEY="your-key"

# Run a council task
council run implementer "Build a login page with OAuth"

# Multi-model council (Claude + GPT-5 + Gemini debating)
council run architect "Design a caching layer" \
  --models "anthropic/claude-opus-4-5,openai/gpt-5.1,google/gemini-3-flash-preview"

# Or set via environment variable
export COUNCIL_MODELS="anthropic/claude-opus-4-5,openai/gpt-5.1,google/gemini-3-flash-preview"
council run implementer "Build a login page"

# With health check and verbose output
council run implementer "Build a login page" --health-check --verbose

# Disable artifact storage for faster runs
council run implementer "Quick fix" --no-artifacts

# Get structured JSON output
council run planner "Add user authentication" --json

Python API

from llm_council import Council

council = Council(providers=["openrouter"])
result = await council.run(
    task="Build a login page with OAuth",
    subagent="implementer"
)
print(result.output)

Check Provider Health

council doctor

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                           LLM Council                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────────────────┐ │
│  │    CLI      │───▶│  Council    │───▶│     Orchestrator        │ │
│  │  (typer)    │    │   (API)     │    │                         │ │
│  └─────────────┘    └─────────────┘    │  ┌───────────────────┐  │ │
│                                        │  │  Health Checker   │  │ │
│  ┌─────────────────────────────────┐   │  ├───────────────────┤  │ │
│  │        Provider Registry        │◀──│  │ Degradation Policy│  │ │
│  │  ┌─────────┐ ┌─────────┐       │   │  ├───────────────────┤  │ │
│  │  │OpenRouter│ │Anthropic│ ...  │   │  │  Artifact Store   │  │ │
│  │  └─────────┘ └─────────┘       │   │  └───────────────────┘  │ │
│  └─────────────────────────────────┘   └─────────────────────────┘ │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                    Subagent Configs                          │   │
│  │  router | planner | architect | implementer | reviewer | ... │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────────┐   │
│  │                     JSON Schemas                             │   │
│  │  Validation & retry logic for structured outputs             │   │
│  └─────────────────────────────────────────────────────────────┘   │
│                                                                     │
└─────────────────────────────────────────────────────────────────────┘

Pipeline Flow

0. HEALTH CHECK (optional)
   └── Preflight check of all providers, skip unhealthy ones

1. PARALLEL DRAFTS
   ├── Provider A generates draft
   ├── Provider B generates draft
   └── Provider C generates draft
   └── (Graceful degradation on failures)

2. ADVERSARIAL CRITIQUE
   └── Critic identifies weaknesses, contradictions, blind spots

3. SYNTHESIS
   └── Merge best elements, address critique, validate schema

4. VALIDATION
   └── JSON schema check with retry on failure

5. ARTIFACT STORAGE (optional)
   └── Store drafts and outputs for context management

Subagents

Subagent	Purpose	Example
router	Classify and route tasks	"Is this a bug or feature?"
planner	Create execution roadmaps	"Plan the auth implementation"
assessor	Make build/no-build decisions	"Should we use Redis or Memcached?"
researcher	Deep market/technical research	"Research OAuth providers"
architect	Design systems and APIs	"Design the caching layer"
implementer	Generate production code	"Build the login page"
reviewer	Review code for issues	"Review this PR for security"
test-designer	Design test suites	"Design tests for auth module"
shipper	Create release notes	"Generate changelog for v1.2"
red-team	Security threat analysis	"Analyze attack vectors"

Writing a Provider

Providers are pluggable via Python entry points. See the full Provider Development Guide for detailed instructions.

Quick Example

from llm_council.providers.base import ProviderAdapter, GenerateRequest, GenerateResponse

class MyProvider(ProviderAdapter):
    name = "myprovider"

    async def generate(self, request: GenerateRequest) -> GenerateResponse:
        # Your implementation
        return GenerateResponse(text="...", content="...")

    async def doctor(self) -> DoctorResult:
        return DoctorResult(ok=True, message="Healthy")

Register via pyproject.toml:

[project.entry-points."llm_council.providers"]
myprovider = "my_package.providers:MyProvider"

Reference Implementations

Provider	Type	File
OpenRouter	HTTP API	src/llm_council/providers/openrouter.py
Anthropic	Native SDK	src/llm_council/providers/anthropic.py
OpenAI	Native SDK	src/llm_council/providers/openai.py
Google	Native SDK	src/llm_council/providers/google.py
Vertex AI	Native SDK	src/llm_council/providers/vertex.py
Codex CLI	Subprocess	src/llm_council/providers/cli/codex.py

Configuration

Environment Variables

# OpenRouter (recommended - single key for all models)
export OPENROUTER_API_KEY="your-key"

# Direct APIs
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."
export GOOGLE_API_KEY="..."

# Vertex AI - Gemini (Enterprise GCP)
export GOOGLE_CLOUD_PROJECT="your-project-id"
export GOOGLE_CLOUD_LOCATION="us-central1"  # optional
export VERTEX_AI_MODEL="gemini-2.5-pro"     # optional, default: gemini-2.0-flash

# Vertex AI - Claude (Enterprise GCP)
export ANTHROPIC_VERTEX_PROJECT_ID="your-project-id"
export CLOUD_ML_REGION="global"              # Claude uses global region
export ANTHROPIC_MODEL="claude-opus-4-5@20251101"  # model with version

# Auth for Vertex AI: gcloud auth application-default login OR
# export GOOGLE_APPLICATION_CREDENTIALS="/path/to/sa.json"

# Multi-model council: comma-separated OpenRouter model IDs
export COUNCIL_MODELS="anthropic/claude-opus-4-5,openai/gpt-5.1,google/gemini-3-flash-preview"

# Optional: Model pack overrides for specific task types
export COUNCIL_MODEL_FAST="anthropic/claude-3-5-haiku"    # Quick tasks
export COUNCIL_MODEL_REASONING="anthropic/claude-opus-4-5" # Deep analysis
export COUNCIL_MODEL_CODE="openai/gpt-5.1"                # Code generation
export COUNCIL_MODEL_CRITIC="anthropic/claude-sonnet-4-5" # Adversarial critique

Per-Subagent Reasoning Configuration (v0.3.0+)

Subagents can be configured with provider preferences, model overrides, and extended reasoning/thinking budgets in their YAML configs:

# src/llm_council/subagents/red-team.yaml
name: red-team
model_pack: harsh_critic

# Provider preferences
providers:
  preferred: [anthropic, openai]
  fallback: [openrouter]
  exclude: [google]

# Model overrides per provider
models:
  anthropic: claude-opus-4-5
  openai: o3-mini
  google: gemini-3-pro

# Extended reasoning/thinking configuration
reasoning:
  enabled: true
  effort: high           # OpenAI o-series: low/medium/high
  budget_tokens: 32768   # Anthropic: 1024-128000
  thinking_level: high   # Google Gemini 3.x: minimal/low/medium/high

Provider	Parameter	Values	Description
OpenAI	effort	low/medium/high	Reasoning effort for o-series models
Anthropic	budget_tokens	1024-128000	Extended thinking token budget
Google	thinking_level	minimal/low/medium/high	Gemini 3.x thinking level

Default Reasoning Tiers (v0.4.0+)

All subagents have pre-configured reasoning defaults based on task complexity:

Tier	Subagents	Config	Use Case
High	architect, assessor, planner, reviewer, red-team	effort: high, budget_tokens: 16384	Deep analysis, critical decisions
Medium	implementer, researcher	effort: medium, budget_tokens: 8192	Balanced code/research tasks
Disabled	router, shipper, test-designer	enabled: false	Fast tasks, no overhead

Config File

# ~/.config/llm-council/config.yaml
providers:
  - name: openrouter
    api_key: ${OPENROUTER_API_KEY}
    default_model: anthropic/claude-opus-4-5

defaults:
  timeout: 120
  max_retries: 3
  summary_tier: actions

CLI Reference

council run <subagent> "<task>"    # Run a council task
council doctor                      # Check provider health
council config                      # Show configuration

# Options
--providers, -p    Comma-separated provider list
--models, -m       Comma-separated OpenRouter model IDs for multi-model council
--timeout, -t      Timeout in seconds (default: 120)
--max-retries      Max retry attempts (default: 3)
--health-check     Run preflight health check
--no-artifacts     Disable artifact storage
--no-degradation   Disable graceful degradation
--json             Output structured JSON
--verbose, -v      Verbose output

Development

# Clone the repository
git clone https://github.com/sherifkozman/the-llm-council.git
cd the-llm-council

# Install with dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check src/
mypy src/llm_council

Contributing

Contributions are welcome! See our Roadmap for planned features and Contributing Guide for details.

Quick Start

# Fork and clone
git clone https://github.com/YOUR_USERNAME/the-llm-council.git
cd the-llm-council

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Run linting
ruff check src/ && mypy src/llm_council

Contribution Workflow

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Make your changes
4. Test your changes (pytest)
5. Lint your code (ruff check src/ && mypy src/llm_council)
6. Commit with a clear message (git commit -m 'Add amazing feature')
7. Push to your branch (git push origin feature/amazing-feature)
8. Open a Pull Request

What We're Looking For

• New Providers: Add support for more LLM backends
• New Subagents: Create specialized agents for specific tasks
• Bug Fixes: Found a bug? We'd love a fix!
• Documentation: Improvements to docs are always welcome
• Tests: More test coverage is great

Security

For security concerns, please see our Security Policy or email [e-mail].

Key security features:

• CLI adapters use exec-style subprocess (no shell injection)
• Environment variable allowlisting prevents secret leakage
• Path traversal protection in artifact storage
• Configurable secret redaction in logs

License

MIT License - see LICENSE for details.

Acknowledgments

Built with:

• Pydantic - Data validation
• Typer - CLI framework
• Rich - Terminal formatting
• httpx - Async HTTP client

---

When one model isn't enough, convene a council.

_{~ vibe coded by Sherif Kozman & The LLM Council ~}