deliberate 0.1.0

Multi-LLM ensemble orchestrator for code generation and review

deliberate

Multi-LLM ensemble orchestrator for code generation and review.

Overview

deliberate is a CLI tool that orchestrates multiple LLM agents to collaboratively plan, execute, and review coding tasks. It uses ensemble methods - independent generation, peer voting, and synthesis - to produce higher-quality outputs than any single agent.

Core insight: LLMs are better at evaluating code than generating it. By having multiple agents generate candidates and then cross-evaluate, we exploit this asymmetry.

Features

• Multi-agent planning: Multiple LLMs propose plans, optionally debate, and vote on the best approach
• Isolated execution: Each agent works in its own git worktree to prevent conflicts
• Peer review: Multiple agents score and vote on execution results
• Budget enforcement: Hard limits on tokens, cost, and time
• CI integration: Non-interactive mode with structured output for automation
• OpenTelemetry tracing: Built-in observability support
• Testable: Fake adapters enable full workflow testing at $0

Installation

# Clone the repository
git clone https://github.com/hardbyte/deliberate.git
cd deliberate

# Install with uv (recommended)
uv sync

# Or install with pip
pip install -e .

# With dev dependencies
uv sync --dev
# or
pip install -e ".[dev]"

# With tracing support
uv sync --extra tracing
# or
pip install -e ".[tracing]"

Quick Start

1. Initialize a configuration file:

deliberate init

2. Edit .deliberate.yaml to configure your agents

3. Set up environment variables (copy .env.example to .env and add your API keys):

cp .env.example .env
# Edit .env and add your API keys

4. Run a task:

# Run with inline task description
deliberate run "Add a function to calculate fibonacci numbers"

# Or load task from a file
echo "Add a function to calculate fibonacci numbers" > task.txt
deliberate run "@task.txt"

Configuration

Configuration File Locations

Deliberate searches for configuration files in the following order:

1. Explicit path: deliberate run --config /path/to/config.yaml
2. Current directory: ./.deliberate.yaml or ./deliberate.yaml
3. User config directory: OS-specific user configuration directory
   • Linux/macOS: ~/.config/deliberate/config.yaml
   • Windows: %APPDATA%\deliberate\config.yaml

To create a user-level configuration:

# Create config in user config directory
deliberate init --user

# Or create in current directory (default)
deliberate init

Configuration Format

Edit .deliberate.yaml to configure agents and workflow:

agents:
  claude:
    type: cli
    command: ["claude", "--print", "-p"]
    capabilities: [planner, executor, reviewer]

  gemini:
    type: cli
    command: ["gemini", "-y", "--output-format", "json"]
    # API key loaded from GEMINI_API_KEY environment variable
    capabilities: [planner, reviewer]

workflow:
  planning:
    enabled: true
    agents: [claude, gemini]

  execution:
    enabled: true
    agents: [claude]

  review:
    enabled: true
    agents: [gemini, codex]

Environment Variables

Deliberate uses pydantic-settings to load environment variables:

1. System environment: export GEMINI_API_KEY=your-key
2. .env file: Create a .env file (see .env.example)
3. Config file inline: Add under agents.<name>.env (not recommended for secrets)

Priority: config inline > .env file > system environment

CLI Commands

# Run a task through the deliberate
deliberate run "your task description"

# Load task from a file
deliberate run "@task.txt"
echo "Add error handling to the API" > task.txt
deliberate run "@task.txt"

# Run with specific agents
deliberate run "task" --agents claude,gemini

# Skip planning or review phases
deliberate run "task" --skip-planning
deliberate run "task" --skip-review

# Output as JSON
deliberate run "task" --json

# Enable tracing
deliberate run "task" --trace
deliberate run "task" --otlp-endpoint http://localhost:4317

# Validate configuration
deliberate validate

# Create new config file
deliberate init                # Create .deliberate.yaml in current directory
deliberate init --user         # Create config.yaml in user config directory

Workflow Phases

1. Planning Phase

Multiple agents propose plans for the task. They can optionally debate their approaches, and a judge (another LLM or voting) selects the best plan.

2. Execution Phase

The selected plan is executed by one or more agents in isolated git worktrees. Each agent produces a diff and summary of changes.

3. Review Phase

Multiple agents review the execution results, scoring on criteria like correctness, code quality, completeness, and risk. Votes are aggregated using Borda count or approval voting.

Voting Methods

• Borda count: Points based on ranking position
• Approval voting: Count approvals above a threshold
• Weighted Borda: Borda with weighted reviewer importance

Testing

# Run all tests
uv run pytest

# Run unit tests only
uv run pytest tests/unit/

# Run integration tests
uv run pytest tests/integration/

# Run with coverage
uv run pytest --cov=deliberate

# Skip live LLM tests (default)
uv run pytest -m "not live_llm"

# Run live tests (costs money!)
RUN_LIVE_LLM_TESTS=1 uv run pytest -m live_llm

Architecture

deliberate/
├── src/deliberate/
│   ├── cli.py           # Typer CLI entrypoint
│   ├── config.py        # Pydantic configuration
│   ├── orchestrator.py  # Main workflow coordinator
│   ├── types.py         # Core type definitions
│   ├── adapters/        # LLM backend adapters
│   ├── phases/          # Planning, execution, review
│   ├── voting/          # Vote aggregation algorithms
│   ├── git/             # Worktree management
│   ├── budget/          # Cost tracking
│   ├── tracing/         # OpenTelemetry integration
│   └── prompts/         # Prompt templates
└── tests/
    ├── unit/            # Fast, no-cost unit tests
    ├── integration/     # Workflow tests with fakes
    └── live/            # Optional real API tests

License

MIT