blessthis-llm-council

Blind multi-model councils, and direct seat chat, as a local MCP server.

Ask one question — get N independent answers from N different LLMs, each running as a real agent CLI on your machine, each answering anonymously so no model can bandwagon another's take. A synthesizer merges the answers, the council scores them blind, and only then are the hats lifted. When you don't need a whole council, open a direct 1:1 chat with any seat over the same MCP connection. You bring your own models, gateways, and credentials — the package is pure orchestration.

How it works

 you (via your agent host)
        │  brief
        ▼
  council_start ──► N anonymous seats (hat_1 … hat_N)
        │            each seat = one LLM family, spawned as a
        │            headless CLI subprocess (claude / pi / codex)
        ▼
  council_poll  ◄── watch seats work (live progress)
        │
        ▼
  council_answer / council_ask ──► collect answers, cross-examine
        │
        ▼
  synthesis ──► council_score (blind: hats still on)
        │
        ▼
  council_reveal ──► hats off: which model wrote what
        │
        ▼
  council_close

The conductor (your agent host: Claude Code, pi, Cursor, …) talks to the MCP server. The server spawns the seat CLIs, tracks per-(seat, model) health and cooldowns, and persists councils, scores, and chat sessions in a local SQLite database. No seat ever sees another seat's answer before scoring.

How the council works: blind synthesis

The diagram above compresses the flow; here is what actually happens and why it is built that way. The full step-by-step procedures live in the ready-made council agents under agents/ — start with agents/pi/blessthis-council-architect.md.

Hat blindness. Every seat gets an opaque label — hat1, hat2, … — and the hat→model map stays hidden on the server until you call council_reveal. The orchestrating agent (and the human reading its output) therefore collects, cross-examines, and synthesizes answers blind: model identity cannot bias scoring or synthesis. Scores are attached to models server-side only at reveal time, feeding the historical per-model leaderboard (model_scores).

The loop is agent-driven, not server-driven. The MCP server is deliberately dumb — start / poll / answer / ask / reveal / score / close and nothing else. It makes no hidden LLM calls and applies no judgment. All thinking lives in the council agent (shipped for claude/codex/cursor/copilot/gemini/pi), which runs the loop:

 council_start ─► council_poll ─► council_answer ×N ─► council_ask? ─►
                  (long-poll        (fetch each hat's   (cross-examine
                   until done)       full proposal)      conflicts)

        ─► verify + synthesize ─► council_score ─► council_reveal ─► council_close
           (mechanical checks,     (1–10 per hat,     (hats off,           (records
            ONE coherent answer)    hats still on)     de-anonymized)       kept)

1. Collect all hat answers, treating every hat as equally credible until assessed.
2. Cross-examine conflicts with council_ask — probe a specific claim against a specific file or constraint; never nudge seats toward consensus.
3. Verify each proposal on four axes: requirements coverage, codebase fit, trade-off soundness, and a method-driven failure-path sweep — reading the real code, not trusting the seat's self-assessment.
4. Synthesize ONE coherent answer: the strongest proposal as the spine, verified compatible ideas grafted from the others, then an omission audit — every point raised by a non-chosen proposal is either included or explicitly excluded, never silently dropped.
5. Score blind (1–10 per hat) before council_reveal, then close out.

Choosing the best is not a vote. Agreement is not evidence — models share biases, so a popular answer can be systematically wrong. The agent mechanically verifies each option's claims against the real codebase and the stated requirements, and labels every option SOUND, FLAWED (with the specific violation), or UNPROVEN (carried forward as an explicit risk). One seat with a verified trade-off outranks three that merely converge. Where a load-bearing trade-off is genuinely balanced, both options are surfaced with the deciding question — that call stays with the human.

Quick start

Requirements: uv, and at least one seat CLI installed (claude, pi, or codex) with credentials. Linux/macOS first-class; Windows untested (best-effort). Tip: see seats.example.yaml in the repo for a commented full example.

uvx blessthis-llm-council

That's it. The interactive wizard will:

1. Build your seats.yaml (seat templates for common setups, per-seat env prompts, optional 1-token probe per seat).
2. Ask about telemetry (on by default; opt out anytime).
3. Wire up one host of your choice (MCP registration + council agent files). Re-run blessthis-llm-council install to add more hosts.

Then restart/reload your host and either ask the installed council agent ("run a council on …") or call the tools directly (e.g. mcp__llm-council__council_start).

Useful CLI commands (same binary):

blessthis-llm-council seats list|add|edit|remove|probe   # manage seats.yaml
blessthis-llm-council doctor                             # full diagnostics
blessthis-llm-council status                             # fast wiring overview
blessthis-llm-council uninstall                          # clean removal

Two console scripts ship in the package: blessthis-llm-council is the installer/management CLI; blessthis-llm-council-server is the MCP server your host launches (you never run it by hand — the wizard registers it).

seats.yaml

All seat topology and secrets live in one file: ~/.blessthis-llm-council/seats.yaml, mode 0600, never committed anywhere.

telemetry:
  enabled: true          # explicit; ON by default (Decision #22)

seats:
  fable:
    models: [claude-fable-5, claude-opus-4-8]   # preferred first
    agent:
      bin: claude
      args: ["-p", "{prompt}", "--output-format", "json",
             "--dangerously-skip-permissions",
             "--model", "{model}", "--add-dir", "{workdir}"]
      env:
        ANTHROPIC_API_KEY: __REPLACE_ME__

  glm:                       # pi reads its own models.json — no env needed
    models: [glm-5.2]
    agent:
      bin: pi
      args: ["--mode", "json", "--no-extensions", "--no-skills",
             "--no-prompt-templates", "--no-context-files",
             "-p", "{prompt}", "--model", "{model}"]
      env: {}
      # add --tools <allowlist> to sandbox this seat (cuts off MCP
      # servers + extensions)

  gpt:                       # codex runner via an OpenAI-wire gateway
    models: [gpt-5]
    agent:
      bin: codex
      args: ["exec", "--json", "--skip-git-repo-check", "-s", "read-only",
             "--color", "never", "-m", "{model}", "-C", "{workdir}", "{prompt}"]
      env:
        OPENAI_BASE_URL: https://your-gateway.example/v1
        OPENAI_API_KEY: __REPLACE_ME__

A seat is an LLM family, not a runner — the same binary can back several seats with different credentials (e.g. native API + gateway fallback as two seats, preferred-first). args is a pure exec-array (one argv token per element, {prompt} / {model} placeholders required). The wizard writes this file for you; edit later with blessthis-llm-council seats edit <name>.

Codex seats are OpenAI-wire. Non-OpenAI models behind a codex seat need an OpenAI-compatible gateway — set OPENAI_BASE_URL (plus the key) in that seat's agent.env — because the codex CLI only speaks the OpenAI protocol. Codex authentication itself (codex login or OPENAI_API_KEY) is your own concern and is never managed by the council: a missing/failed login simply surfaces as a seat error (nonzero exit with stderr, e.g. codex auth failed), where you can inspect it via seat_health or the error response.

Tools reference (17)

Council (10)

Direct seat chat (6)

Discovery (1)

Host support

Everywhere the server is registered under the canonical name llm-council as { "command": "uvx", "args": ["blessthis-llm-council-server"] } — a non-destructive merge that never touches your other servers or agents (we only ever write/remove entries matching our fingerprint and blessthis-council-* files).

Philosophy

• Why blind? Models bandwagon. Show an LLM another model's answer and you get agreement theater, not independent judgment. Hats stay on until scoring is done, so scores reflect the answer, not the author.
• Why real CLIs instead of API calls? Your seat CLIs already have working auth, tooling, session resume, and model routing. The council spawns the same claude / pi / codex binaries you use interactively — headless, with your config — so there's no second credential store to break and no reimplemented client to drift. This package never makes a direct LLM HTTP call.
• BYO everything. Models, gateways, API keys, base URLs — all yours, all in seats.yaml. No hosted component, no account, no lock-in.

Telemetry

On by default; opt out anytime (telemetry.enabled: false, asked during install with default Yes — Decision #22).

When enabled, we send only anonymized scoring events: {model, kind, score, usage, tool_version, ts, council_uuid, agent}. agent is which agent CLI hosted the council (claude/pi/codex/cursor/copilot/gemini), "unknown" if undetectable.

We never send: your code, file paths, briefs, answer content, notes, seat names, credentials, or seat-health data.

Upgrading from the Postgres pre-release

The store moved from Postgres to SQLite (~/.blessthis-llm-council/state.db). Migration is stop-the-world (A15):

1. Stop the old server (quit every MCP host using it).
2. Run python scripts/migrate_pg_to_sqlite.py --pg postgres://localhost:5433/llm_council --sqlite sqlite:///$HOME/.blessthis-llm-council/state.db --verify — it carries councils, hats (renamed session_id), scores, and model health (seat='legacy'), and verifies per-table row counts (non-zero exit on mismatch).
3. Unset DATABASE_URL and start the new server. Postgres remains supported via DATABASE_URL=postgres://... (install the [postgres] extra).

Uninstall & privacy

blessthis-llm-council uninstall            # removes MCP entries + agent files
blessthis-llm-council uninstall --purge    # also deletes seats.yaml + state.db

Everything lives under ~/.blessthis-llm-council/ (seats, SQLite state) plus the llm-council entry and blessthis-council-* files in your host configs. Nothing else on your system is touched. seats.yaml contains your API keys — keep it 0600 and never commit it.

Upgrading from pre-v1 (Postgres)

The internal pre-release stored state in Postgres. Migration is stop-the-world: stop the old server, run python scripts/migrate_pg_to_sqlite.py --pg <url> --sqlite sqlite:///$HOME/.blessthis-llm-council/state.db --verify (requires pip install blessthis-llm-council[postgres]), then start the new server. Row counts are verified per table.

License

AGPL-3.0-or-later. Fork it, improve it — but improvements to this package must be shared back under the same license.