council-gate

Cross-model adversarial review with an asymmetric, learned escalation gate.

⚠️ Pre-stable. The 1.x line is functionally complete, but CLI flags / env var names / report format aren't frozen until 2.0. Pin a version in CI. See CHANGELOG → Stability.

council-gate runs your document, proposal, or PR diff past 3+ AI models from different providers (Claude, GPT, Gemini, DeepSeek, Kimi, …), then tells you where they agree, where they disagree, and what they're statistically likely to have missed together. Single-model reviews are biased toward their own outputs — consensus from one model isn't a real signal.

Who is this for?

No AI/ML background required. You need a file and ~60 seconds.

Install (one command)

The fastest path on each platform:

uvx council-gate fetches and runs straight from PyPI — no install step, no PATH fix, just works. The native installers handle the same plus persistent PATH so council-gate works in fresh terminals.

Then:

council-gate init                                  # paste your OpenRouter key (free at https://openrouter.ai/keys)
council-gate review path/to/proposal.docx          # report saved to ./council-gate-proposal-<timestamp>.md

That's it. The default model mix runs on ~$1–2 of OpenRouter credit per review. Open the saved markdown in any viewer (Cursor, VS Code, GitHub, even Notes.app).

Claude Code skill (in-chat review)

Drop the skill into your local Claude Code config — it auto-activates on phrases like "review this proposal" or "council review":

mkdir -p ~/.claude/skills/council-gate
curl -sL https://raw.githubusercontent.com/AdishAssain/council-gate/main/skills/council-gate/SKILL.md \
  -o ~/.claude/skills/council-gate/SKILL.md

Restart Claude Code. Then in any session: "review this proposal: ~/path/to/proposal.docx" — Claude will run uvx council-gate review … for you, prompt for your OpenRouter key on first use, and surface the verdict.

Why not /plugin install? A .claude-plugin/marketplace.json is shipped in this repo, but /plugin install from third-party marketplaces is currently blocked by an upstream Claude Code bug (anthropics/claude-code#41653) — the remote backend rejects every non-Anthropic source. The manual skill drop above works on every Claude Code version. We'll switch the recommendation back to /plugin install once the upstream fix ships.

What you can review

Supported formats — auto-detected, no flags:

• Documents: .docx, .pdf, .pptx, .xlsx, .odt, .rtf, .epub (converted to markdown via MarkItDown)
• Plain text & code: .md, .txt, .diff, .patch, source code in any language — read verbatim

Review styles are auto-picked: .docx/.pdf/.odt → proposal; diffs / code / .md → eng. Override with --mode:

Custom prompt? --prompt path/to/my-prompt.md.

What a report looks like

council-gate review proposal.docx saves a single markdown file. Excerpt:

# Council review — `proposal.docx`

**The council disagreed.** Reviewers did not converge on a single set of findings.
Read the individual reviews below before acting.

## At a glance
| | |
|---|---|
| Verdict | ESCALATE |
| Reviewers | 4 returned reviews · 1 errored |
| Escalation score | 0.62 on a 0–1 scale (threshold 0.5; higher = more likely to escalate) |
| Mode | proposal |

## What each reviewer said

### claude-sonnet-5
_Overall: **revise** · worst severity major_ — Core argument is sound but two claims lack support.

| Severity | Kind | Conf | Where | Issue |
|---|---|---|---|---|
| MAJOR | defect | high | Section 2 | Causal claim ("X drives Y") not supported by cited data |
| MINOR | gap | med | Abstract | "Significantly improves" is unquantified |
…

Each finding carries a severity (calibrated against anchors shared by every reviewer), a kind (defect / risk / gap / question / endorse), and the reviewer's confidence. Each reviewer also gives an artifact-level overall verdict (block / revise / accept).

Three verdicts:

• ESCALATE — reviewers disagreed on the findings, or their overall verdicts conflict (one would block what another would accept). Needs human judgement.
• CONSENSUS_CHECK — reviewers agreed, but the report ships with a checklist of dimensions where frontier AI models tend to share blindspots. Don't trust agreement as approval.
• INSUFFICIENT — too few reviewers returned usable output (network, quota, etc).

Other commands

How it works

                  artifact (spec / diff / plan)
                              │
                              ▼
        ┌──────────────────  Council  ──────────────────┐
        │  Adapter A      Adapter B      Adapter C       │  (different providers,
        │  e.g. Claude    e.g. GPT       e.g. Gemini     │   generator excluded)
        └────────┬───────────┬───────────────┬───────────┘
                 │           │               │
                 ▼           ▼               ▼
                Review      Review          Review        (structured findings + raw)
                              │
                              ▼
                     Learned Gate
             (classifier over the review form)
                              │
              ┌───────────────┴───────────────┐
              ▼                               ▼
    escalation score ≥ τ            escalation score < τ
      or block-vs-accept split                │
              │                               │
              ▼                               ▼
        ESCALATE                      CONSENSUS_CHECK
   (formatted message               (verify against known
    for human channel)               correlated blindspots)

The two original primitives are the council (cross-model, not cross-prompt) and the asymmetric gate — only high disagreement is a clean signal; low disagreement is treated as suspect, not as approval.

Why cross-model, not cross-prompt

Same-model self-evaluation is biased. Panickssery et al. (2024) showed that LLM evaluators recognise and favour their own generations — the bias is consistent and measurable. A "council" of three Claude personas reviewing Claude-generated code is doing performance, not adversarial work.

council-gate enforces this with one rule: the generator is excluded from the council. Host integrations declare which provider produced the artifact via COUNCIL_GENERATOR_PROVIDER, and that provider's seats are dropped before the council runs.

Why the gate is asymmetric

The naive design — low disagreement means trust, high disagreement means escalate — is half wrong.

Kim et al. (2025) studied 350+ LLMs and found that models agree roughly 60% of the time when both err. The reported inter-model error correlation of r ≈ 0.77 implies an effective ensemble size of ~1.3 from three models — barely more diversified than asking one. The drivers: shared providers, shared architectures, and shared capability tier. Larger frontier models are more correlated even across providers, not less, because they converge on similar training distributions.

Shin et al. (2025) sharpened this: frontier LLMs systematically over-weight technical validity and under-weight novelty when reviewing scientific work. A shared blindspot, not random noise.

The gate handles this asymmetrically:

In practice, low-disagreement output ships with a checklist (novelty, edge cases, failure modes, missing-data handling, long-term maintenance) for the human to verify against, rather than a green check.

Configuration

Lives at ~/.config/council-gate/.env (XDG-compliant). council-gate never reads from the working directory; nothing lands in your repo.

The settings that matter:

• COUNCIL_MODELS — comma-separated OpenRouter model ids. Default is cost-conscious across six model families (OpenAI, Google, Anthropic, DeepSeek, Moonshot, Zhipu) — works on a $1–2 OpenRouter balance. Swap in flagship variants for higher-stakes reviews; see commented alternatives in .env.
• COUNCIL_GENERATOR_PROVIDER — slug (anthropic, openai, google) of whichever model produced the artifact. The corresponding seats are excluded from the council.
• GATE_THRESHOLD — escalation-score threshold ∈ [0, 1]. Default 0.5.
• COUNCIL_GATE — verdict model: lr (default), tabpfn-lr, tabpfn-gb. All local, pure Python.
• COUNCIL_STRUCTURED_OUTPUT — ask providers to enforce the review-form JSON schema server-side (default on; seats that don't support it fall back automatically). Set 0 to disable.

For an extra council seat using OpenAI's Codex CLI, install and authenticate codex separately (openai/codex).

Privacy & secret-leak guardrails

council-gate sends the artifact body to LLM APIs of every council seat. Three layers of protection ship by default:

1. Filename refusal. The CLI refuses to read files matching obvious secret-bearing patterns (.env, .pem, .key, id_rsa, *credentials*, *secret*, *.gpg, *.kdbx).
2. Inline redaction. The artifact body is scanned for known secret patterns (OpenAI/Anthropic/Google/AWS/Slack/GitHub keys, JWTs, PEM private-key blocks) and redacted with [REDACTED:…] placeholders before any model sees it. Redaction count is logged.
3. Disclosure (this section). Don't pass files containing secrets, PII, or confidential third-party data. The redaction layer is defence in depth, not a substitute for judgement.

To bypass both, pass --skip-redaction-check. Don't use this flag unless you've audited the artifact yourself.

Integrations

council-gate is a CLI; host integrations are thin wrappers that set COUNCIL_GENERATOR_PROVIDER before invoking it.

• Claude Code plugin — .claude-plugin/ (blocked by an upstream marketplace bug; use the manual skill drop until it ships)
• Claude Code skill — integrations/claude-code/
• Codex CLI — integrations/codex/
• GitHub Action — integrations/github-action/

Each is a copy-paste install. None require modifying council-gate itself.

Contributing

PRs, issues, and dogfood reports all welcome. The most useful contribution is running council-gate on your real proposals/PRs/specs and filing the friction.

See CONTRIBUTING.md for dev setup, where help is most needed, and the project's two non-negotiable design primitives.

Related work

• Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. NeurIPS 2024. arXiv:2404.13076
• Kim, E., Garg, A., Peng, K., & Garg, N. (2025). Correlated Errors in Large Language Models. ICML 2025. arXiv:2506.07962
• Shin, H. et al. (2025). Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews. EMNLP 2025 (Oral). arXiv:2502.17086

How disagreement is measured

Two layers, both deterministic and fully local (no LLM judge in the verdict path, no extra downloads):

1. Overall-verdict conflict. Every reviewer files an artifact-level recommendation (block / revise / accept). If one reviewer would block what another would accept, the gate escalates immediately — reviewers can produce near-identical findings and still disagree on whether the artifact is acceptable.
2. A learned classifier over the review form. The gate scores escalation probability from the structured form itself — how many reviewers concurred, the severity and disposition mix of their findings, and how their recommendations split. The default lr model is a logistic regression whose weights ship in the package and run in pure Python, so every verdict can cite exactly which factors drove it. Two alternatives (--gate tabpfn-lr, --gate tabpfn-gb) were distilled from a TabPFN v2 teacher and scored slightly higher in evaluation; all three were selected on a held-out split against source-derived labels.

License

MIT. See LICENSE.

The optional tabpfn-lr / tabpfn-gb gate models are Built with PriorLabs-TabPFN: they were distilled from a TabPFN v2 teacher under the Prior Labs License v1.1 (Apache 2.0 derivative; a copy ships in the package as TABPFN_LICENSE.txt). The default lr gate has no third-party provenance.