ai-jury

Convene a cross-vendor multi-agent review jury: native coding-agent CLIs from different vendors review the same diff, PR, or issue, cross-examine each other, verify, and reach one verdict — a chair's synthesis or a panel vote.

Install once. Run a cross-vendor review jury anywhere.

Most "multi-model review" tools call models at the API level. This one drives any AI agent provider: vendor native CLI agents (claude, codex, agy), hosted API providers (Anthropic, OpenAI, Gemini, xAI Grok, OpenRouter, DeepSeek, Groq, and any other OpenAI-compatible API such as Mistral), free local open-weight models (Ollama, llama.cpp, vLLM, LM Studio), and arbitrary coding-agent CLIs (vendor = "cli" like Aider, Goose, OpenHands) — so every reviewer runs in its own environment with its own tooling. Each agent runs headless; the orchestrator owns the round structure.

        ┌──────── round 1 ────────┐   ┌─ round 2 (adaptive) ─┐   ┌─ verify + synthesis ─┐
diff ──▶ claude codex agy deepseek  ▶ each rebuts the      ▶ chair verifies, then   ▶ verdict
         (parallel, independent)           others' findings       consolidates             + report

Highlights: universal agent provider support (cloud CLIs, hosted APIs, local models, arbitrary CLI agents) · free/offline local reviews · secure by default (reviewers run sandboxed/read-only) · jury init setup · debate + verification · chair verdict or a panel vote · review a PR, a diff, or an issue · live / full-transcript output · CI gating · incremental review · suggested patches · large-diff chunking. Configure once in jury.toml; mix cloud CLIs, hosted APIs, and local models however you like.

Why

Different models miss different things. Running them as an adversarial panel — each seeing the others' findings and arguing — surfaces more real issues and filters more false positives than any single reviewer. The research-backed lever is vendor heterogeneity, not more rounds — and universal provider support lets you mix frontier cloud CLIs, hosted APIs (OpenRouter, DeepSeek, Groq), custom CLI agents (Aider), and free local models at zero marginal cost. See docs/architecture.md and docs/feasibility.md.

Install

# macOS & Linux (Homebrew — recommended):
brew install berkayturanci/ai-jury/ai-jury

# Or standalone curl installer:
curl -fsSL https://ai-jury.dev/install.sh | sh

# Or via pipx / pip:
pipx install ai-jury

Requires Python 3.11+. Try it instantly, no config: jury --mock runs the full offline deliberation on a diff bundled with the package (add --theater to watch the panel animate). Then scaffold a config with jury init (it detects your installed agents and local models). You need at least one reviewer: an agent CLI (claude, codex, agy, aider), a free local model via Ollama, or a hosted-API reviewer (Anthropic, OpenAI, Gemini, xAI Grok, OpenRouter, DeepSeek, Groq — or Moonshot Kimi, Mistral or any other OpenAI-compatible API through vendor = "openai-compatible") — no CLI install or interactive login needed, useful for CI and containers; missing/unreachable/unkeyed reviewers are skipped. gh is needed for --pr / --post.

Install into an agent

The above installs the CLI. Installing ai-jury as a plugin — so its skill is available inside an agent session — is a separate step, and the CLI is still required either way. Jump to the agent you use:

Each badge jumps to that agent's box; open it for the commands. (A browser scrolls to a collapsed <details>; it does not expand one.)

The agent names elsewhere in this README are reviewers the jury convenes. Here the same agents are hosts you install ai-jury into — a different role.

claude plugin marketplace add https://github.com/berkayturanci/ai-jury
claude plugin install ai-jury@ai-jury

claude plugin marketplace update ai-jury
claude plugin update ai-jury@ai-jury

codex plugin marketplace add https://github.com/berkayturanci/ai-jury
codex plugin add ai-jury@ai-jury

codex plugin marketplace upgrade
codex plugin add ai-jury@ai-jury

agy plugin install https://github.com/berkayturanci/ai-jury
agy plugin enable ai-jury

agy plugin install https://github.com/berkayturanci/ai-jury

git clone --depth 1 https://github.com/berkayturanci/ai-jury   ~/.cursor/plugins/local/ai-jury

git -C ~/.cursor/plugins/local/ai-jury pull

Full detail, including what each route actually registers and where files land: docs/install.md.

For development, install the dev extras (linting, build, and coverage tooling):

pip install -e ".[dev]"   # or: make install

That resolves ruff, build and coverage fresh. To get the exact versions CI uses — the ones pinned in the tracked uv.lock, with hashes — install from the lock instead. It is also the ruff the ruff-format pre-commit hook runs:

uv sync --locked --extra dev

Coverage

Test coverage is measured with coverage.py — a dev-only dependency. The runtime stays standard-library-only.

Measure it locally with one command:

make coverage          # run the suite under coverage, print the report, write htmlcov/
# or, without make:
./scripts/coverage.sh

Either entry point runs:

python3 -m coverage run -m unittest discover -s tests
python3 -m coverage report

Threshold. The enforced minimum total coverage is 98% — one number, fail_under in pyproject.toml under [tool.coverage.report], enforced by a dedicated coverage job in CI (.github/workflows/ci.yml, Ubuntu / Python 3.13). CI fails if total coverage drops below that floor. The gate runs in a single job rather than across the whole test matrix to keep CI cheap and free of cross-OS path noise. tests/test_docs_coverage_gate.py asserts the percentage in this paragraph is still the one pyproject.toml sets, because the two were out of step for several releases.

The floor is not the measurement. The gate is deliberately set below what the suite actually reaches, so that an unrelated refactor moving the total by a fraction of a point is caught in review rather than by a red CI job, and it is raised on purpose as coverage improves. For the current measured total see the comment directly above fail_under in pyproject.toml, the live coverage badge, or your own make coverage run — not this paragraph, which would go stale.

Measurement method. Branch coverage is enabled (branch = true) and the package is measured by import name (source = ["ai_jury"]).

Exclusions. Intentionally-untested paths are excluded so the number stays honest:

• src/ai_jury/__main__.py is omitted (a thin python -m entry shim).
• Lines matching these patterns are excluded from the count: pragma: no cover, if __name__ == "__main__":, raise NotImplementedError, if TYPE_CHECKING:, and abstract-method decorators.

Add # pragma: no cover to any new line that is genuinely not worth testing.

Live smoke tests

The default test suite uses mock adapters only, so the real native CLIs are never invoked — a breakage in argv format, stdin handling, or output capture in the concrete adapters would go unnoticed until a live run. The optional live smoke tests close that gap: they run a tiny, cheap review prompt (a two-line diff) through each installed real adapter and assert the run succeeds (ok, non-empty output, error_code is None).

They are opt-in and skipped entirely unless JURY_LIVE=1 is set, so they never run in make test or in CI.

Requirements for a meaningful live run:

• The agent CLIs you want to exercise must be installed and on your PATH (claude, codex, agy) and authenticated for non-interactive use.
• Any agent whose CLI is not installed is skipped individually, so a machine with only claude still exercises that one adapter.

Run them:

make live-smoke
# equivalent to:
JURY_LIVE=1 PYTHONPATH=src python3 -m unittest discover -s tests -v

They are intentionally excluded from the CI matrix (no CLIs, auth, or secrets are available there) and are meant to be run locally before a release or when touching the adapter layer.

Review-quality benchmark

A small, directional benchmark (benchmark/) measures whether a jury's findings line up with hand-authored expectations for a handful of fixture diffs (obvious logic bug, subtle boolean-guard bug, missing error handling, a false-positive trap, and a docs-only change).

It is not a universal quality claim. The default offline mode is deterministic and runs with no live CLIs: it scores each fixture's recorded sample findings against an expected spec, which validates the scorer and the recorded baselines — it does not measure live review quality. (--mock is deliberately not used per fixture: the mock adapter emits a fixed finding regardless of the diff, so it would be fake signal.) Only the opt-in live mode (JURY_BENCH_LIVE=1) runs real agents and measures quality.

make benchmark                                          # offline, deterministic
JURY_BENCH_LIVE=1 PYTHONPATH=src python3 -m ai_jury.benchmark  # live (opt-in)

The lift, measured. A config-sweep (benchmark/sweep.py) runs the same labeled fixtures with each model solo vs the panel vs the full jury — isolating what the panel adds, without the result hinging on which single model you'd pick. On a live four-vendor run (v1.1.0): every model missed seeded bugs run alone (best: Claude/Qwen 67%; Codex/Agy 33%, all at 100% precision), while the four-vendor panel caught 100% of them. Vendor diversity is the robust lever (recall); the precision/verification effect is within noise at N=5 and not claimed. Directional, reproducible — full table + method in docs/benchmark-results.md.

See benchmark/README.md for the fixture list, the expected/recorded schema, and the match/scoring rules.

Usage

Set up — detect your agents and write a jury.toml:

jury init                     # scaffold (detects agents + local models)
jury init --wizard            # guided setup: numbered, skippable questions
jury init --preset balanced   # from a preset: offline | fast | balanced | thorough

Review a pull request:

jury --pr 123                  # review a GitHub PR
jury --pr 123 --auto           # auto-depth: scale rounds/verify to the diff
jury --pr 123 --decision vote  # verdict by panel vote instead of a single chair
jury --pr 123 --post           # post the verdict back as a PR comment

Review an issue — completeness & clarity (READY / NEEDS-INFO / UNCLEAR):

jury --issue 42
jury --issue 42 --decision vote   # tie-break order: NEEDS-INFO > UNCLEAR > READY
jury --issue 42 --live --post     # stream + post each step to the issue thread

Watch it happen — live in your terminal:

jury --pr 123 --live                            # stream each step as it lands
jury --pr 123 --transcript                      # full play-by-play (reviews, debate, reasoning)
jury --pr 123 --theater                         # animated "deliberation" view
jury --pr 123 --theater --theater-style pixel   # …as a pixel-art room (truecolor terminal)

Post to the PR:

jury --pr 123 --post-progress             # one sticky comment, updated each round/chunk
jury --pr 123 --post --post-mode phased   # Round 1 / debate / decision as separate comments
jury --pr 123 --suggest-patches           # also emit inspectable patches for verified findings
jury --pr 123 --incremental               # review only changes since the last run

Local diffs & offline:

git diff origin/HEAD... | jury --diff-file -   # review the current branch
jury --diff-file changes.diff                  # review a diff file
jury --mock                                     # offline demo on a bundled sample, no live CLIs
jury --mock --rounds 1                          # offline: independent review only (no debate)

Inspect:

jury config show   # the effective resolved config + its source
jury --doctor      # readiness check + actionable next steps

Drive one agent from an orchestrator — jury run-agent runs a single configured agent for one role and prints an ai-jury.run-agent.v1 JSON result with attribution, using the same adapters, read-only flags and error taxonomy as a panel run. review/gate/chair are always read-only; implement/fix need an explicit --allow-write. See cookbook §21.

jury run-agent --agent claude --role review --prompt-file gate.md
jury run-agent --agent codex:gpt-5.2 --role implement --allow-write --prompt-file task.md

Pre-commit hook (.pre-commit-config.yaml):

- repo: https://github.com/berkayturanci/ai-jury
  rev: v1.19.1
  hooks:
    - id: ai-jury

GitHub Action (.github/workflows/ai-jury.yml):

name: ai-jury
on: [pull_request]
jobs:
  review:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write   # the default args (--post) write the review to the PR
    steps:
      - uses: actions/checkout@v4
      - uses: berkayturanci/ai-jury@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          anthropic-api-key: ${{ secrets.ANTHROPIC_API_KEY }}

Commit a jury.toml naming *-api seats (each with a model) for the keys you pass — the built-in default seats are agent CLIs a runner does not have, and the keys alone do not form a panel, so a keyless/config-less run exits with no usable agents. jury init scaffolds those seats; fill in each seat's model.

A sample report is in docs/example-run.md. For a real four-vendor run where the jury reviews its own repository (with honest notes on false positives and the local model's contribution), see docs/example-live-review.md.

Theater mode — --theater

An opt-in, animated deliberation view of a live run: the models sit around a table and take turns speaking as the run moves through review → debate → verify → decision, then reach a decision together — by panel vote, or recorded by the chair. There is no judge; the jurors deliberate with each other.

It renders in two styles, both live in your terminal. An opt-in pixel-art room (--theater-style pixel; needs a truecolor + unicode terminal):

…and the default flat ANSI scene (--theater):

It is presentation-only — it reads the same per-phase on_event stream the report does, so it mirrors the real run (jury --mock --theater gives a deterministic, offline demo). It adapts to PR vs issue and chair vs vote, shows debate rounds / early-stop / disputes, seats many jurors (with a compact roster fallback), and degrades to the plain --live step stream on a non-interactive terminal (or, for pixel, to the flat scene without truecolor/unicode). Details: docs/theater-design.md.

Replay a saved run — jury replay

Re-watch a finished run without re-spending a single token:

jury replay run.json --theater                 # replay in the deliberation theater
jury replay run.json --theater --theater-style pixel
jury replay run.json --decision vote --mode issue   # re-tally the panel-vote finale

replay loads a serialized outcome — a bare outcome_to_dict dump or a result-cache entry (via its outcome key) — and re-drives the theater with the exact per-phase event sequence the live run emitted (review → debate → verify → synthesis, then the vote/chair finale). No orchestration, no network, no agents. Off a tty (or without --theater) it degrades to the plain --live step stream. The serialized outcome does not record the run mode, so --mode code|issue selects the vote vocabulary when you replay with --decision vote. A jury --format json report cannot be replayed — it does not carry the per-agent deliberation stream — and is rejected with a pointed message.

Output formats

Use --format {markdown,json,sarif,keel-reviews} (default markdown) to control what is written to stdout or --output. --metadata-json is independent and always writes the metadata block to its own file, and the --ci exit code is computed the same way regardless of format.

jury --diff-file changes.diff --format json         -o report.json
jury --diff-file changes.diff --format sarif        -o report.sarif
jury --diff-file changes.diff --format keel-reviews -o reviews.json

JSON

A structured report with these top-level keys:

The output is deterministic for a deterministic run (e.g. --mock) and contains only legitimate finding fields — never raw diff or prompt text.

This JSON drives the "Load a real run" panel on the website: drag it on to play the real reviewers, findings, and verdict through the in-browser theater (fully client-side — nothing is uploaded). For the terminal jury replay (above), save the default outcome dump (-o run.json, not --format json) instead — the --format json report omits the per-agent deliberation stream, so jury replay rejects it, as noted above.

keel-reviews

A JSON array of per-reviewer review records — one per seat that ran, plus the chair as reviewer: "chair" — in the shape a consumer of head-pinned per-reviewer verdicts accepts (keel's keel review --reviews <file>). The chairing agent sits on the panel: it reviews in round 1 like anyone else, so a three-agent bench yields three ballots — the chairing agent's among them — and a fourth record, the chair's synthesis. Each record says which agent chaired and what became of its ballot, so a chair that reviewed is not indistinguishable from one that only synthesised. --min-reviews N requires a review count: the ceiling is checked before the panel runs, the number actually supplied afterwards.

Every record names what it read, and the name has to exist: scope carries the reviewer's own Checked: line — resolved against the change, so only tokens naming a path or a symbol actually in the diff count — the files it attached to its findings, or, under --issue, where a finding has no file to name, the claims it raised; testing carries what it ran or says plainly that nothing was. A reviewer whose reply names nothing checkable is recorded as an ABSTAIN whose scope says why, never as an APPROVE with a placeholder — an agent that named nothing did not review, and neither did one whose Checked: nothing merely had the shape of naming something.

A ballot is not automatically a review. counts_as_review on each record is the one definition (ai_jury.panel.is_review): a panelist entry, with a scope naming something checkable, and a verdict that is not ABSTAIN. The chair's synthesis, a seat that returned nothing, and a seat that answered without naming anything are all in the bundle — the report has to be able to say which agent produced what — and none of them is counted, because the consumer would refuse it. --min-reviews counts the same thing the consumer does.

And a ballot that did not review says why. abstention_cause carries one of five — silent, named_nothing, not_in_change, refused, adapter_failed — and every count of the seats that supplied no review is that field tallied, in the report line, the --min-reviews failure and --metadata-json's panel block alike. A seat that named a file and then refused is reported as a refusal, not as one that named nothing; a seat that named src/made/up.py is reported as having named something this change does not contain, which is a different failure again.

[
  {
    "reviewer": "claude",
    "verdict": "REQUEST_CHANGES",
    "scope": "Checked, as stated by the reviewer: `src/example.py`. Named 1 file(s): `src/example.py`.",
    "findings": [
      {"severity": "major", "path": "src/example.py", "line": 42, "message": "..."}
    ],
    "testing": "Tested, as stated by the reviewer: python -m pytest tests/test_example.py — 12 passed",
    "vendor": "anthropic",
    "model": "claude-sonnet-4-5",
    "model_source": "requested",
    "counts_as_review": true
  }
]

See the report-format contract for how each field is derived.

SARIF

Valid SARIF 2.1.0 suitable for GitHub code scanning. Results are drawn from consensus group representatives (falling back to raw findings). Each result maps to a physicalLocation (artifactLocation.uri = file, region.startLine = line when known), message.text = the claim, and a stable ruleId of jury/<severity>. Severity maps to the SARIF level as:

Upload to GitHub code scanning:

The standard way is the github/codeql-action/upload-sarif GitHub Action. Results then show up in the PR's Code scanning view and the repo's Security tab. The job needs security-events: write (to upload) and contents: read:

name: Jury code scanning

on:
  pull_request:

permissions:
  contents: read
  security-events: write   # required by upload-sarif

jobs:
  jury:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # need the base commit to diff against

      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"

      - name: Install jury
        run: pip install ai-jury   # or: pip install .

      - name: Produce SARIF from the PR diff
        run: |
          git diff "origin/${{ github.base_ref }}...HEAD" > pr.diff
          jury --diff-file pr.diff --format sarif -o jury.sarif

      - name: Upload to code scanning
        uses: github/codeql-action/upload-sarif@7211b7c8077ea37d8641b6271f6a365a22a5fbfa # v4.36.0
        with:
          sarif_file: jury.sarif

This uses a diff file so no agent CLIs or gh token are required to generate the SARIF. To review the PR via --pr instead (which shells out to gh), set GH_TOKEN: ${{ github.token }} on that step and ensure the agent CLIs are installed and authenticated on the runner.

As a manual alternative, upload an existing SARIF file with gh:

gh api -X POST repos/OWNER/REPO/code-scanning/sarifs \
  -f commit_sha="$SHA" -f ref="$REF" \
  -f sarif="$(gzip -c report.sarif | base64 -w0)"

Configuration — jury.toml

Don't hand-write it the first time — scaffold one with jury init, which detects your installed agents and writes a valid config:

jury init                              # interactive: pick agents, rounds, chair
jury init --wizard                     # guided: numbered questions for the most-used settings (skippable)
jury init --preset balanced            # one-command preset: offline | fast | balanced | thorough
jury init --list-agents                # show known agents + availability
jury init --list-models                # list local (Ollama) models you can pick
jury init --agents claude,codex,qwen --rounds 2   # non-interactive / scriptable

It detects which agent CLIs are installed and, for a local agent, discovers the models on your Ollama/OpenAI-compatible server so you can pick one in the interactive prompt (e.g. gemma:2b, deepseek-coder:6.7b). It uses the secure-by-default agent templates and refuses to overwrite an existing file without --force. The resulting jury.toml:

[jury]
rounds = 2          # 1 = review only, 2 = review + debate
chair  = "claude"   # which agent synthesizes the verdict
timeout = 300       # per-agent wall-clock seconds (a hung CLI is killed at this bound)
parallel = true

[[agent]]
name = "claude"
vendor = "anthropic"   # anthropic | openai | google | xai
command = "claude"
# model = "claude-opus-4-8"
extra_args = ["--output-format", "text", "--disallowed-tools", "Edit,Write,NotebookEdit,Bash", "--dangerously-skip-permissions"]

Override per run with --rounds, --chair, --config.

Fail-soft is not a multi-vendor guarantee. A missing or failing agent is skipped and the run continues — which is right for one reviewer having a bad day, and wrong as a silent default, because a panel that collapsed to one vendor produces a report shaped exactly like a healthy one. Two guards keep them apart: an agent that exits 0 with a refusal or its own usage banner is recorded as a typed failure rather than a review, and a run that claimed cross-vendor consensus and did not form one exits 3 (--min-vendors, default 2). The claim is what your config NAMES, so a jury.toml naming three vendors fails on a machine carrying one installed CLI — a missing CLI is not an exemption. Opt out with --no-min-vendors ([jury.ci] min_vendors = 0), or use --strict to catch the missing CLI at startup instead. jury --doctor reports reachable vendors, but never contribution — only a run can prove that. See docs/parameters.md.

The config is validated on every run. Check it without running a review with jury --config-validate (exit 0 valid, 2 invalid); add --strict-config to turn warnings into errors. See the parameter reference for every field, allowed value, and default (CLI flags + jury.toml), and docs/configuration.md for the validation behaviour (hard errors vs. warnings) and other config semantics.

Local / open-weight reviewer (free, offline)

A panelist can run on a local open-weight model via any OpenAI-compatible server (Ollama, llama.cpp llama-server, vLLM, LM Studio). Use vendor = "local" with an endpoint and a model — it talks plain HTTP (stdlib only, no extra deps) and participates in every round and the consensus exactly like a CLI agent:

[[agent]]
name = "qwen"
vendor = "local"
model = "qwen2.5-coder:7b"
endpoint = "http://localhost:11434/v1"   # default; Ollama's OpenAI-compatible API

# One-time: install Ollama and pull a coding model.
ollama pull qwen2.5-coder:7b
# Review with zero cloud cost, fully offline:
jury --diff-file changes.diff --config local-only.toml

Trade-off (be honest): a small local model is a weaker reviewer than a frontier CLI — its value is added diversity (the load-bearing advantage for a jury) and zero marginal cost, not parity. The sweet spot is mixing one local panelist with one or two cloud CLIs: more vendor heterogeneity, lower spend. An unreachable server fails fast with a typed connection_error (the run continues with the other agents). See the benchmark note for the measured diversity contribution.

Repository review policy (optional)

A repository under review may ship an optional, separate review policy that tells the jury what to care about for that project (high-risk paths, focus areas, forbidden output, severity overrides, a checklist, and doc links). This is distinct from the runtime jury.toml: it configures what reviewers should care about, not how the agent runs.

It is auto-discovered from .jury/policy.toml or jury-policy.toml, or pointed at explicitly:

jury --mock --diff-file examples/sample.diff --policy .jury/policy.toml

The policy is maintainer-authored, so it is injected into the review prompts as a clearly separated, trusted section (distinct from the untrusted diff/context fences). Missing policy files are allowed (no-op); a malformed one is a clear error. See examples/policy.toml for a generic example and the architecture docs for the schema.

Data flow / privacy

What gets sent to each agent is governed by [jury.context] in jury.toml (and overridable per run on the CLI):

[jury.context]
mode = "diff-only"      # "diff-only" (default) or "expanded"
redact_secrets = true   # scrub recognized secrets before sending (default on)

Context modes — what leaves your machine for each reviewer:

• diff-only (default): agents receive only the diff. Any surrounding PR context (title/body) is dropped. This is the smallest data surface.
• expanded: agents additionally receive the PR title/body context (when reviewing with --pr) to improve review quality. Use this only when you trust the configured agent endpoints.

Either way, no source files outside the diff, no repository history, and no environment variables are read or sent.

Secret redaction — before anything is sent to an agent, the diff (and any context) is passed through a redactor (src/ai_jury/redaction.py) that masks recognized secrets: PEM private keys, AWS access keys, GitHub/OpenAI tokens, Bearer tokens, and generic api_key/secret/token assignments (including base64-style values). Each hit becomes [REDACTED:<kind>]. Redaction is on by default.

Controls:

• jury.toml: [jury.context] mode = "diff-only"|"expanded" and redact_secrets = true|false.
• CLI (override config for a single run): --context-mode diff-only|expanded, and --redact / --no-redact.

Posting to GitHub (--post-summary, --post-inline) sends the rendered report / comments to the GitHub API; use --dry-run with --post-inline to preview the inline payload without any network call. See SECURITY.md for the full data-flow and redaction reference.

No telemetry (by default and always) — this project collects and sends no telemetry and no analytics, not now and not behind any opt-in flag. The tool never phones home. The only network activity is performed by the agent CLIs you explicitly configure (and gh for --pr / --post*).

Diagnostics — jury --doctor

Run a local readiness check that surfaces common configuration problems:

jury --doctor                          # print a readable report
jury --doctor --json                   # the same facts as one JSON document
jury --doctor --write diagnostics.json # also write the report as JSON

The report covers the tool version, Python version, OS, a config summary (rounds, chair, context mode, enabled agents), which agent CLIs are available on your PATH, and any detected config warnings. The output is safe to share: secret-like config values are redacted via the same redactor used for prompts, and the report never includes the diff under review or any agent output. Diagnostics are built locally and only written to disk when you pass --write PATH.

--doctor --json prints the same facts as one JSON document on stdout and nothing else, for wizards and orchestrators — per agent: transport (cli / api / local), availability and the reason when unavailable, the command or endpoint, resolved binary, version, capabilities, discovered models, and effort support. The schema is ai-jury.doctor.v1, documented in docs/configuration.md:

jury --doctor --json | jq '.agents[] | select(.available) | .name'

Use it from another project (skill)

A Claude Code skill ships in skills/ai-jury/. Install it as a plugin from this repo (it doubles as a single-plugin marketplace):

/plugin marketplace add berkayturanci/ai-jury
/plugin install ai-jury@ai-jury

That is Claude Code's short form; docs/install.md has the CLI commands, the other three agents, and the update path — which is not a re-install, since plugin install is a no-op on an installed plugin.

Or drop skills/ai-jury/ into a project's .claude/skills/ manually. Either way the agent can convene the jury on demand, and it composes with existing review workflows: run the jury for a cross-vendor pass, then act on the consensus findings. For support status per platform, see the platform support matrix.

For a concrete consumer: keel's review gate (step s7, auto-enabled on its tier-3 risk class) runs ai-jury as its cross-vendor jury — fail-soft, a no-op when ai-jury isn't installed, so keel never hard-depends on it. keel's optional keel-visual run visualizer surfaces the jury on the review step when a run used it.

How it works

Report format contract

The markdown report is the tool's user-facing output and a contract for downstream skill/workflow consumers, so it changes only deliberately. tests/test_report_golden.py renders the report for several scenarios (full jury run, single-round, verified-finding, failed-agent, missing-agent) and compares each against a committed snapshot in tests/golden/*.md. Unintended formatting drift fails CI; an intentional change shows up as a reviewable fixture diff. Durations (the only non-deterministic token) are normalized to 0s before comparison. Regenerate fixtures after an intentional change with UPDATE_GOLDEN=1 PYTHONPATH=src python3 -m unittest tests.test_report_golden. See docs/report-format.md for details.

Prior art & how this differs

This is a known pattern, not a new invention. The closest project is Magpie (multi-vendor CLI review + debate, with a benchmark showing debate lifts bug detection to ~80%); see also agent-jury, the-jury, and Mozilla.ai's Star Chamber. The jury/courtroom metaphor shows up in research too — e.g. VulTrial (ICSE 2026), which casts prosecutor / defense / judge / jury LLM agents at vulnerability detection. ai-jury aims to be the smallest drop-in version: stdlib-only Python, a single jury.toml, and a Claude Code skill that snaps into an existing repo's review workflow. See the ecosystem comparison & capability matrix for how it differs from hosted, API-level, and other native-CLI tools, and docs/feasibility.md for the supporting research.

Status

Active (v1.19.1). The full pipeline runs end-to-end with the real CLIs and the offline --mock path is covered by tests. Shipped: structured findings + tiered consensus (consensus / majority / single-reviewer), a verification pass that drops false positives, universal agent provider support (cloud CLIs, hosted APIs, arbitrary coding-agent CLIs, local models), chair verdict or panel vote (--decision vote), PR / diff / issue / commit review (--pr, --diff-file, --issue, --commit, --commits), abstention accounting in the panel, live play-by-play (--live) and full-transcript / verbose output, animated deliberation theater (--theater, --theater-style pixel), anonymized debate, adaptive early-stop, severity-gated CI exit codes, secure-by-default sandboxing, run budget/retries, large-diff filtering + semantic chunking, an optional result cache, incremental review, interactive patch applier (jury apply), static analysis hint injection (--hints), frugal model tiering (--tiered), official GitHub Action and pre-commit hooks, comment-command triggering, a local / open-weight adapter (free, offline), and jury init scaffolding (with an opt-in --wizard). See docs/architecture.md and the milestones.

The phased plan and how to pick up a session's worth of work is in ROADMAP.md; issues are tracked under milestones.

Security & the Codex sandbox

The jury performs read-only review orchestration — it sends a diff to each agent CLI and collects their feedback; it does not apply edits.

The Codex adapter pipes the prompt on stdin (codex exec with no positional prompt) so non-interactive runs never hang waiting for input, and defaults extra_args to ["-s", "read-only"] — a secure-by-default sandbox. The diff is fetched by the jury (gh), not by codex, so the reviewer only needs to read its prompt and print findings; a prompt injection in the diff can't make it write files, run shell, or reach the network. The agy agent runs under --sandbox, and claude under a write-tool denylist, for the same reason.

Need codex to write or reach the network for your flow? Widen extra_args for the codex agent in jury.toml (e.g. -s workspace-write). See docs/security.md for details.

CLI compatibility contract

The jury command is this project's public API. The surfaces below are stable and are locked by tests/test_cli_contract.py (including a width/color-pinned snapshot of jury --help under tests/golden/) so accidental changes are caught in review.

Stable flags (names, short aliases, and semantics), grouped by intent:

• Input: --pr, --issue, --repo, --diff-file, --commit, --commits
• Depth: --rounds, --max-rounds, --early-stop / --no-early-stop, --auto / --no-auto, --verify / --no-verify, --effort {low,medium,high}
• Budget / reliability: --total-timeout, --phase-timeout, --retries
• Large diffs: --max-diff-bytes, --chunk / --no-chunk, --exclude, --include
• Context & privacy: --context-mode {diff-only,expanded}, --redact / --no-redact
• Config / policy: --config, --policy, --chair, --seed, --mock, --strict, --config-validate, --strict-config
• Output: -o / --output, --write, --metadata-json, --format {markdown,json,sarif,keel-reviews}, --decision {chair,vote}, --transcript / --no-transcript, --verbose, --live, -q / --quiet
• GitHub posting: --post-summary / --post, --post-inline, --post-progress, --post-mode {single,phased}, --dry-run, --label
• CI gating: --ci, --fail-on, --min-vendors, --no-min-vendors, --min-reviews
• Cache & incremental: --cache, --clear-cache, --cache-dir, --incremental
• Patches: --suggest-patches, --patches-out
• Misc: --doctor, --json (with --doctor), --version, -h / --help

A version-independent test (test_documented_flags_match_parser_exactly) asserts the documented flag set and the parser's actual long options stay exactly in sync in both directions, so a new flag can't be added without being documented and a documented flag can't silently disappear.

Stable error messages and exit codes:

Stable report headings (substrings other tooling may parse): AI Jury, Chair verdict, Round 1 (and subsequent Round N).

Policy: Any breaking change to the surfaces above — renaming or removing a flag, changing an error message or exit code, or altering a report heading — requires a CHANGELOG.md entry describing the break. When the change is intentional, regenerate the help snapshot with UPDATE_GOLDEN=1 PYTHONPATH=src python3 -m unittest tests.test_cli_contract. The help-snapshot exact match is pinned to Python 3.13 argparse formatting; the flag-presence checks run on all supported versions (3.11–3.14).

Documentation

• Parameter reference — every CLI flag and jury.toml key with allowed values and defaults.
• Configuration behaviour & validation — validation, budgets, adaptive rounds, large-diff handling, and the cache.
• Positioning — mission, what makes it different, principles, and non-goals.
• Cookbook — copy-paste recipes for local, PR, advisory-comment, CI, mock, and assistant workflows.
• Architecture — components, round structure, adapters, supported platforms.
• Ecosystem comparison — capability matrix vs hosted / API-level / native-CLI tools.
• Feasibility & prior art — research grounding and verified CLI invocations.
• Installing into an agent — per-agent plugin install and update, for Claude Code, Codex, Antigravity and Cursor.
• Platform support matrix — where you can install/run the jury and how.
• Skill packaging & install — install/version the review jury as a reusable skill artifact.
• Release readiness checklist — the bar before a public release.
• The Homebrew release chain — how a release reaches brew install, every guard on the way, and what has already gone wrong.
• Report format contract — the golden-file snapshot tests and how to regenerate them.
• SECURITY.md — data-flow and secret-redaction reference.
• Agent-readable: llms.txt (concise) and llms-full.txt (full reference).

License

MIT — see LICENSE.