codejoust 0.2.0

A CLI arena for AI coding agents. Throw one bug at Claude Code, Codex, aider — let them race, auto-score, and pick the winner.

CodeJoust

Pit AI coding agents against the same bug. Score them on tests, diff size, cost, and time. Merge the winner.

中文 → README_CN.md

---

Same model, different harness. Independent testing found Claude Sonnet scored 77% through Claude Code but 93% through Cursor on the same benchmark — a 15-point gap that is pure tooling, not model quality. Which "AI coding assistant" is right for your task is not a model question, it's a task-level empirical question.

CodeJoust answers it. One CLI command fires the same task at Claude Code, aider, (soon) Codex, Cursor CLI, Gemini CLI in parallel — each in its own git worktree — then auto-grades them and hands you the winning patch.

Why not just open three terminals?

That's what most people do. It's also why most people never actually benchmark their tools — running three agents, waiting, eyeballing three diffs, manually tallying tokens is work, so they pick one and stick with it.

CodeJoust takes that hour down to one command:

codejoust run "fix the off-by-one in Scheduler.next_fire" \
  --agents claude-code,aider,codex --test "pytest tests/test_scheduler.py"

You get:

• A side-by-side terminal table ranked by test pass-rate → cost → diff size → latency
• A single-file HTML report with each agent's full diff
• One .patch file per agent — apply the winner with git apply

Install

pip install codejoust

Versions. The current PyPI release (codejoust 0.1.0) ships Claude Code and aider adapters only. The Codex and Gemini adapters live on main and fold into the next PyPI bump. To preview them today:

pip install "git+https://github.com/he-yufeng/CodeJoust"

You'll also need whichever agent CLIs you want to race. Install as many or as few as you like:

# Claude Code
npm install -g @anthropic-ai/claude-code

# aider
pip install aider-chat

# OpenAI Codex CLI
npm install -g @openai/codex

# Google Gemini CLI
brew install gemini-cli  # or: npm install -g @google/gemini-cli

Set the usual API keys in your environment (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.) — CodeJoust just shells out to each CLI, so whatever auth setup you already use keeps working.

Quickstart

cd ~/code/my-project

codejoust run "add a --dry-run flag to the deploy command"

You'll see:

─────────── CodeJoust — 2 agents ────────────
task:   add a --dry-run flag to the deploy command
repo:   /Users/you/code/my-project
agents: claude-code, aider

                  CodeJoust — add a --dry-run flag...
  #  agent         status    diff        tests    cost      time
★ 1  claude-code   success   +38/-2      8/8      $0.028    71.3s
  2  aider         success   +21/-1      7/8      $0.019    55.7s

winner: claude-code
  — git log and merge via: cat .codejoust/runs/.../claude-code.patch | git apply

report: /Users/you/code/my-project/.codejoust/runs/20260424-222310/report.html

All artifacts live in .codejoust/runs/<timestamp>/:

file	what
report.html	single-file HTML, side-by-side diffs, shareable
session.json	structured run data (for scripting and CI)
<agent>.patch	the agent's changes, apply with git apply
logs/<agent>/*.log	raw stdout/stderr from each CLI

How it scores

Four signals, in order:

1. Test pass ratio (tests_passed / tests_total). Auto-detects pytest or npm test; override with --test.
2. Cost (USD). Lower is better. Pulled from each CLI's own usage output.
3. Diff size (added + removed lines). Smaller is better — more conservative changes are usually safer.
4. Wall time. Tie-breaker.

The first signal where one agent strictly beats another decides the winner. This is intentional: if Claude Code passes 8/8 tests and aider passes 7/8, we don't care that aider was cheaper — test correctness dominates.

LLM-as-judge scoring (subjective code quality) is planned for v0.2 behind --judge; for now the scoring is fully objective so it's reproducible and cheap.

Agents

codejoust agents
#   claude-code    cli: claude
#   aider          cli: aider
#   codex          cli: codex
#   gemini         cli: gemini

v0.2 ships with Claude Code, aider, OpenAI Codex CLI, and Google Gemini CLI. Still on the roadmap:

• Cursor CLI (cursor-agent) — flagged experimental until the CLI stabilises
• OpenHands (openhands --headless)

Each adapter is ~50 lines. PRs welcome.

CLI reference

codejoust run TASK [OPTIONS]

  -a, --agents TEXT      comma-separated list. default: claude-code,aider
  -r, --repo DIR         repo root. default: current directory
      --timeout INT      per-agent timeout in seconds. default: 600
      --test CMD         test command. default: auto-detect pytest / npm test
      --model NAME       optional model override passed to every agent
      --keep-worktrees   don't clean up worktrees afterwards
      --html / --no-html write report.html. default: on
      --open             open the report in your browser when done

vs. other tools

Project	What it does	What it doesn't
CodeJoust (this)	CLI, parallel agents, auto-score (tests+cost+diff), HTML report	LLM-as-judge is Phase 2
Claude Squad (7k★)	tmux + worktree session manager	no scoring, no diff comparison — manual
parallel-code (544★)	parallel runs + diff viewer	no auto-score, no test integration
Cursor 3 /best-of-n	best-of-N inside Cursor	closed source, IDE-only, paid, no external CLIs
GitHub Agent HQ	multi-agent on GitHub Cloud	closed source, paid, cloud only
Terminal-Bench / SWE-bench	fixed benchmark evaluation	can't throw your own issue at it
CodeClash	LLM tournaments on fixed arenas (BattleSnake, Poker)	not for your codebase

The empty cell CodeJoust fills: open-source, CLI-first, real agents (not just models), auto-scored, on your own repo.

FAQ

Does it need internet / a credit card? Only what the underlying agent CLIs need. CodeJoust itself is offline.

What about cost blowup? Each agent runs once per task. Default --timeout 600 caps wall time. Tokens and USD are reported per run; you'll see exactly what each task cost.

Can I run more than two agents at once? Yes: --agents claude-code,aider,codex. Each runs in its own worktree, fully isolated. Watch your API rate limits.

Does it work on Windows? Tested on macOS and Linux. Windows should work inside WSL; native Windows has not been tested and is unlikely to work cleanly because the agent CLIs themselves are Unix-first.

What if I don't have tests? CodeJoust still ranks by cost, diff size, and wall time. But the really useful ranking signal is test pass-rate — if you can write a single failing test for your bug first, you'll get much better picks.

How do I stop the project from churning through my API quota? Start with --timeout 120 for small tasks. Each agent is independently rate-limited by its own API key. CodeJoust makes no network calls itself.

Roadmap

• v0.1.0: Claude Code + aider, objective scoring, HTML report.
• v0.1.1: OpenAI Codex CLI adapter — three-way races out of the box.
• v0.2.0 (now): Google Gemini CLI adapter; four agents on the same task.
• v0.3: --judge for LLM-as-judge tie-breaking, YAML config for reusable agent profiles, Cursor + OpenHands adapters, batch mode, Markdown PR export.
• later: server mode for team/CI use, public arena leaderboard.

Kill criteria: if claude-squad or parallel-code ship built-in auto-scoring, CodeJoust repositions as the lightweight standalone scorer and deprecates its orchestration layer.

Contributing

Adapters are the main contribution surface. See src/codejoust/adapters.py — each adapter is a subclass of AgentAdapter with build_command() and parse_usage(). Open a PR with your agent of choice.

License

MIT. See LICENSE.