KaizenAI

Kaizen (改善): continuous, incremental improvement. KaizenAI is an autonomous agent that finds failing tests, proposes the smallest fix, and opens a gated pull request for human review — it never commits directly. Built so it can be pointed at any project without changes to the framework itself.

How it works

1. Runner agent connects to a target project — via shell commands and/or MCP (the open protocol both agents use to access a project's tools) — and executes that project's eval suite: a small set of representative tasks with clear pass/fail criteria.
2. Every result (pass or fail) is written to a structured trace store (JSONL), including which prompt/tool-schema version produced it.
3. Fixer agent reads failing traces and proposes the smallest possible patch — preferring prompts/config/tool descriptions over application code when that's the root cause.
4. Every patch is scoped into a risk tier (safe text/config vs. application code vs. never-touch) and opened as a pull request — never committed directly. A human always reviews and merges.

Runner ──runs evals──▶ Target project (via MCP / shell)
   │
   ▼
Trace store (structured failures)
   │
   ▼
Fixer ──proposes patch──▶ git branch ──▶ PR ──▶ Human review ──▶ merge

Quickstart (5 minutes, no target project of your own needed)

This repo ships examples/sample_python_project/ — a tiny, self-contained project with one deliberately broken function — so you can watch the Runner catch a real failure with zero setup:

git clone https://github.com/sajjikazemi/kaizen-ai.git
cd kaizen-ai
poetry install

poetry run kaizen run --config examples/sample_python_project/kaizen.yaml

That runs pytest against the example project and reports one failing case — subtract() has a bug (see calculator.py). No API key needed for this step.

To see the Fixer actually propose a patch, you'll need an LLM key (cp .env.example .env, fill in ANTHROPIC_API_KEY/OPENAI_API_KEY, or log into the GitHub Copilot CLI — see below) and then:

poetry run kaizen fix --config examples/sample_python_project/kaizen.yaml

Heads up: since the example lives inside this repo rather than a separate one, kaizen fix checks out a real kaizen/fix/... branch in your kaizen-ai clone (that's how it behaves against any target repo). Skip --open-pr to just inspect the diff, or try this against a scratch clone if you'd rather not touch your working branch.

Adding a new target project

1. Add a kaizen.yaml to the target repo and pick an adapter — dotnet (see examples/sample_dotnet_project/kaizen.yaml) or pytest (see examples/sample_python_project/kaizen.yaml). Add a new language by implementing ProjectAdapter (src/kaizen/adapters/base.py) and registering it in src/kaizen/adapters/__init__.py.
2. Add an evals/ directory with one *.eval.yaml file per case, tagged by feature area (see either example's evals/).
3. Run:

   kaizen baseline --config path/to/kaizen.yaml      # cache ground-truth (run on a cadence, e.g. nightly)
   kaizen run --config path/to/kaizen.yaml           # find today's failures
   kaizen fix --config path/to/kaizen.yaml --open-pr # propose, evaluate, and PR each fix
   

Statistically-gated regression safety

Fixing failures and verifying nothing else broke are two different problems — at 200+ eval cases you can't afford to re-run everything on every candidate, nor can you trust a 5-question spot check. Kaizen uses a paired treatment/control evaluation (kaizen evaluate), the same methodology used for online A/B testing and NLP model regression testing:

• Treatment split = the failing case(s) + any case sharing a tag (feature area) with them.
• Control split = a stratified random sample of everything else, rotated across cycles so full coverage accumulates over time instead of being paid for on every run.
• Baseline vs. candidate results on both splits are compared with McNemar's test (the statistically correct test for paired binary before/after outcomes), not a plain percentage diff.
• Decision rule: significant gain on treatment + flat control → fix (open PR). Gain on treatment + significant loss on control → trade (PR opened but flagged for explicit human judgment). No significant treatment gain → rejected (no PR; Fixer iterates again).

See docs/README.md for the supporting papers (McNemar 1947; Dietterich 1998; Dror et al. 2018; Ribeiro et al. 2020; Sculley et al. 2015; Kohavi et al. 2009; Rothermel & Harrold 1996; Yoo & Harman 2012; Cochran 1977).

Risk tiers

Agent runtime: two swappable backends

Runner and Fixer each delegate to a backend, picked per project via model.backend in kaizen.yaml — nothing else in the codebase needs to change to switch:

Both live behind a thin dispatcher (runner/mcp_task.py, fixer/dispatch.py) so FixerAgent/RunnerAgent, tiering, evaluation, and the PR gate are identical either way.

• Runner connects to the target's MCP server (via langchain-mcp-adapters for the langgraph backend, or Copilot CLI's own --additional-mcp-config for copilot_cli) and drives task_prompt eval cases.
• Fixer edits files via Read/Write/Edit/Bash. Tier-3 enforcement differs slightly by backend:
  • langgraph: hard-blocked in the tool itself (fixer/tools.py), plus Tier-2 writes pause for approval via HumanInTheLoopMiddleware.
  • copilot_cli: the CLI has no per-tool-call hook, so Tier-3 is enforced post-hoc — after the session, any Tier-3 file in the diff is hard-reverted (fixer/git_tier_guard.py) before anything reaches the PR. Either way, Tier-2/3 changes always still require human review at the PR.

Where do the reports/tables actually show up?

• CLI (kaizen run / kaizen fix / kaizen evaluate) — pretty rich tables, including the treatment/control McNemar report. This is the primary place to read results.
• Pull request body — the same verdict/tier summary, as markdown, so reviewers see it without touching a terminal.
• LangGraph Studio (see below) is not a reporting dashboard — it shows live graph execution/state (useful for watching/approving an in-flight Tier-2 edit), not a rendered evaluation table. The raw report data is attached to the graph's state so it's inspectable there too, but the table itself lives in the CLI and the PR.

Running LangGraph Studio

Note: Studio visualizes the langgraph backend's Fixer graph specifically — there's no graph to visualize when model.backend: copilot_cli is active (it's a plain subprocess call), so Studio is only useful if you've switched a project to langgraph.

cp .env.example .env        # fill in ANTHROPIC_API_KEY / OPENAI_API_KEY / LANGSMITH_API_KEY etc.
export KAIZEN_CONFIG=/path/to/your/project/kaizen.yaml   # optional, defaults to the bundled example
poetry run langgraph dev

This opens LangGraph Studio against the Fixer graph (kaizen.graphs:fixer_graph, declared in langgraph.json) — you can send it a prompt, watch each tool call, and approve/reject Tier-2 code edits visually instead of (or in addition to) the CLI's y/N prompt.

LANGSMITH_API_KEY (in .env) is optional but recommended: Studio works without it, but you won't see the "Trace"/run-history tab populate. Free to create at smith.langchain.com → Settings → API Keys. Without it, Studio shows a "LangSmith API key is missing" banner — harmless, just no trace history.

Prerequisite for the copilot_cli backend

Install the GitHub Copilot CLI and run copilot /login once on the machine running Kaizen. No .env/API key needed for this backend — it uses that login session.

Development

poetry install
poetry run pytest

Status

Core pieces (eval schema, trace store, .NET/pytest adapters, tiering, PR gate, treatment/control evaluation, LangGraph-based Runner/Fixer agents, Studio wiring) are implemented and tested. Try the bundled quickstart above, or point it at a real target project with API keys configured.