Kibitzer

The person watching your chess game who can't help offering opinions.

Kibitzer is a Claude Code extension that watches how agents use tools and suggests better alternatives. It applies mode-based path protection to write tools (canonicalized, fail-closed) and to the Bash write vectors it can see statically, intercepts bash commands that have structured alternatives, and coaches agents toward more effective tool usage — all without an LLM in the decision loop. It is a guidance layer that composes under the Claude Code permission system and an OS-level sandbox, not a replacement for either (see Scope and limits).

Install

pip install kibitzer
cd your-project/
kibitzer init --hooks --mcp

This registers PreToolUse/PostToolUse hooks in .claude/settings.json and starts an MCP server with two tools the agent can call: ChangeToolMode and GetFeedback.

For richer coaching with Fledgling conversation analytics:

pip install kibitzer[fledgling]

What it does

Path protection

Each mode defines which paths the agent can write to. The path guard checks every Edit, Write, and NotebookEdit call, and statically scans Bash commands for common write vectors (redirections, tee, cp, mv, rm, sed -i, ln, ...).

Before comparing, the guard canonicalizes the target against the project directory — symlinks are followed, ./.. are collapsed, absolute and relative spellings of the same location compare equal — and matches by path segment, so src/ never matches src_secret/. The guard fails closed: an unknown mode, a missing target path, an unparseable Bash command in a restricted mode, or an internal guard error denies the write.

Mode        Writable                  Use case
─────────── ───────────────────────── ───────────────────────────
free        everything                prototyping, floor only
implement   src/, lib/, tests/, test/ normal dev (tests included)
test        tests/, test/, spec/      writing tests — source protected
docs        docs/, README.md          documentation only
explore     nothing                   read-only investigation
review      nothing                   read-only code review

Writable entries may also be absolute or ~-prefixed glob patterns, expanded at check time — the implement/test/docs defaults include ~/.claude/projects/*/memory/ so Claude auto-memory writes don't require a mode flip.

Independent of the mode, a safety floor deny-list runs first on every write (including statically extracted Bash write targets): secret dirs (~/.ssh, ~/.aws, ...), system paths, raw .git internals, and writes outside any git working tree are blocked even in free mode. It's configurable via [floor] in .kibitzer/config.toml and fails open on its own internal errors.

When a write is denied, the agent sees why and how to fix it:

Path 'notebooks/explore.ipynb' is not writable in the current mode (writable: ['src/', 'lib/', 'tests/', 'test/', '~/.claude/projects/*/memory/']).
Use the ChangeToolMode tool to switch modes.

In testing, agents consistently read this message and call ChangeToolMode to switch — no documentation or pre-training needed.

Scope and limits

Path protection is one layer, not a sandbox:

• Bash coverage is static. The guard analyzes the command string; it is not a shell interpreter. Writes hidden behind interpreter one-liners (python -c ...), scripts fed on stdin, make targets, or git commands that mutate the tree are not visible to it. Opaque write constructs it can spot — xargs rm, process substitution targets, unparseable quoting — are denied in restricted modes rather than waved through.
• Check-then-act window. Symlinks are resolved when the call is checked; a race that swaps a path component between the check and the actual write is not preventable at this layer.
• free mode skips the mode guard by design (writable = ["*"]) and is the default; only the mode-independent safety floor still applies. Set a restricted default_mode in .kibitzer/config.toml if you want the full guard active from the first call.

For a hard boundary, compose kibitzer with the Claude Code permission system and an OS-level sandbox (e.g. an umwelt-compiled policy under nsjail). Kibitzer keeps a cooperating-but-fallible agent on task; the sandbox bounds what a misbehaving process can do.

Interception

Interceptor plugins watch Bash calls for commands that have structured alternatives:

Three interception modes form a ratchet — start in observe (log silently), graduate to suggest (show alternative), then redirect (deny bash, require structured tool). Each graduation is a one-line config change.

Coaching

The coach fires every N tool calls and detects patterns from ~250 experimental runs. Suggestions only reference tools the agent actually has — discovered from .mcp.json at runtime.

State-based patterns (always available):

• Repeated edit failures — "Edit failed 3 times on src/handler.py. Try Read() first to see exact content."
• Edit streak without tests — "You've made 7 edits without running tests." (mentions blq run test if blq is available)
• Sequential file reads — "You've read 5 files one at a time." (mentions FindDefinitions if fledgling is available)
• Bash-heavy usage — "You've run 6 bash commands without using structured tools."
• Analysis loop — "You've spent 18 turns reading without changes. Start with the most confident fix."
• Semantic tool underuse — "FindDefinitions shows all functions in one call." (only fires if fledgling is available)
• Mode oscillation — "Frequent mode switches. Consider using free mode."

TDD patterns:

• Test overfit — "test_auth.py has been edited 4 times. Stabilize test expectations before adjusting further."
• Implement before test — "You edited source before writing tests. Consider starting with a failing test."

Fledgling-powered patterns (when fledgling is installed):

• Repeated search patterns — "You've searched for 'def handle_request' 4 times via Grep."
• Replaceable bash commands — "You've run 'grep' 3 times. FindDefinitions provides structured output."

All patterns are mode-aware: the analysis loop doesn't fire in explore mode (not editing is correct there), edit-without-test doesn't fire in docs mode (docs don't need tests).

Auto-transitions

The mode controller watches for failure patterns:

• 3+ consecutive failures → auto-switch to explore
• 20+ turns in explore → auto-switch back to implement

An oscillation guard prevents rapid switching: if the agent just left a mode (< 5 turns), it won't auto-switch back. After 6+ total switches, auto-transitions stop.

Composes with Claude Code's permission model

The Claude Code harness has its own permission system (per-tool allow/deny, prompting before sensitive actions). Kibitzer's modes layer on top of these permissions — not beside them. The two regulate at different granularities:

• Harness permissions: coarse, universal, set at install — "can this agent ever call Bash?"
• Kibitzer modes: fine, contextual, changed per task — "is Bash appropriate given the current mode?"

They compose via set intersection: a tool call succeeds iff the harness permits it AND (if kibitzer is active) the current mode permits it. Kibitzer can narrow the harness's permissions; it can never widen them.

When a call is denied, the response identifies which layer denied it so the agent can respond correctly — switching modes (kibitzer) or asking the user to grant permission (harness) are different remedies.

MCP tools

The agent can call two tools explicitly:

ChangeToolMode(mode, reason?) — Switch modes. Returns the new mode's writable paths and strategy.

GetFeedback(status?, suggestions?, intercepts?) — Check current status, get coaching suggestions, and see which bash commands have been intercepted.

Configuration

Override defaults in .kibitzer/config.toml:

# Monorepo: widen writable paths
[modes.implement]
writable = ["packages/core/src/", "packages/api/src/"]

# Add custom modes
[modes.deploy]
writable = ["infra/", "deploy/"]
strategy = "Verify before applying."

# Graduate jetsam to suggest mode
[plugins.jetsam]
mode = "suggest"

# More aggressive coaching
[coach]
frequency = 3

Python API

Use kibitzer from Python without the hook protocol:

from kibitzer import KibitzerSession

with KibitzerSession(project_dir=".") as session:
    # Check if a tool call is allowed
    result = session.before_call("Edit", {"file_path": "src/auth.py"})

    # Record a completed tool call
    session.after_call("Edit", {"file_path": "src/auth.py"}, success=True)

    # Batch-validate a program's planned calls
    violations = session.validate_calls([
        {"tool": "Edit", "input": {"file_path": "tests/foo.py"}},
    ])

Full API docs at kibitzer.readthedocs.io/python-api.

Coordinates with

Kibitzer suggests but never wraps these tools — each is independent:

• blq — structured build/test capture
• jetsam — git workflow acceleration
• Fledgling — AST-aware code intelligence

None are required. Kibitzer degrades gracefully — path guard and coach work with nothing else installed. When tools are available, suggestions reference them specifically. When they're not, suggestions give generic advice.

Documentation

Full docs at kibitzer.readthedocs.io:

• Modes — path protection, switching, auto-transitions
• Coach — all patterns, experimental evidence, model dependency
• Interceptors — the observe/suggest/redirect ratchet
• Configuration — full config.toml reference, resilience, optional deps
• Architecture — how the pieces fit together
• Integration — blq, jetsam, fledgling, superpowers

License

MIT