Kyoko

Kyoko is a fully local system for measuring, debugging, and improving AI agents.

Add telemetry, run your agent, and Kyoko shows where performance breaks across runs. It groups recurring failures into evidence-backed issues, lets Codex or Claude Code draft fixes, and only applies changes after checks and evals pass.

It is built around the manual workflow developers already use: inspect traces, understand the failure, patch the prompt, context, or harness, rerun evals, and decide what ships. Kyoko makes that workflow repeatable while keeping you in control.

Everything stays local by default: traces, issues, proposals, evals, database, and dashboard. For analysis and fix drafting, Kyoko can use the coding-agent CLI you already have, like Codex or Claude Code, so there is no separate Kyoko model API key or hosted service.

Why Kyoko

• Finds the failures that repeat across runs. Kyoko looks across runs, groups recurring problems into evidence-backed issues, and shows where each one happened.
• Turns issues into fixes. Accepted issues become proposed changes to your agent context, skills, or harness.
• Measures whether fixes worked. Kyoko reruns failing traces, runs deterministic checks, and compares eval results before applying a fix.
• Keeps the developer in control. Review every issue, proposal, and apply decision manually, or automate only the parts that pass the gate.
• Uses the tools you already have. Codex, Claude Code, OpenClaw, Hermes, or a generic command can analyze evidence and draft fixes through existing CLI auth.
• Runs locally by default. SQLite, loopback dashboard, local traces, local proposals, and explicit external calls.
• Connects to real agent stacks. OTLP/GenAI, Python and TypeScript SDKs, importers, JSON CLI, dashboard, and MCP.

The loop

        ┌─────────────────┐           ┌─────────────────┐
        │  1. Analyse     │ ───────▶ │  2. Issues      │
        │  traces in      │           │  recurring      │
        │                 │           │  failures       │
        └─────────────────┘           └─────────────────┘
                 ▲                            │
                 │ measure                    │ accept
                 │                            ▼
        ┌─────────────────┐  ┌──────┐ ┌─────────────────┐
        │  4. Evals       │◀─┤ gate ├─│  3. Proposals   │
        │  failure rate   │  └──────┘ │  fixes          │
        │                 │   apply   │                 │
        └─────────────────┘           └─────────────────┘

Kyoko keeps the repair loop explicit. Every step creates something you can inspect in the dashboard or CLI.

1. Analyse: Kyoko reads real traces from your agent and looks across runs for repeated behavior: tool mistakes, missing context, policy drift, brittle routing, bad handoffs, or eval failures.
2. Issues: recurring failures become evidence-backed issues with category, severity, occurrence count, and links to the spans where they happened.
3. Proposals: accepted issues become concrete fixes to your agent context, skills, evals, or harness. The fix stays reviewable before it can apply.
4. Evals: Kyoko reruns failing traces, runs deterministic checks, and compares eval results so the gate can decide whether the fix worked.

The gate is the control point. It applies a fix only when checks, replay evidence, autonomy policy, and human locks allow it.

Run it your way. The same loop, the same gate. You pick the autonomy level:

• Human-in-the-loop: Kyoko surfaces issues and drafts fixes, and you review and approve each change before it applies.
• Fully autonomous: the policy auto-applies any change that clears replay, evals, and human locks, and parks anything that doesn't for you to look at.

Quick demo

Try Kyoko without wiring up an agent. The demo creates a local database, loads bundled fixture runs, and serves the dashboard.

pipx install kyoko
kyoko demo --db /tmp/kyoko-demo.db --json
kyoko serve --db /tmp/kyoko-demo.db

Open http://127.0.0.1:8765.

Requires Python 3.12 or newer. No live model, framework adapter, or replay server is needed for the demo.

Get started

From the root of your agent project (needs Python 3.12+):

pipx install kyoko
kyoko project-bootstrap
kyoko serve

Open http://127.0.0.1:8765. pip install kyoko and uv tool install kyoko work too; see docs/INSTALL.md.

Bootstrap writes a local .kyoko/ workspace: database, scaffolds, MCP config, and operator presets. Every later kyoko command finds that database automatically, so no --db flags are needed inside your project.

Then wire up telemetry. This is the step that makes everything else work: Kyoko can only find and fix what it can see. The easiest way is to let your coding agent do the wiring:

kyoko install-skill   # then run /kyoko-instrument in your coding agent

This installs the bundled /kyoko-instrument skill into .claude/skills/, where Claude Code picks it up automatically; for Codex, Cursor, or other agents, kyoko install-skill --print prints the same playbook to paste in. The skill finds your agent's entry point, records one real run, and verifies it shows up in Kyoko.

To connect your agent over MCP instead, or to wire telemetry by hand (Python or TypeScript SDK, OTLP, importers), see Getting Started.

What you get

• Run capture: Python SDK, TypeScript SDK, generated source adapters, OTLP/GenAI JSON, Hermes import, and OpenClaw import.
• Issue queue: recurring failures grouped into evidence-backed issues with category, severity, occurrence count, and span links.
• Fix proposals: accepted issues become validated LearningProposal records for context, skills, evals, or harness changes.
• Verification: bounded replay, deterministic checks, and eval comparison before a fix can apply.
• Operator path: Codex, Claude, OpenClaw, Hermes, or a generic command can analyze evidence and draft fixes through existing CLI auth.
• Control surfaces: local dashboard, JSON-everywhere CLI, and stdio MCP server, all sharing the same gated apply path.

See docs/INTEGRATIONS.md and examples/README.md.

The gate and local boundary

Every behavior-changing path (operator output, imports, MCP tools, and kyoko improve) flows through one gate:

1. Validate the structured proposal.
2. Resolve the evidence it references.
3. Generate or select checks.
4. Run bounded replay and deterministic checks.
5. Evaluate the autonomy policy.
6. Enforce human locks on protected targets.
7. Apply context or harness changes only if the gate allows it.

Operator agents can analyze evidence and draft fixes; they do not directly mutate Kyoko state. Context writes update Kyoko-managed skills and delivery rules. Harness writes create reviewable patch transactions against an explicit workspace root.

Replay server URLs are loopback-only unless you pass --allow-remote-server. Evidence exported to prompts, MCP, API, or bundles is redacted by default. See docs/SECURITY.md and docs/ARCHITECTURE.md.

Documentation

• Getting Started: demo, project bootstrap, telemetry, inspection, and the repair loop.
• Install: install paths, verification, data location, and common setup fixes.
• Integrations: source adapters, replay adapters, operator agents, MCP, and SDKs.
• CLI Reference: grouped command reference.
• Architecture: runtime model, data model, and the gate.
• Security: local data, loopback serving, tokens, redaction, and write boundaries.
• Scope: what v0 is and is not.
• Development: tests, dashboard bundle, release smoke, and contract artifacts.

Specs, schemas, fixtures, and design decisions live under docs/ as reference contracts.

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for local setup, the test and validation gates, and how to submit a change. To report a security vulnerability, follow SECURITY.md rather than opening a public issue.

Repository layout

kyoko/              Python import package, CLI runtime, dashboard/API, bundled assets
frontend/           React/Vite dashboard source
sdk/typescript/     Dependency-free TypeScript telemetry SDK
examples/           Source and replay hook examples
scripts/            Installer, release smoke, fixture and artifact helpers
tests/              Python unittest suite and CLI contract tests
docs/               User docs plus specs, schemas, fixtures, and decisions

License

Apache-2.0. See LICENSE.