kensa 0.8.0

The open source agent evals harness

Kensa is the open source harness for evaluating agents.

---

Agents are non-deterministic. Prompts drift. Tools change. Models behave differently. Any change can make them slower, more expensive, or just plain unreliable.

kensa gives coding agents, like Claude Code, a repeatable loop to eval your agents, and catch regressions every time you make a change.

Installation

Run these from the root of your Python agent repo.

Paste this into your coding agent

Open your coding agent and paste:

Run `uvx kensa init --cli --agent all`, then use the audit-evals skill and
follow the eval lifecycle.

The agent installs the CLI, scaffolds .kensa/, drops in the five skills, and runs your first eval. Works with Claude Code, Codex, Cursor, OpenCode, and Gemini CLI.

Or install yourself, then ask your agent

If you want to control the install step but still let your coding agent drive the eval workflow:

uvx kensa init --cli --agent all

Then in Claude Code, Codex, Cursor, OpenCode, or Gemini CLI:

> /audit-evals

The skill captures a real run, generates scenarios, runs evals, and reports back.

Or CLI-only

If you want to skip the coding-agent loop entirely and drive kensa as a regular CLI:

uvx kensa init                                       # dev dep + bare .kensa/
kensa capture -i "<example input>" -- <your agent>   # record one real run as a trace
kensa generate                                       # synthesize scenarios from the capture
kensa eval                                           # run + judge + report

Or Claude Code plugin

If you primarily use Claude Code, install via the plugin marketplace:

/plugin marketplace add satyaborg/kensa
/plugin install kensa

Quickstart

Tell your coding agent what you want:

You say	Kensa does
"Evaluate this agent"	Audit setup, create or reuse scenarios, and run evals.
"Why are evals failing?"	Inspect results and traces, then diagnose the root cause.
"Add coverage for tool use"	Write scenario YAML with tool or trajectory checks.
"The judge seems wrong"	Create or validate structured judge prompts.

How it works

• Zero to eval: your coding agent drafts scenarios; you review them.
• Runs become traces: each scenario runs in a subprocess with LLM calls, tool use, tokens, cost, and latency captured.
• Checks gate judges: deterministic checks run before any LLM judge call.
• Ship with evidence: reports show verdicts, traces, cost, latency, and failure details.

Instrumentation

Zero code changes. kensa captures LLM calls, tool use, tokens, cost, and latency without modifying your agent. OpenTelemetry (OTel) compatible.

Provider extras

uv add "kensa[anthropic]"
uv add "kensa[openai]"
uv add "kensa[langchain]"
uv add "kensa[all]"

Core commands

Command	What it does
kensa init	Scaffold .kensa/ (bare; pass --example for a demo agent + scenario)
kensa capture -i "<input>" -- <cmd>	Record one real agent run as a trace
kensa doctor	Check instrumentation, config, and environment readiness
kensa generate	Synthesize scenario YAMLs from captured traces via an LLM
kensa eval	Run + judge + report in one command
kensa report	Show the latest results in terminal, Markdown, JSON, or HTML

See the CLI docs for run, judge, analyze, mcp, and the full command reference.

MCP server

One-liner for Claude Code (run from your project root):

claude mcp add kensa -- uvx kensa-mcp

For other JSON-based MCP clients, add to your project's .mcp.json or .cursor/mcp.json:

{
  "mcpServers": {
    "kensa": {
      "command": "uvx",
      "args": ["kensa-mcp"]
    }
  }
}

For Codex, add to your project-scoped .codex/config.toml:

[mcp_servers.kensa]
command = "uvx"
args = ["kensa-mcp"]

See the MCP server docs for tools, resources, and manual config.

Manual workflow

If you want to author evals yourself:

kensa init
kensa doctor

Scenarios live in .kensa/scenarios/*.yaml and point at your agent entrypoint with run_command.

id: classify_ticket
input: "Our entire team can't log in. SSO has returned 502 since 7am."
run_command: [python, agent.py]   # input is appended as the final argv element

checks:
  - type: trajectory
    params:
      steps:
        - tool: classify_ticket
      max_steps: 1
      max_tokens: 2000
  - type: output_matches
    params: { pattern: "^P[123]$" }

criteria: |
  P1 is for outages or data loss affecting multiple users.

For complete examples, see examples/. See the scenario docs and checks docs for the full field and check reference.

CI

- name: Run evals
  run: uv run kensa eval --format markdown

If you only use deterministic checks, you do not need API keys. If you use criteria or judge, add judge LLM provider secrets in CI.

Need more?

• Docs
• examples/ has sample agents and scenarios
• CONTRIBUTING.md covers local development
• Homepage

License

• MIT License