KUMA

KUMA Python SDK
Knowledge-grounded Universal Measurement for Agents

English  |  Simplified Chinese

KUMA is a framework-neutral Python SDK for evaluating AI Agents. It delivers a test Case as step-by-step Inputs, captures bounded Evidence, and produces a Judgment through official or custom Providers. KUMA does not run your Agent or expose private evaluation logic.

What you can do

• Run repeatable Case-to-Judgment evaluations.
• Use official services or custom Case and Judge Providers, including fully local workflows.
• Capture structured Evidence from results, file changes, logs, and optional OpenTelemetry traces.
• Select a versioned Strategy Group and describe the Agent with an Agent Profile.

Install

Requires Python 3.10 or newer:

python -m pip install --upgrade kuma-defuzex

The package is kuma-defuzex; Python imports and the CLI use kuma.

Quick start

Run a deterministic local check without an account, API key, Docker, or network:

kuma quickstart

Successful output starts with:

Local check: PASS
Score: 100/100
Reason: Output exactly matched the published rule.

This checks the bundled local example, not your Agent or the hosted Judge. To evaluate your Agent, follow the SDK guide.

Next steps

• Run an official evaluation with the full Case and Judge workflow.
• Observe an Agent locally without a Case, Judge, or automatic upload.
• Try the full-stack example with KUMA and mini-SWE-agent in Docker. It may use service credit and model budget.

Documentation

SDK guide · Python API reference · Versions and releases · Detailed Judge assessment

Project

Security · Contributing · Apache License 2.0