yama
English | 简体中文
yama is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls an OpenAI-compatible chat completions endpoint, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the design doc (Chinese) for the full Case schema and execution contract.
Quick start
# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
cd dingding-simple
YAMA_LLM_API_KEY=... uv run --project yama yama
# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
YAMA_LLM_API_KEY=... uv run --project yama yama --plugin dingding-simple
Instead of prefixing every command, you can put credentials in a .env file: on startup the CLI loads the nearest .env found from the working directory upward (variables already set in the environment are not overridden). The default endpoint can be overridden with YAMA_LLM_BASE_URL.
Judge routing
Evaluated Agent calls always use the configured OpenAI-compatible API. In an ordinary non-CI run, Judge routing recognizes OpenAI/GPT/o-family names as Codex, Claude/Anthropic names as Claude Code, and Grok/xAI names as Cursor. An exact per-agent map may translate an API model name to a local CLI model slug. Without a mapping, every recognized family first passes the original Judge model name to its local Agent. If any local attempt that explicitly names a model fails, Yama retries the same Agent with its default model; if that default attempt also fails, Yama calls the original Judge API. Other families become local-first only through an exact mapping. This local-first chain never enters the fallback Agent.
[defaults.judge.local.models.codex]
"openai/gpt-5.4" = "gpt-5.4-high"
[defaults.judge.local.models.claude]
"anthropic/claude-opus-4.1" = "claude-opus-4-1"
[defaults.judge.local.models.cursor]
"x-ai/grok-4" = "grok-4-fast"
"kimi.k3" = "kimi-k3"
[defaults.judge.local.agent]
name = "claude" # "claude", "codex", or "cursor"
model = "claude-sonnet-4-5"
An unmapped, unrecognized model is API-first. Only after its original API exhausts retryable failures may [defaults.judge.local.agent] run as the terminal fallback: configured local model, then the same Agent's default whenever that named-model attempt fails, then end. There is no fallback-model API call. External cancellation always stops the chain immediately. The HTML report shows the complete Judge attempt chain, including mapped/configured/default/fallback source, retries, failures, and the effective local model when the Agent reports it.
CI and pr-run are API-only. Local agents receive the mapped model slug or original Judge model name and a compatible reasoning_effort, but never temperature, credentials, or endpoint settings. Cursor Judge passes --trust only for its per-attempt empty temporary workspace so non-interactive execution cannot stop at Workspace Trust; it still omits --force/--yolo, and Yama does not edit Cursor's global trust configuration. Evaluated Agent API attempts default to a 180-second transport timeout; Judge API attempts and local Judge processes default to 120 seconds. The whole-Step timeout remains independently configured by execution.timeout_seconds_per_step (default 300 seconds).
Writing a Case
A Case is a YAML file organized as context (what the model sees) → mocks (how tool calls get executed) → steps (user turns sent in order, each with assertions) → outcome (the pass bar for the whole Case):
description: The model reads the DSL before proposing creative directions
context:
system_prompt: { default: true } # use SYSTEM.md from the plugin root
tools:
- file: tools/read-dsl.yaml # tool schema, resolved relative to __evals__
mocks:
tools:
read_dsl:
respond:
result:
visualStyle: { name: Vintage Film }
steps:
- id: request-directions
user: Give me a few creative directions
assert:
hard:
- tool_called: { name: read_dsl }
- assistant_contains: creative direction
outcome:
require: {}
description(optional) is a one-line summary of what the Case tests; it is shown in the CLI output and the HTML report.context.toolsis the tool schema sent to the model;mocks.toolsis how the runner responds when a tool call arrives. The two sets of tool names must match exactly.- To use a Skill, declare it by name under
context.skillsand also declare{builtin: skill}undercontext.tools; the model reads the Skill body viaskill(name, file?, call_reason?). - Skills resolve from
<plugin-root>/skills/<name>/SKILL.mdfirst. If that local Skill is absent, Yama uses the sibling.common-skills/<name>/versions/...tree only when<plugin-root>/config.jsonexplicitly declares the name inskill_refs; an undeclared missing Skill is a collection error. - To simulate command-line tools, declare
{builtin: bash}and configure per-command output undermocks.cli. steps[].assert.hardare deterministic checks (nine types, includingtool_called,tool_arguments, andassistant_contains); untagged hard checks must all pass by default. Addassert.judgefor LLM scoring.
For the full schema (every Skill/tool/mock form, the bash sandbox, message injection, judge configuration, and more), see the design doc (Chinese).
CLI usage
uv run --project yama yama --plugin dingding-simple --report
| Flag | Meaning |
|---|---|
paths (positional, repeatable) |
Explicit Case YAML paths to run |
--plugin-root PATH |
Use the given path as the single plugin root |
--plugin NAME (repeatable) |
Select plugins by name from yama.toml |
--all-plugins |
Run every plugin configured in yama.toml |
-k, --filter PATTERN (repeatable) |
Only run cases whose case key (path relative to the plugin root) matches — a substring, or an fnmatch glob when the pattern contains * ? [; with several patterns a case runs if any matches |
--result-dir PATH |
Override the artifact root (default <plugin_root>/.yama/runs) |
--concurrency N |
Maximum number of cases running in parallel (default 10; 1 runs cases one at a time) |
--run-concurrency N |
Maximum number of runs active across all cases (default 32) |
--report [PATH] |
Also generate a single-file HTML report. Without PATH it writes a timestamped .yama/reports/report-<ts>.html (history accumulates) and keeps .yama/reports/latest.html as a symlink to the newest one; an explicit PATH writes exactly that file |
--preview |
Don't run anything (no LLM calls): resolve the collected Cases and write an HTML case preview (default .yama/reports/preview.html; --report PATH overrides), or print a JSON preview to stdout when combined with --json |
--list |
Only print the collected Cases, without running them |
--no-artifacts |
Write no artifact files at all |
--json |
Emit machine-readable JSON instead of Rich tables |
pr-run --upload-report --commit-id SHA |
With --report, upload the push report and merge/upload affected Plugin latest reports; requires ADMIN_HOST and YAMA_EVAL_REPORT_API_TOKEN |
--plugin, --all-plugins, and --plugin-root are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing yama.toml and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.
Dashboard
uv run --project yama yama dashboard --plugin dingding-simple
yama dashboard starts a local HTTP server (default http://127.0.0.1:8765/, change with --host/--port; it also accepts the same plugin selection flags as the main command plus -k/--filter and --result-dir). The index page lists the historical HTML reports accumulated under .yama/reports/ (newest first, with the latest one flagged), /reports/<name> serves each report, and /preview renders the same case preview as --preview — re-collecting and re-resolving the case files on every refresh (collection errors render as an error page), so you can edit a Case YAML and just reload the browser. No LLM calls are ever made by the dashboard.
PR evals
uv run --project yama yama pr-plan --base origin/main --json
YAMA_LLM_API_KEY=... uv run --project yama yama pr-run --base origin/main --report --json-summary yama-pr-summary.json
yama pr-plan selects PR-scoped eval cases without model calls. yama pr-run executes the same selection and writes a JSON summary for external CI jobs to render or upload; both evaluated-Agent and Judge calls are API-only, and credentials stay in environment variables or .env, not CLI flags. pr-run always forces builtin bash execution to the Linux linux-bubblewrap sandbox, regardless of Case backend settings, so the runner must be Linux with bwrap/bubblewrap installed. See the CI integration guide for workflow examples and the summary JSON contract.
For CI environments that publish reports to the Yama Eval Report service, add
--upload-report --commit-id <sha> together with --report. The command reads
ADMIN_HOST and YAMA_EVAL_REPORT_API_TOKEN from the environment, uploads the
current commit report, then updates each affected Plugin latest report by merging
the just-run cases into the previous remote Plugin report.
Output
Cases run in parallel, up to 10 at a time by default (--concurrency changes the cap). Runs produced by one Case's execution.repeat also run in parallel: omitted execution.repeat_concurrency means all expanded runs, subject to the command-wide --run-concurrency cap (default 32); set it to 1 to restore serial runs for that Case. Case results keep collection order and run results keep run_index order regardless of completion order. Whenever Cases or runs overlap, a transient dashboard at the bottom of the terminal shows one row per Case and each completed Case's full run/step detail is printed above it as one block, so concurrent output never interleaves. Fully serial execution streams each Case, run, and assertion as it completes. Either way a per-Case summary table closes the output, and with --json the progress stream goes to stderr so stdout stays valid JSON.
Metadata
Release files for python-yama 0.10.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| python_yama-0.10.4.tar.gz | 491.9 kB | Without provenance | Details for python_yama-0.10.4.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| python_yama-0.10.4-py3-none-any.whl | Python 3 | none | any | Details for python_yama-0.10.4-py3-none-any.whl |
Total release size: 620.7 kB
Release files / python_yama-0.10.4.tar.gz
| Download URL | python_yama-0.10.4.tar.gz |
|---|---|
| Size | 491.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9e5f84b27b898c54e964d2b05b998a93be3eeddbfe9f6b7990ec4aa68ba1ca80
|
|
BLAKE2b-256 checksum How to use checksums |
6ed8826097a1dce65210b502c63bccd4ceb51e27bb844adbb3ee768c39ca22ba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / python_yama-0.10.4-py3-none-any.whl
| Download URL | python_yama-0.10.4-py3-none-any.whl |
|---|---|
| Size | 128.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6da3b18e4aa12dd1f9a7a037f0c3d687a59b699435c9cc5a12bfa255edf065a3
|
|
BLAKE2b-256 checksum How to use checksums |
72fe3b4a2deddf4fdda1145e91e0043996a12b2954baf13f11f0e74220767293
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.5 {"installer":{"name":"uv","version":"0.12.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for python-yama 0.10.4
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.