Skip to main content

yama

English | 简体中文

yama is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls the LLM through LiteLLM, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the design doc (Chinese) for the full Case schema and execution contract.

Quick start

# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
cd dingding-simple
OPENAI_API_KEY=... uv run --project yama yama

# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple

Instead of prefixing every command, you can put credentials in a .env file: on startup the CLI loads the nearest .env found from the working directory upward (variables already set in the environment are not overridden).

Writing a Case

A Case is a YAML file organized as context (what the model sees) → mocks (how tool calls get executed) → steps (user turns sent in order, each with assertions) → outcome (the pass bar for the whole Case):

description: The model reads the DSL before proposing creative directions

context:
  system_prompt: { default: true }     # use SYSTEM.md from the plugin root

  tools:
    - file: tools/read-dsl.yaml         # tool schema, resolved relative to __evals__

mocks:
  tools:
    read_dsl:
      respond:
        result:
          visualStyle: { name: Vintage Film }

steps:
  - id: request-directions
    user: Give me a few creative directions
    assert:
      hard:
        - tool_called: { name: read_dsl }
        - assistant_contains: creative direction

outcome:
  require: {}
  • description (optional) is a one-line summary of what the Case tests; it is shown in the CLI output and the HTML report.
  • context.tools is the tool schema sent to the model; mocks.tools is how the runner responds when a tool call arrives. The two sets of tool names must match exactly.
  • To use a Skill, declare it by name under context.skills and also declare {builtin: skill} under context.tools; the model reads the Skill body via skill(name, file?, call_reason?).
  • To simulate command-line tools, declare {builtin: bash} and configure per-command output under mocks.cli.
  • steps[].assert.hard are deterministic checks (nine types, including tool_called, tool_arguments, and assistant_contains); untagged hard checks must all pass by default. Add assert.judge for LLM scoring.

For the full schema (every Skill/tool/mock form, the bash sandbox, message injection, judge configuration, and more), see the design doc (Chinese).

CLI usage

uv run --project yama yama --plugin dingding-simple --report
Flag Meaning
paths (positional, repeatable) Explicit Case YAML paths to run
--plugin-root PATH Use the given path as the single plugin root
--plugin NAME (repeatable) Select plugins by name from yama.toml
--all-plugins Run every plugin configured in yama.toml
-k, --filter PATTERN (repeatable) Only run cases whose case key (path relative to the plugin root) matches — a substring, or an fnmatch glob when the pattern contains * ? [; with several patterns a case runs if any matches
--result-dir PATH Override the artifact root (default <plugin_root>/.yama/runs)
--concurrency N Maximum number of cases running in parallel (default 10; 1 runs cases one at a time)
--report [PATH] Also generate a single-file HTML report. Without PATH it writes a timestamped .yama/reports/report-<ts>.html (history accumulates) and keeps .yama/reports/latest.html as a symlink to the newest one; an explicit PATH writes exactly that file
--preview Don't run anything (no LLM calls): resolve the collected Cases and write an HTML case preview (default .yama/reports/preview.html; --report PATH overrides), or print a JSON preview to stdout when combined with --json
--list Only print the collected Cases, without running them
--no-artifacts Write no artifact files at all
--json Emit machine-readable JSON instead of Rich tables

--plugin, --all-plugins, and --plugin-root are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing yama.toml and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.

Dashboard

uv run --project yama yama dashboard --plugin dingding-simple

yama dashboard starts a local HTTP server (default http://127.0.0.1:8765/, change with --host/--port; it also accepts the same plugin selection flags as the main command plus -k/--filter and --result-dir). The index page lists the historical HTML reports accumulated under .yama/reports/ (newest first, with the latest one flagged), /reports/<name> serves each report, and /preview renders the same case preview as --preview — re-collecting and re-resolving the case files on every refresh (collection errors render as an error page), so you can edit a Case YAML and just reload the browser. No LLM calls are ever made by the dashboard.

PR evals

uv run --project yama yama pr-plan --base origin/main --json
OPENAI_API_KEY=... uv run --project yama yama pr-run --base origin/main --report --json-summary yama-pr-summary.json

yama pr-plan selects PR-scoped eval cases without model calls. yama pr-run executes the same selection and writes a JSON summary for external CI jobs to render or upload; credentials stay in environment variables or .env, not CLI flags. pr-run always forces builtin bash execution to the Linux linux-bubblewrap sandbox, regardless of Case backend settings, so the runner must be Linux with bwrap/bubblewrap installed. See the CI integration guide for workflow examples and the summary JSON contract.

Output

Cases run in parallel, up to 10 at a time by default (--concurrency changes the cap); runs within one case stay sequential, and results keep collection order regardless of completion order. While several cases run concurrently, a transient dashboard at the bottom of the terminal shows one row per case, updated in place — waiting, then a spinner with the run/step currently executing, then its ✓/✗ verdict — and each case's full run/step detail is printed above the dashboard as one block the moment it finishes, so blocks of different cases never interleave. With --concurrency 1 (or a single collected case) it streams instead — a spinner marks the step currently executing (on a terminal), and each case, run, and assertion result is printed the moment it completes. Either way a per-case summary table closes the output, and with --json the progress stream goes to stderr so stdout stays valid JSON.

Metadata

Release files for python-yama 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for python-yama 0.8.0
File Size Uploaded Provenance Actions
python_yama-0.8.0.tar.gz 415.4 kB Without provenance Details for python_yama-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for python-yama 0.8.0
File Interpreter ABI Platform Actions
python_yama-0.8.0-py3-none-any.whl Python 3 none any Details for python_yama-0.8.0-py3-none-any.whl

Total release size: 514.4 kB

Release files / python_yama-0.8.0.tar.gz

Download URL python_yama-0.8.0.tar.gz
Size 415.4 kB
Tags Source
SHA-256 checksum
How to use checksums
2bc1f798ad73570e8b77de02d7edb5ed3d539e0684b9b0465d9c469e32bb8c5e
BLAKE2b-256 checksum
How to use checksums
052d1933d8259457779a1ed4d498e50371a573eb572ac7fc09e717cf3b3635b8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / python_yama-0.8.0-py3-none-any.whl

Download URL python_yama-0.8.0-py3-none-any.whl
Size 99.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
cc6d045a9f01216aedbde0d0e2aee88d18fe8b69e88ed9219dac00aac5b4f000
BLAKE2b-256 checksum
How to use checksums
bd3264f96568910b956109dfdf56b547d0749a75ca815d59512a8edae057c649
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.12.0 {"installer":{"name":"uv","version":"0.12.0","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for python-yama 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page