python-yama 0.2.0

A reproducible evaluation runner for tool-using Agent skills

yama

English | 简体中文

yama is a reproducible evaluation framework for tool-using Agent Skills: a declarative YAML Case describes what the model sees — the system prompt, Skill metadata, tool schemas, and multi-turn user messages — then yama calls the LLM through LiteLLM, drives the tool loop, and scores the transcript with deterministic hard checks or an LLM judge. See the design doc (Chinese) for the full Case schema and execution contract.

Quick start

# Run from a plugin root; collects __evals__/cases/**/*.yaml by default
cd dingding-simple
OPENAI_API_KEY=... uv run --project yama yama

# Or, from a workspace root (a directory with yama.toml), pick a plugin by name
OPENAI_API_KEY=... uv run --project yama yama --plugin dingding-simple

Writing a Case

A Case is a YAML file organized as context (what the model sees) → mocks (how tool calls get executed) → steps (user turns sent in order, each with assertions) → outcome (the pass bar for the whole Case):

context:
  system_prompt: { default: true }     # use SYSTEM.md from the plugin root

  tools:
    - file: tools/read-dsl.yaml         # tool schema, resolved relative to __evals__

mocks:
  tools:
    read_dsl:
      respond:
        result:
          visualStyle: { name: Vintage Film }

steps:
  - id: request-directions
    user: Give me a few creative directions
    assert:
      hard:
        - tool_called: { name: read_dsl }
        - assistant_contains: creative direction

outcome:
  require:
    hard_checks: all_pass

• context.tools is the tool schema sent to the model; mocks.tools is how the runner responds when a tool call arrives. The two sets of tool names must match exactly.
• To use a Skill, declare it by name under context.skills and also declare {builtin: skill} under context.tools; the model reads the Skill body via skill(name, file?).
• To simulate command-line tools, declare {builtin: bash} and configure per-command output under mocks.cli.
• steps[].assert.hard are deterministic checks (nine types, including tool_called, tool_arguments, and assistant_contains); add assert.judge for LLM scoring.

For the full schema (every Skill/tool/mock form, the bash sandbox, message injection, judge configuration, and more), see the design doc (Chinese).

CLI usage

uv run --project yama yama --plugin dingding-simple --report

Flag	Meaning
paths (positional, repeatable)	Explicit Case YAML paths to run
--plugin-root PATH	Use the given path as the single plugin root
--plugin NAME (repeatable)	Select plugins by name from yama.toml
--all-plugins	Run every plugin configured in yama.toml
--result-dir PATH	Override the artifact root (default <plugin_root>/.yama/runs)
--report [PATH]	Also generate a single-file HTML report (default .yama/reports/latest.html)
--list	Only print the collected Cases, without running them
--no-artifacts	Write no artifact files at all
--json	Emit machine-readable JSON instead of Rich tables

--plugin, --all-plugins, and --plugin-root are mutually exclusive. When none of them is given, yama walks up from the current directory to the nearest directory containing yama.toml and uses it as the workspace root (falling back to the cwd), running it as a single plugin root.