Agent Skill Evals

skill-eval tests an Agent Skill as a versioned artifact. It runs the same cases with and without the skill, records what the agent did, grades the evidence, measures skill lift and constraint coverage, runs adversarial probes, and applies release gates.

The system is offline-first. Its schemas, validation, local and custom runners, deterministic graders, coverage analysis, reports, gates, SQLite registry, and portfolio summary do not depend on a hosted service. Microsoft Waza is the default agent runner when it is installed; local and custom command adapters are also available.

Install for development

You need Python 3.11 or newer and uv.

uv sync --locked --all-groups
uv run skill-eval --version

All examples below use uv run skill-eval. After installing the project as a tool, you can use skill-eval directly.

To exercise an unpublished checkout from another repository, use uvx without installing it globally:

uvx --from /path/to/agent-skill-evals skill-eval --version

After a release is published, pin the package rather than relying on a moving checkout:

uvx --from 'agent-skill-evals==0.1.0rc4' skill-eval --version

The release candidate is configured as the MIT-licensed agent-skill-evals distribution, authored by Andrew Brookins and intended for publication from the abrookins PyPI account. The Python import remains skill_eval, and the installed command remains skill-eval.

To execute the default runner path, install Waza v0.38.2 and make waza available on PATH. The adapter checks the exact version before every run. See the pinned integration contract for the verified distributions, capabilities, exit codes, and known gaps. Waza and any model credentials are separate from this package.

The evaluation loop

An eval suite lives beside its skill:

my-skill/
├── SKILL.md
├── skill.yaml                  # optional structured skill metadata
└── evals/
    ├── eval.yaml               # canonical cases, conditions, and graders
    ├── gates.yaml              # release thresholds and waivers
    ├── runner.yaml             # waza, local, or custom execution
    ├── synthesis.yaml          # draft-generation policy
    ├── fixtures/               # inputs copied into isolated workspaces
    ├── graders/                # schemas and grader programs
    ├── probes/                 # adversarial probe catalog
    ├── generated/              # extracted constraints and reviewable drafts
    ├── runs/                   # immutable normalized run evidence
    └── reports/                # JSON, HTML, Markdown, and JUnit output

The core loop is:

SKILL.md -> analyze -> author or synthesize -> validate -> run -> grade
         -> coverage -> report -> gate -> registry and portfolio

Cases can run under four counterfactual conditions:

• no_skill: the target skill is unavailable.
• with_skill: the target skill can be retrieved normally.
• forced_skill: the harness requires the target skill and checks invocation evidence.
• with_distractors: the target appears beside unrelated skills.

Comparing the same case and trial across conditions shows whether the skill helps, has no effect, or makes the result worse. A result is not release-ready merely because it passes with the skill: gates can also require positive lift, correct invocation, broad behavior coverage, stable trials, bounded cost and latency, and no critical security failure.

Conditions that need another skill tree take explicit, domain-neutral source options. Repeat --distractor, --malicious-distractor, or --library-skill for multiple sources; use --previous-skill and --stale-skill for their respective version conditions. For example:

uv run skill-eval run ../my-skill --condition with_distractors \
  --distractor ../unrelated-skill --distractor ../another-skill

Add evals to a skill in under 15 minutes

Start with a skill that already has SKILL.md:

uv run skill-eval init ../my-skill
uv run skill-eval analyze ../my-skill

init is additive and never overwrites existing files. analyze writes editable constraints, trigger examples, and a security-boundary inventory under evals/generated/ while preserving source locations.

Next, add one high-value case to evals/eval.yaml. Use the generated constraints to give the case a stable ID, an observable outcome, explicit conditions, and deterministic graders. Copy needed input files into evals/fixtures/ and refer to them with relative paths. A good first suite has at least:

• one normal positive case;
• one edge or malformed-input case;
• one negative trigger that must not invoke the skill;
• one case for each critical security boundary;
• no_skill and with_skill conditions on utility cases.

The complete sample at examples/skills/csv-summary-skill includes normal, missing-value, injection, distractor, and prose-negative cases with paired conditions, fixtures, JSON Schema grading, tool restrictions, and release gates.

Validate before spending model time:

uv run skill-eval validate ../my-skill
uv run skill-eval security-probes ../my-skill

Then run the suite and use the printed run and grade IDs in the remaining commands:

uv run skill-eval run ../my-skill --conditions no_skill,with_skill --trials 3
uv run skill-eval grade ../my-skill --run-id RUN_ID
uv run skill-eval coverage ../my-skill --run-id RUN_ID --grade-id GRADE_ID
uv run skill-eval report ../my-skill --run-id RUN_ID --grade-id GRADE_ID
uv run skill-eval gate ../my-skill --run-id RUN_ID

Open evals/reports/RUN_ID/report-REPORT_ID/index.html to review prompts, outputs, artifacts, formal grades, evidence, previous-run comparisons, and human feedback. The report and gate commands print the exact versioned path. Keep the JSON and JUnit files as CI artifacts.

Optional synthesis and description optimization

The deterministic analyzer can turn extracted constraints into draft case plans. Drafts remain separate from eval.yaml until a person accepts and promotes them:

uv run skill-eval synthesize ../my-skill
uv run skill-eval review ../my-skill --reviewer "$USER"
uv run skill-eval promote ../my-skill --by "$USER"

--seed FILE imports a canonical seed file and --waza-task FILE imports an existing Waza task; both options may be repeated. Review decisions and promotions are audited. Regenerating analysis or synthesis never overwrites reviewed artifacts.

Description optimization runs repeated held-out trigger trials through Waza. It reports a candidate but cannot change SKILL.md until a reviewer approves the exact evaluated text:

uv run skill-eval optimize-description ../my-skill --model MODEL --trials 5
uv run skill-eval approve-description ../my-skill CANDIDATE_ID --reviewer "$USER"
uv run skill-eval promote-description ../my-skill CANDIDATE_ID --by "$USER"

Runners

evals/runner.yaml selects one runner. skill-eval run --runner TYPE and --model MODEL can override the file for one run.

Waza: the default

New suites use this configuration:

schema_version: 1
runner:
  type: waza
  command: waza
  model: null
  env: {}
  timeout_seconds: 600
  max_tokens: 200000
  capture:
    stdout: true
    stderr: true
    filesystem_diff: true
    tool_calls: true
    artifacts: true
    trace: true

Set model in this file or pass --model. The adapter translates representable cases to Waza schema 1.2, invokes the exact pinned binary without a shell, and normalizes Waza results into the canonical trace format. It uses Waza's native no-skill baseline and delegates compatible graders. Forced-skill checks, distractor semantics, project-specific security boundaries, coverage, waivers, and multidimensional gates remain canonical.

Local command

Use type: local for an agent command that reads the prompt from standard input and writes its final response to standard output:

schema_version: 1
runner:
  type: local
  command: my-agent --non-interactive
  model: local-model
  env: {}
  timeout_seconds: 600
  max_tokens: 200000
  capture:
    stdout: true
    stderr: true
    filesystem_diff: true
    tool_calls: false
    artifacts: true
    trace: true

The command runs directly, not through a shell. It receives AGENT_MODE, AGENT_RUN_ID, AGENT_CASE_ID, AGENT_SKILL_ID, AGENT_MODEL, AGENT_WORKSPACE, AGENT_SKILLS_DIR, AGENT_SKILL_LIBRARY_DIRS, AGENT_INSTRUCTION_FILES, AGENT_FORCE_SKILL, and AGENT_RETRIEVAL_TARGET. The built-in local adapter cannot capture tool calls, token use, or cost, so disable gates that require unavailable evidence or use a custom adapter.

Custom command

Use type: custom when a command can emit normalized trace JSON. The built-in json_stdout parser accepts one JSON object containing final_output plus optional turn, tool, cost, token, skill-event, and case-failure fields:

schema_version: 1
runner:
  type: custom
  command: my-agent-adapter
  model: custom-model
  env: {}
  timeout_seconds: 600
  max_tokens: 200000
  parser:
    name: json_stdout
    options: {}

Custom commands receive the same standard input, workspace, and environment contract as local commands. An unknown parser or malformed output is a runner error, never a case failure.

Evidence-specific gates

Release checks are enabled by default. When a runner cannot produce meaningful evidence for a specific metric, disable only that check in evals/gates.yaml; the gate report retains it as an auditable waived check rather than silently dropping it:

release_gates:
  utility:
    check_distractor_degradation: false
  invocation:
    check_positive_invocation_rate: false
    check_false_trigger_rate: false
  efficiency:
    check_cost_delta: false
    check_latency_delta: true

Keep checks enabled when the runner supplies real evidence. These switches describe a stable runner capability; use expiring, owner-attributed waivers for temporary exceptions to an available check.

Security runs

Security probes have their own run lineage and do not inflate utility scores:

uv run skill-eval security-probes ../my-skill
uv run skill-eval security-run ../my-skill --condition with_skill --trials 3
uv run skill-eval gate ../my-skill --run-id RUN_ID --security-run-id SECURITY_RUN_ID

Use --probe PROBE-1,PROBE-2 to select probes. Critical boundary failures, forbidden tools, secret exfiltration, and configured probe thresholds can block release.

CI and exit codes

Run these checks before merging changes to an eval suite:

uv run skill-eval validate ../my-skill
uv run skill-eval run ../my-skill --conditions no_skill,with_skill
uv run skill-eval grade ../my-skill --run-id RUN_ID
uv run skill-eval coverage ../my-skill --run-id RUN_ID
uv run skill-eval report ../my-skill --run-id RUN_ID
uv run skill-eval gate ../my-skill --run-id RUN_ID

The CLI uses stable process exit codes:

Code	Meaning
0	Pass
1	Eval or release-gate failure
2	Warning-only gate outcome
3	Invalid suite or configuration
4	Runner or environment error

Do not retry code 1 as an infrastructure failure. Code 4 means the evidence is incomplete because execution did not finish reliably. Gate waivers must name an owner and reason, have a bounded validity window, and match the intended run or status. Expired, revoked, mismatched, or missing waivers do not suppress failures and remain visible in JSON, HTML, and JUnit output.

This repository's CI workflow runs linting, formatting, strict type checking, tests with coverage, packaging, and the full end-to-end path with network access disabled. It publishes JUnit and report artifacts even when a gate fails.

Offline behavior and optional dependencies

The following operations are deterministic and can run without network access once dependencies and the selected local executable are installed: initialization, analysis, synthesis planning, review and promotion, suite validation, local/custom execution, canonical grading, coverage, reporting, gates, registry migrations, and portfolio summaries.

These operations need an external capability and fail explicitly when it is unavailable:

• Waza execution needs the pinned Waza binary; Copilot-backed execution needs its model access.
• Description optimization currently needs runner.type: waza and a model.
• A structured LLM rubric needs a configured provider or compatible Waza prompt grader.
• Waza installation, model downloads, and dependency installation are not offline operations.

There is no silent hosted fallback. A missing external runner produces exit code 4; invalid or unavailable optional configuration produces an actionable error.

Command reference

Every command takes a required SKILL directory unless noted. Comma-separated selectors reject blank values. Run uv run skill-eval COMMAND --help for generated usage text.

Command	Purpose	Command-specific arguments and options
init	Add an eval-suite scaffold without overwriting files.	No options.
analyze	Extract constraints, trigger prompts, and security boundaries.	--regenerate writes a content-addressed candidate set while preserving reviewed files.
synthesize	Create reviewable draft cases without changing eval.yaml.	Repeatable --seed PATH; repeatable --waza-task PATH.
review	Interactively accept, edit, reject, skip, inspect, or regenerate drafts.	Required --reviewer TEXT; optional --case ID.
promote	Add accepted drafts to the canonical suite.	Required --by TEXT; optional --case ID.
optimize-description	Compare descriptions on repeated held-out trigger trials.	--trials N (at least 2, default 3); --model TEXT; --executor TEXT (default copilot-sdk); repeatable --candidate TEXT.
approve-description	Approve one exact evaluated description.	Required CANDIDATE_ID argument and --reviewer TEXT.
promote-description	Write an approved description to SKILL.md.	Required CANDIDATE_ID argument and --by TEXT.
security-probes	Generate or validate the deterministic probe catalog.	No options.
security-run	Execute probes in isolated security lineage.	--probe ID,ID; --conditions NAME,NAME or --condition NAME; --trials N (default 1); --runner TEXT; --model TEXT; --run-id TEXT; the same condition-source options as run.
validate	Validate schemas, IDs, selectors, graders, fixtures, constraints, and gates.	No options.
run	Execute selected cases, conditions, and trials.	--conditions NAME,NAME or --condition NAME; --case ID,ID; --tag TAG,TAG; --trials N; --runner TEXT; --model TEXT; --output-dir PATH; --previous-skill PATH; --stale-skill PATH; repeatable --distractor PATH, --malicious-distractor PATH, and --library-skill PATH.
grade	Grade a persisted run.	Required --run-id TEXT; --canonical; --evidence-dir PATH.
coverage	Compute observed behavior-constraint coverage.	Required --run-id TEXT; --grade-id TEXT; --evidence-dir PATH.
report	Write versioned JSON, self-contained HTML, and JUnit.	Required --run-id TEXT; --grade-id TEXT; --evidence-dir PATH; --previous-run-id TEXT; --feedback-file PATH.
gate	Evaluate release thresholds and return a stable exit code.	--run-id TEXT or --report PATH; --security-run-id TEXT.

Global options are --version, --install-completion, --show-completion, and --help.

Project quality checks

uv run ruff check .
uv run ruff format --check .
uv run mypy src tests
uv run pytest --cov=skill_eval --cov-report=term-missing
uv build
uvx --from mfcqi mfcqi-py analyze --skip-llm src

The CSV Summary demo is the smallest complete example. The automated end-to-end test exercises analysis, synthesis, human approval, description optimization, a contract-faithful Waza run, paired lift, grading, coverage, reporting, gate failure, bounded waivers, registry persistence, and portfolio classification.