SkillEvaluator

📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator is an open-source, multi-tier framework for evaluating AI agent artifacts, starting with agent skills: deterministic quality gates, semantic overlap detection, synthetic eval dataset generation, and live agent evaluation.

Agent skills extend AI agents with instructions and supporting files, as defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline. Skills that pass are published to the NVIDIA skills catalog.

Research foundation. Tier 3 implements methods from Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills: paired with/without-skill trials and Skill Lift; SkillEvaluator provides validation, deduplication, dataset authoring, and reporting.

Three-tier overview

Run each tier independently.

Tier	Purpose	Run one tier	Requires
Tier 1: Validation	Safe & well-formed?	skillevaluator tier1 ./my-skill	External scanners for full coverage; a provider enables rubric and LLM security checks
Tier 2: Deduplication	Repeated or overlapping guidance?	skillevaluator tier2 ./my-skill	Embeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison
Tier 3: Live Evaluation	Does it help the agent?	skillevaluator tier3 ./my-skill	Provider access, a supported agent, and a running sandbox (Docker by default)

SkillSpector provides Tier 1 security scanning; Harbor, the open-source agent evaluation framework, powers Tier 3 sandboxed agent runs. See the tier guides.

Quickstart

Install with uv:

uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"

NVIDIA Build: set just two variables. API key:

export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'

This key covers chat, Tier 2 embeddings, and supported Tier 3 agents in Docker or local mode.

For gateways, set SKILL_EVAL_LLM_PROVIDER=openai-compatible, SKILL_EVAL_LLM_BASE_URL, and SKILL_EVAL_LLM_API_KEY. Model defaults are overridable; the gateway must support the selected models and APIs.

Run all tiers on ./my-skill, a directory containing SKILL.md:

skillevaluator validate ./my-skill

Run individual tiers:

skillevaluator tier1 ./my-skill
skillevaluator tier2 ./my-skill
skillevaluator tier3 ./my-skill

• Tier 1 runs static checks, including dependency checks, plus rubric, LLM security analysis, and finding verification when a provider is configured. Without a provider, LLM stages are reported as skipped. Use --no-llm to opt out of LLM checks.
• Tier 2 checks for repeated content inside the skill. Add --catalog ./skill-catalog.json to compare against other skills too; without a catalog, the output explicitly says inter-skill comparison did not run.
• Tier 3 creates one starter evaluation case if no dataset or task source exists, then runs the provider-native agent in Docker with and without the skill. Valid existing sources are reused unchanged; invalid ones produce an error. Review starter cases; they provide limited coverage.

A tier without a path shows help. Expert commands such as tier1 security-scan, tier2 similarity-check, and tier3 create-eval-dataset remain available.

Runtime requirements

[all] installs Python extras; full Tier 1 needs Semgrep, SkillSpector, and Gitleaks, and Tier 3 needs an agent runtime and sandbox. Follow the installation guide, then check readiness:

skillevaluator doctor --env-mode docker

For a keyless check without external scanners, run skillevaluator quality-check ./my-skill. If the command is unavailable, run uv tool update-shell and reopen your terminal.

Other providers and optional model overrides

OpenAI uses two variables and defaults to Codex for Tier 3:

export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'

NVIDIA Build defaults to nvidia/nemotron-3-super-120b-a12b for evaluator chat, Tier 3 agent execution, and judging. Its embedding default is nvidia/nemotron-3-embed-1b; OpenAI defaults to gpt-6.1-sol and text-embedding-3-small. These are configured defaults, not automatically updated selections of the latest models. To select a different evaluator model:

export SKILL_EVAL_LLM_MODEL='your-provider-model-id'

The standard judge inherits that model unless overridden. Tier 3 defaults to OpenCode for NVIDIA Build, Codex for OpenAI, and Claude Code for Anthropic. Use --agents and --agent-model to override the agent and its model separately.

Anthropic and Bedrock need a separate embedding provider for Tier 2. Custom OpenAI-compatible endpoints also require explicit endpoint and model settings. See Providers & Credentials for these setups and advanced overrides.

Run the combined pipeline

validate runs Tier 1, Tier 2, and Tier 3 with autopilot by default for skills:

skillevaluator validate ./my-skill

--full remains supported but is unnecessary. Use --tiers 1,2 or --no-tier3 to skip live evaluation; --tiers 1 runs only the static suite. --no-autopilot requires an existing evaluation source. Rules, workflows, and plugins retain their applicable validation stages without automatically enabling skill evaluation.

Generate and review broader coverage; templates omit negatives unless authored in EVAL.md:

skillevaluator tier3 create-eval-dataset ./my-skill --full

Tier 1 always gates validate. Tier 2 gates by default; --no-block-on-dedup makes its findings advisory. Tier 3 is advisory by default; --block-on-agent-eval makes its findings gate too. Live model calls and managed sandboxes can incur charges; start with one agent and a small dataset. See the Tier 3 guide for runtime and cost details.

Run from a source checkout

git clone https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv sync --python 3.13 --extra all
uv run --extra all skillevaluator tier1 ./my-skill

Use the provider exports above. Prefix other commands with uv run --extra all to select this checkout.

Research and citation

Resources: paper, PDF, DOI, BibTeX, and the NVIDIA Developer blog. Cite the paper for methodology and results. For reproducibility, record the SkillEvaluator release or commit. See CITATION.cff.

Documentation

See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.

Installation and third-party software

Follow the installation guide to choose the full installation or a smaller per-tier setup.

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Contributing

Contributions are welcome. Read CONTRIBUTING.md, include tests for behavior changes, and run the checks before opening a pull request:

make lint && make test && make build

Project governance is described in GOVERNANCE.md. Participation is governed by the Code of Conduct.

Support

Support level: Experimental. SkillEvaluator is community-supported on a best-effort basis with no SLA or NVIDIA enterprise support entitlement. Report reproducible bugs and feature requests through GitHub Issues; see SUPPORT.md for details.

Security

Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.

Releases

Release changes are recorded in CHANGELOG.md and GitHub Releases.

License

Apache License 2.0 — see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.