Skip to main content

SkillEvaluator

Evaluate and improve AI agent skills and plugins: quality, security, overlap, compatibility, and real impact across agents

License Python PyPI Documentation Paper Blog Stars Forks

📺 Livestream: Ask the Experts: Evaluating Agent Skills

SkillEvaluator checks AI agent skills and plugins, and measures a skill's effect on task results: start with quality checks, then compare runs with and without a skill.

Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.

Get started

Install the CLI in a Python 3.12–3.14 environment:

pip install skillevaluator
Or use uv
uv tool install --python 3.13 skillevaluator

For uv setup or direct GitHub source installation, see Installation.

Then check skills from your terminal, CI, or agent:

Run the CLI directly

Quickly check your skill without an LLM key:

skillevaluator quality-check ./my-skill

After all-tier setup with NVIDIA Build, run the combined pipeline:

skillevaluator validate \
  ./my-skill

Add it to your repo via CI

Our CI template checks every skill change, no code to maintain. Run in your GitHub repo to open a setup PR:

skillevaluator ci init

GitLab details: repository guide.

Wire it to your agent

Add the skill to Claude Code, Codex or another agent:

npx skills add \
  NVIDIA/SkillEvaluator

Then ask: "Is my skill any good?" See the agent guide.

⚡ Harness check: will your agent load your skill?

Included in SkillEvaluator 1.0.1; see Installation for package availability. No model key or agent run.

skillevaluator harness-check ./my-skill

Illustrative summary of versioned profiles. Full output can include notes and schema diagnostics.

╭─ Harness compatibility · my-skill ──────────────────────────────────────────────────────╮
│  ⚠️ codex        degraded    Description cut to 1,021 chars + '...' in the skill list   │
│  ❌ claude-code  not loaded  .agents/skills is not scanned                              │
│  ✅ opencode     loads                                                                  │
│  ✅ hermes       loads                                                                  │
│  ✅ openclaw     loads                                                                  │
│  ❌ copilot-cli  not loaded  description is 1,028 UTF-16 units, 4 over the 1,024 limit  │
╰────────────────────────────────────────────────────────────────── ❌ 2 of 6 not loaded ─╯

validate in 1.0.1 includes it in Tier 1. 🧪 Ready to go deeper? Try a live agent evaluation (Tier 3).

Quickstart

Check your skill without a key

skillevaluator quality-check ./my-skill

No API key, scanner, GPU or Docker is needed. It prints a score out of 100 with a grade and the findings behind it; the passing threshold defaults to 70. This is static quality only; publication evidence stays INCOMPLETE. Security and agent impact need additional checks. See the keyless walkthrough.

No skill yet? Clone the repository and use the bundled src/skillevaluator/tier3/reference_skills/text-analyzer. Scores can differ by release.

Run all three tiers with a free NVIDIA Build key

Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.

  1. Sign in at build.nvidia.com and complete enrollment.
  2. Open a model page, select Get API Key, and generate your key.
  3. Complete scanner setup, start Docker, and prepare a reviewed dataset. Install every tier, set the provider and key, then validate:
pip install "skillevaluator[all]"
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
skillevaluator validate ./my-skill

Without --tiers, validate runs all three. The key covers embeddings and chat in Tier 2 and supported agents in Tier 3; add --llm for model checks in Tier 1. For uv or GitHub installations, use the matching all-tier install instead of the pip command above. Tier 3 also needs Docker running; add --tiers 1,2 to skip it. More in Run the combined pipeline, the Build walkthrough and provider configuration.

Three-tier overview

Three-tier pipeline: skill, validation, deduplication, live evaluation, reports

Each tier is independent. Use the all-tier installation for all three:

Tier Question Example after setup Additional setup
Tier 1: Validation Is the skill safe, well-formed and harness-compatible? skillevaluator tier1 ./my-skill --no-llm External scanners for full static coverage; a provider and model clients enable LLM checks
Tier 1: Harness Check Will the agent load the skill? skillevaluator harness-check ./my-skill SkillEvaluator 1.0.1; no API key or agent run
Tier 2: Deduplication Does guidance repeat or overlap? skillevaluator tier2 ./my-skill Embeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison
Tier 3: Live Agent Evaluation Does the skill help the agent? skillevaluator tier3 ./my-skill Provider access, a supported agent, and a running sandbox (Docker by default)

Prepare scanners for your installed release; Semgrep is optional. --no-llm does not remove scanner requirements; a scanner that cannot run is INCOMPLETE and exits nonzero. SkillSpector provides specialized skill security scanning; Harbor, the open-source agent evaluation framework, powers sandboxed agent runs.

Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. See tier guides.

Run the combined pipeline

After setting up the provider, scanners, and agent sandbox, run all three tiers:

skillevaluator doctor --env-mode docker
skillevaluator validate ./my-skill

doctor checks readiness, not inference success. validate runs all tiers with autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for static checks. --no-autopilot requires an existing source. Other artifact types retain their applicable stages.

Tier 1 gates validate. Tier 2 gates unless --no-block-on-dedup is set; Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage can incur charges; see runtime and cost.

Standalone generation needs substantive evals/EVAL.md guidance, a --prompt file with task requests and outcomes, or matching retained traces. --force below replaces the autopilot starter; --refine refines it. Explicit --grounding skill permits skill-only starters. Review cases; templates omit unauthored negatives.

skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force

Run from a source checkout

For an editable environment, follow the development setup.

Research and citation

Resources: paper, PDF, DOI, BibTeX, and NVIDIA Developer blog. Cite the paper and record the SkillEvaluator release or commit; see CITATION.cff.

Documentation

See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.

Installation and third-party software

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

Contributing

Read CONTRIBUTING.md and run:

make lint && make test && make build

See project governance and Code of Conduct.

Support

Support level: Experimental. Community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement. Report bugs and feature requests through GitHub Issues; see SUPPORT.md.

Security

Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.

Releases

See CHANGELOG.md and GitHub Releases.

License

Apache License 2.0: see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.

Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Metadata

Release files for skillevaluator 1.0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for skillevaluator 1.0.1
File Size Uploaded Provenance Actions
skillevaluator-1.0.1.tar.gz 3.4 MB Without provenance Details for skillevaluator-1.0.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for skillevaluator 1.0.1
File Interpreter ABI Platform Actions
skillevaluator-1.0.1-py3-none-any.whl Python 3 none any Details for skillevaluator-1.0.1-py3-none-any.whl

Total release size: 6.2 MB

Release files / skillevaluator-1.0.1.tar.gz

Download URL skillevaluator-1.0.1.tar.gz
Size 3.4 MB
Tags Source
SHA-256 checksum
How to use checksums
3cf6984d86f665e321f55702a0f5f73ca7b900c5e2729769258c063be3de05a6
BLAKE2b-256 checksum
How to use checksums
ab40f38fda158aac4d7fa2a3f9f7af91a26a7017d68bef0298ff825c8ed3309d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.8

Release files / skillevaluator-1.0.1-py3-none-any.whl

Download URL skillevaluator-1.0.1-py3-none-any.whl
Size 2.7 MB
Tags Python 3
SHA-256 checksum
How to use checksums
a6b8523d4022f63e2e0b0d3f31287f884d58fde4091d6e53a900bc2ed8d2df1d
BLAKE2b-256 checksum
How to use checksums
4c57671802e1b3da2b43fb0e0deb1a2c104829bcdde136d8e5034d382c12b585
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.14.8

Security for skillevaluator 1.0.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page