Evaluate and improve AI agent skills and plugins: quality, security, overlap, compatibility, and real impact across agents
📺 Livestream: Ask the Experts: Evaluating Agent Skills
SkillEvaluator checks AI agent skills and plugins, and measures a skill's effect on task results: start with quality checks, then compare runs with and without a skill.
Agent skills are folders of instructions and supporting files defined by the Agent Skills specification. SkillEvaluator is part of the NVIDIA Verified Skills pipeline.
Get started
Install the CLI in a Python 3.12–3.14 environment:
pip install skillevaluator
Or use uv
uv tool install --python 3.13 skillevaluator
For uv setup or direct GitHub source installation, see Installation.
Then check skills from your terminal, CI, or agent:
|
Quickly check your skill without an LLM key: skillevaluator quality-check ./my-skill
After all-tier setup with NVIDIA Build, run the combined pipeline: skillevaluator validate \
./my-skill
|
Our CI template checks every skill change, no code to maintain. Run in your GitHub repo to open a setup PR: skillevaluator ci init
GitLab details: repository guide. |
Add the skill to Claude Code, Codex or another agent: npx skills add \
NVIDIA/SkillEvaluator
Then ask: "Is my skill any good?" See the agent guide. |
⚡ Harness check: will your agent load your skill?
Included in SkillEvaluator 1.0.1; see Installation for package availability. No model key or agent run.
skillevaluator harness-check ./my-skill
Illustrative summary of versioned profiles. Full output can include notes and schema diagnostics.
╭─ Harness compatibility · my-skill ──────────────────────────────────────────────────────╮
│ ⚠️ codex degraded Description cut to 1,021 chars + '...' in the skill list │
│ ❌ claude-code not loaded .agents/skills is not scanned │
│ ✅ opencode loads │
│ ✅ hermes loads │
│ ✅ openclaw loads │
│ ❌ copilot-cli not loaded description is 1,028 UTF-16 units, 4 over the 1,024 limit │
╰────────────────────────────────────────────────────────────────── ❌ 2 of 6 not loaded ─╯
validate in 1.0.1 includes it in Tier 1. 🧪 Ready to go deeper? Try a
live agent evaluation (Tier 3).
Quickstart
Check your skill without a key
skillevaluator quality-check ./my-skill
No API key, scanner, GPU or Docker is needed. It prints a score out of 100 with a
grade and the findings behind it; the passing threshold defaults to 70. This is
static quality only; publication evidence stays INCOMPLETE. Security and agent impact need additional checks. See the
keyless walkthrough.
No skill yet? Clone the repository and use the bundled
src/skillevaluator/tier3/reference_skills/text-analyzer. Scores can differ by release.
Run all three tiers with a free NVIDIA Build key
Eligible NVIDIA Developer Program members can use NVIDIA-hosted endpoints for free prototyping, subject to access terms and rate limits.
- Sign in at build.nvidia.com and complete enrollment.
- Open a model page, select Get API Key, and generate your key.
- Complete scanner setup, start Docker, and prepare a reviewed dataset. Install every tier, set the provider and key, then validate:
pip install "skillevaluator[all]"
export SKILL_EVAL_LLM_PROVIDER=nv_build
export NVIDIA_API_KEY='nvapi-...'
skillevaluator validate ./my-skill
Without --tiers, validate runs all three. The key covers embeddings and chat
in Tier 2 and supported agents in Tier 3; add --llm for model checks in Tier 1.
For uv or GitHub installations, use the matching all-tier install instead of the pip command above. Tier 3 also
needs Docker running; add --tiers 1,2 to skip it. More in
Run the combined pipeline, the
Build walkthrough and provider configuration.
Three-tier overview
Each tier is independent. Use the all-tier installation for all three:
| Tier | Question | Example after setup | Additional setup |
|---|---|---|---|
| Tier 1: Validation | Is the skill safe, well-formed and harness-compatible? | skillevaluator tier1 ./my-skill --no-llm |
External scanners for full static coverage; a provider and model clients enable LLM checks |
| Tier 1: Harness Check | Will the agent load the skill? | skillevaluator harness-check ./my-skill |
SkillEvaluator 1.0.1; no API key or agent run |
| Tier 2: Deduplication | Does guidance repeat or overlap? | skillevaluator tier2 ./my-skill |
Embeddings and chat; add --catalog ./skill-catalog.json for inter-skill comparison |
| Tier 3: Live Agent Evaluation | Does the skill help the agent? | skillevaluator tier3 ./my-skill |
Provider access, a supported agent, and a running sandbox (Docker by default) |
Prepare scanners for your installed release; Semgrep is optional.
--no-llm does not remove scanner requirements; a scanner that cannot run is
INCOMPLETE and exits nonzero.
SkillSpector provides specialized skill
security scanning; Harbor, the
open-source agent evaluation framework, powers sandboxed agent runs.
Tier 2 reports when inter-skill comparison did not run. Tier 3 autopilot reuses sources or creates one skill-grounded case with limited coverage. See tier guides.
Run the combined pipeline
After setting up the provider, scanners, and agent sandbox, run all three tiers:
skillevaluator doctor --env-mode docker
skillevaluator validate ./my-skill
doctor checks readiness, not inference success. validate runs all tiers with
autopilot for skills. Use --tiers 1,2 to skip live evaluation or --tiers 1 for
static checks. --no-autopilot requires an existing source. Other artifact types
retain their applicable stages.
Tier 1 gates validate. Tier 2 gates unless --no-block-on-dedup is set;
Tier 3 gates only with --block-on-agent-eval. Model and managed-sandbox usage
can incur charges; see runtime and cost.
Standalone generation needs substantive evals/EVAL.md guidance, a --prompt
file with task requests and outcomes, or matching retained traces. --force
below replaces the autopilot starter; --refine refines it. Explicit
--grounding skill permits skill-only starters. Review cases; templates omit
unauthored negatives.
skillevaluator tier3 create-eval-dataset ./my-skill --full --prompt eval-guidance.md --force
Run from a source checkout
For an editable environment, follow the development setup.
Research and citation
Resources: paper, PDF, DOI, BibTeX, and NVIDIA Developer blog. Cite the paper and record the SkillEvaluator release or commit; see CITATION.cff.
Documentation
See the documentation for setup, tier guides, reports, CI integration, CLI reference, and performance dashboard.
Installation and third-party software
This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.
Contributing
Read CONTRIBUTING.md and run:
make lint && make test && make build
See project governance and Code of Conduct.
Support
Support level: Experimental. Community-supported, best-effort, with no SLA or NVIDIA enterprise support entitlement. Report bugs and feature requests through GitHub Issues; see SUPPORT.md.
Security
Report suspected vulnerabilities using the private process in SECURITY.md. Do not disclose security issues in a public GitHub issue.
Releases
See CHANGELOG.md and GitHub Releases.
License
Apache License 2.0: see LICENSE, NOTICE, and THIRD_PARTY_NOTICES.md.
Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
Metadata
Release files for skillevaluator 1.0.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| skillevaluator-1.0.1.tar.gz | 3.4 MB | Without provenance | Details for skillevaluator-1.0.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| skillevaluator-1.0.1-py3-none-any.whl | Python 3 | none | any | Details for skillevaluator-1.0.1-py3-none-any.whl |
Total release size: 6.2 MB
Release files / skillevaluator-1.0.1.tar.gz
| Download URL | skillevaluator-1.0.1.tar.gz |
|---|---|
| Size | 3.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3cf6984d86f665e321f55702a0f5f73ca7b900c5e2729769258c063be3de05a6
|
|
BLAKE2b-256 checksum How to use checksums |
ab40f38fda158aac4d7fa2a3f9f7af91a26a7017d68bef0298ff825c8ed3309d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Release files / skillevaluator-1.0.1-py3-none-any.whl
| Download URL | skillevaluator-1.0.1-py3-none-any.whl |
|---|---|
| Size | 2.7 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
a6b8523d4022f63e2e0b0d3f31287f884d58fde4091d6e53a900bc2ed8d2df1d
|
|
BLAKE2b-256 checksum How to use checksums |
4c57671802e1b3da2b43fb0e0deb1a2c104829bcdde136d8e5034d382c12b585
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.14.8
|
Security for skillevaluator 1.0.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.