harness-eval 5.1.0

Evaluate and compare AI agent setups through experiments, inspections, and rubric scoring.

harness-eval

Evaluate AI code agent setups for best practices, redundancy, security, and cross-component issues.

Available as a CLI tool, a GitHub Action, a Claude Code plugin, and Cursor commands.

Supports Claude Code, Cursor, Copilot, Gemini CLI, and OpenCode projects. Auto-detects which tool(s) a project uses. Also discovers third-party modules installed via package managers.

What it does

Most tools test whether a skill produces correct output. This tool checks the setup itself: CLAUDE.md, GEMINI.md, AGENTS.md, skills, commands, hooks, MCP configs, agents, .cursor/rules/*.mdc, .cursorrules, .github/prompts/, .opencode/.

Five commands, same engine:

Command	What it does	LLM in CLI	LLM in Claude Code / Cursor
lint	64 deterministic rules + system analysis (token budget, trigger overlaps, dependencies). Fast, CI-suitable. Supports --format sarif for GitHub code scanning.	No	No
review	Per-component rubric review with 0-3 scoring per dimension, 21 cross-type checks. KEEP/REVIEW/REMOVE verdicts.	Yes (API key)	Yes (in-session)
security	All security rules + YARA + CVE lookups + semantic review. SAFE/CAUTION/UNSAFE.	Scan: no. Semantic review: --review flag	Yes (in-session)
skill	Deep-evaluate one skill individually and in context of the full setup.	Lint: no. Rubric: --rubric flag	Yes (in-session)
rules	List all available rules with ID, severity, target type, and description. Filter by --category or --target.	No	No

Supported AI Assistants

Auto-detects which tool(s) a project uses and evaluates all discovered components.

Assistant	What it discovers
Claude Code	CLAUDE.md, skills/, commands/, .claude/agents/, .claude/settings.json, .mcp.json
Cursor	.cursor/rules/*.mdc, .cursorrules, .cursor/commands/, .cursor/skills/, .cursor/hooks.json, .cursor/mcp.json
Copilot	.github/skills/, .github/prompts/, .github/agents/
Gemini CLI	GEMINI.md, .gemini/commands/
OpenCode	AGENTS.md, .opencode/commands/, .opencode/agents/
Third-party modules	.lola/modules/ (skills, commands, agents installed via package managers)

Multi-tool projects are fully supported. When a project contains files for multiple assistants, all are discovered, deduplicated, and evaluated together.

Install

See INSTALL.md for all installation options, CI integration, and configuration (Available as a CLI tool, GitHub Action, Claude Code plugin, and Cursor commands)

Inspection Rules (64)

Category	Rules	What they check
Structural	1	SKILL.md exists
Frontmatter	3	Description required/quality, format valid
Content	4	Duplicate detection (TF-IDF), broken references, circular references, token budget
Quality	8	Imprecise instructions, redundant guidance, unfinished content, example gap, stale references, negative-only prohibitions, scope overreach, trigger manipulation
Security	13	Credential access, prompt injection, data exfiltration, obfuscation, reverse shells, AST analysis, Python taint tracking, bash taint tracking, MCP least-privilege, tool poisoning, coercive override, stealth persistence, prompt exfiltration
Security (opt-in)	2	YARA signatures, CVE lookups via OSV.dev
Commands	11	Description, script exists, duplicates, credentials, injection, exfiltration, obfuscation, reverse shells, skill overlap, shadows built-in, references nonexistent skill
CLAUDE.md	3	Exists, skill duplication, generic advice detection
MCP	4	Configuration structure, duplicate servers, suspicious endpoints (localhost/private IPs), wildcard tool exposure
Hooks	5	Structure validation, script boundary, dangerous commands, env variable leakage, network access
Agents	10	Description, model specified, skills exist, tool format, constraint matching, credentials, injection, exfiltration, obfuscation, reverse shells

Four presets: recommended (default), strict, security, pre-workflow.

Security

For a full overview of how this tool protects your code, your credentials, and your supply chain, see how-can-you-know-its-safe-to-use-this-tool.md.

Privacy and Data Handling

harness-eval reads files from your project directory to analyze your AI agent setup. Here is what happens with your data in each mode:

Command	Sends data externally?	What is sent	Where
lint	No	Nothing. Fully offline.	N/A
review	Yes (CLI only)	Code snippets from your setup files	Gemini or Anthropic API (your choice via --provider)
security	Scan: No. --review: Yes (CLI only)	Code snippets from flagged files	Gemini or Anthropic API
skill	Lint: No. --rubric: Yes (CLI only)	The skill content being evaluated	Gemini or Anthropic API

When used as a Claude Code plugin, review/security/skill commands use the existing Claude session. No additional API calls are made.

When used as Cursor commands, the evaluation happens in the Cursor session. No additional API calls are made.

File access: The tool only reads files within the project directory you point it at. Path traversal protections prevent reading files outside the project boundary.

Contributing

See CONTRIBUTING.md for adding rules and submitting PRs.

Changelog

See CHANGELOG.md for release history.

Roadmap

See open issues for planned improvements and feature requests.