harness-eval

CI PyPI Python 3.11+ Rules License: Apache 2.0

Evaluate AI code agent setups for best practices, redundancy, security, and cross-component issues.

Most tools test whether a skill produces correct output. This tool checks the setup itself: CLAUDE.md, GEMINI.md, AGENTS.md, skills, commands, hooks, MCP configs, agents, .cursor/rules/*.mdc, .cursorrules, .github/prompts/, .opencode/.

Quick start

pip install harness-eval
harness-eval lint .          # 84 deterministic rules, fully offline
harness-eval security .      # security scan (18 rules)

See docs/INSTALL.md for all installation options and configuration.

How to use it

Available as a CLI tool, a GitHub Action, a Tekton Task (OpenShift Pipelines), a Claude Code plugin, and Cursor commands. Each is documented in docs/INSTALL.md.

Command What it does LLM needed?
lint 84 deterministic rules + system analysis (token budget, trigger overlaps, dependencies). Fast, CI-suitable. Supports --format sarif. No
review Per-component rubric review with scoring, 21 cross-type checks, KEEP/REVIEW/REMOVE verdicts. CLI: [llm] extra. Plugin/Cursor: in-session.
security All security rules + YARA + CVE lookups + optional semantic review. SAFE/CAUTION/UNSAFE. Scan: no. --review: [llm] extra or in-session.
skill Deep-evaluate one skill individually and in context of the full setup. Lint: no. --rubric: [llm] extra or in-session.
rules List all rules. Filter by --category or --target. No

Supported AI tools

Auto-detects which tool(s) a project uses and evaluates all discovered components together. Multi-tool projects are fully supported.

Assistant What it discovers
Claude Code CLAUDE.md, skills/, commands/, .claude/agents/, .claude/settings.json, .mcp.json
Cursor .cursor/rules/*.mdc, .cursorrules, .cursor/commands/, .cursor/skills/, .cursor/hooks.json, .cursor/mcp.json
Windsurf .windsurfrules, .windsurf/rules/*.md
Cline .clinerules (file or directory of *.md)
Copilot .github/copilot-instructions.md, .github/skills/, .github/prompts/, .github/agents/
Gemini CLI GEMINI.md, .gemini/commands/, .gemini/settings.json (MCP)
OpenCode AGENTS.md, .opencode/commands/, .opencode/agents/, opencode.json (MCP)
Third-party modules .lola/modules/ (skills, commands, agents installed via package managers)

Inspection rules

84 deterministic rules across 11 categories: structural, frontmatter, content, quality, security, cross-component, commands, CLAUDE.md, MCP, hooks, and agents. Four presets: recommended (default), strict, security, pre-workflow.

Cross-component analysis is the core differentiator. Most linters check files in isolation; harness-eval builds a component graph that traces data flows across skills, agents, hooks, and MCP servers. This catches threats like credential exfiltration chains and confused deputy attacks.

For the complete rule list with examples, detection techniques, and framework mappings (OWASP, MITRE ATLAS), see docs/rules-reference.md.

Privacy

lint and security (without --review) are fully offline. LLM review is opt-in: only review, security --review, and skill --rubric send snippets to a remote provider (Gemini or Anthropic via CLI, or in-session as a plugin/command).

Before any remote LLM call, likely secrets (tokens, PEM keys, API_KEY= assignments, known prefix patterns) are replaced with [REDACTED] (HE-2). Scans also skip .env, credentials paths, and *.pem / *.key / id_rsa globs by default (HE-3); add more with --exclude.

See docs/how-can-you-know-its-safe-to-use-this-tool.md for details.

Contributing

See CONTRIBUTING.md for adding rules and submitting PRs.

Changelog

See CHANGELOG.md for release history.

Roadmap

See open issues for planned improvements and feature requests.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (7.5.0)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for bkapner from gravatar.com bkapner

License expression

Apache-2.0
View SPDX License List

Requires

Python >=3.11

Provides Extra

bash-ast dev llm tiktoken watch yara

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

harness_eval-7.2.0.tar.gz (253.3 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

harness_eval-7.2.0-py3-none-any.whl (213.3 kB view details)

Uploaded Python 3

File details

Details for the file harness_eval-7.2.0.tar.gz.

File metadata

  • Download URL: harness_eval-7.2.0.tar.gz
  • Upload date:
  • Size: 253.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for harness_eval-7.2.0.tar.gz
Algorithm Hash digest
SHA256 ba0bb8243742c6bfde9e04a6780b4e7a51eed7824f3926e477a15311f893823d
MD5 b886bd6f3298e45ca479fac3c46351c7
BLAKE2b-256 fc6987cdc38271a8ba6129b43622ed079e45de491acdfb4df38a311218b32f86

See more details on using hashes here.

Provenance

The following attestation bundles were made for harness_eval-7.2.0.tar.gz:

Publisher: publish.yml on redhat-community-ai-tools/harness-eval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file harness_eval-7.2.0-py3-none-any.whl.

File metadata

  • Download URL: harness_eval-7.2.0-py3-none-any.whl
  • Upload date:
  • Size: 213.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for harness_eval-7.2.0-py3-none-any.whl
Algorithm Hash digest
SHA256 7e2b1fb83e685b5b5b8bd7ea7d619a4abe8f182d50b09a304a33a9003d73a202
MD5 7c9d609ec415c5cb37c8d95dafc24dd2
BLAKE2b-256 95a8c17513f2566bc8e53aa990b2aed761d4c8c21f404ec95fd02b0ab8d227c4

See more details on using hashes here.

Provenance

The following attestation bundles were made for harness_eval-7.2.0-py3-none-any.whl:

Publisher: publish.yml on redhat-community-ai-tools/harness-eval

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page