agent-eval-planner

Site: saulofilho.github.io/agent-eval-planner-python

Python library that turns an agent contract (system prompt, tools, policy gates) into a guardrail evaluation pipeline:

1. Action plan — SCOPE / INJECT / ROLE / PII / TOOL / HALLUC / EXFIL vectors
2. suite.jsonl — executable cases. generic and marketing-copilot keep the legacy row; --harness native emits the native envelope (target_component, turns, suite=regression, metadata.vector_id)
3. Remediations — prompt / gate / evaluator quick wins

Sibling of security-pentest-planner, applied to LLM agents. Port of the Ruby gem agent_eval_planner.

Canonical smoke test: carrot cake (off-topic). If the agent answers with a recipe, its scope is not limited.

What it does

• Parses agent contracts (Markdown, text, JSON, YAML).
• Generates a Markdown eval plan, a JSONL suite, and remediations.
• Validates suites. A guardrail that forbids no tool the target can call is a hard fail. A catalog tool the target cannot call is a warning. Leftover {{...}} placeholders fail.

What it does NOT do

• Execute the evaluation against a live agent.
• Replace pentest of APIs/infra (use security-pentest-planner).

Installation

pip install agent-eval-planner

Or using uv / poetry:

uv add agent-eval-planner
# or
poetry add agent-eval-planner

CLI usage

# Full pipeline → directory (plan + suite + remediations)
agent-eval-planner agent.md --tools funnel_analytics,open_service_center_ticket \
  -t "Platform Team" -a analytics -o ./out

# Validate suite (hard-fail on empty forbidden_tools / placeholders)
agent-eval-planner validate ./out/suite.jsonl \
  --known-tools funnel_analytics,open_service_center_ticket

# native row (target_component, turns, suite=regression, metadata.vector_id)
agent-eval-planner agent.md --harness native -a analytics \
  --tools funnel_analytics,open_service_center_ticket -o ./out
agent-eval-planner validate ./out/suite.jsonl --schema native \
  --bound-tools funnel_analytics \
  --known-tools funnel_analytics,open_service_center_ticket \
  --target-components analytics

# Plan only to stdout
agent-eval-planner agent.md --plan-only --agent analytics

Options

validate

suite: smoke is invalid on a native row; smoke lives in metadata.split. OK WITH WARNINGS does not mean the suite is ready to hand off. Any {{...}} left in the file fails, including runtime slots such as {{today_date}}.

Programmatic usage

import agent_eval_planner

result = agent_eval_planner.generate(
    input_path="agent.md",
    team="Platform Team",
    agent_name="analytics",
    tools=["funnel_analytics"],
    harness="native",
)

with open("evaluation-plan.md", "w", encoding="utf-8") as f:
    f.write(result.plan)

errors = agent_eval_planner.validate_suite(
    "out/suite.jsonl",
    known_tools=["funnel_analytics", "docs_search"],
    bound_tools=["funnel_analytics"],
    target_components=["analytics"],
    schema="native",
)

Publish to PyPI

python -m pip install build twine
python -m build
python -m twine upload dist/*

Development and testing

git clone https://github.com/saulofilho/agent-eval-planner-python.git
cd agent-eval-planner-python

python -m pip install .
python -m unittest discover -s tests -v

Recommended workflow

Agent contract  →  agent-eval-planner  →  Plan + suite.jsonl + remediations
                                                     ↓
                                            Harness / CI execution
                                                     ↓
                                            Failures + remediations

License

MIT — see LICENSE.