agent-eval-planner
Site: saulofilho.github.io/agent-eval-planner-python
Python library that turns an agent contract (system prompt, tools, policy gates) into a guardrail evaluation pipeline:
- Action plan — SCOPE / INJECT / ROLE / PII / TOOL / HALLUC / EXFIL vectors
- suite.jsonl — executable cases.
genericandmarketing-copilotkeep the legacy row;--harness nativeemits the native envelope (target_component,turns,suite=regression,metadata.vector_id) - Remediations — prompt / gate / evaluator quick wins
Sibling of security-pentest-planner, applied to LLM agents. Port of the Ruby gem agent_eval_planner.
Canonical smoke test: carrot cake (off-topic). If the agent answers with a recipe, its scope is not limited.
What it does
- Parses agent contracts (Markdown, text, JSON, YAML).
- Generates a Markdown eval plan, a JSONL suite, and remediations.
- Validates suites. A guardrail that forbids no tool the target can call is a hard fail. A catalog tool the target cannot call is a warning. Leftover
{{...}}placeholders fail.
What it does NOT do
- Execute the evaluation against a live agent.
- Replace pentest of APIs/infra (use
security-pentest-planner).
Installation
pip install agent-eval-planner
Or using uv / poetry:
uv add agent-eval-planner
# or
poetry add agent-eval-planner
CLI usage
# Full pipeline → directory (plan + suite + remediations)
agent-eval-planner agent.md --tools funnel_analytics,open_service_center_ticket \
-t "Platform Team" -a analytics -o ./out
# Validate suite (hard-fail on empty forbidden_tools / placeholders)
agent-eval-planner validate ./out/suite.jsonl \
--known-tools funnel_analytics,open_service_center_ticket
# native row (target_component, turns, suite=regression, metadata.vector_id)
agent-eval-planner agent.md --harness native -a analytics \
--tools funnel_analytics,open_service_center_ticket -o ./out
agent-eval-planner validate ./out/suite.jsonl --schema native \
--bound-tools funnel_analytics \
--known-tools funnel_analytics,open_service_center_ticket \
--target-components analytics
# Plan only to stdout
agent-eval-planner agent.md --plan-only --agent analytics
Options
| Flag | Description |
|---|---|
-t, --team TEAM |
Team name in the document title |
-a, --agent NAME |
target_agent (legacy) or target_component (native) |
--tools LIST |
Comma-separated real tool names. On native these are the tools bound to the target |
--scope TEXT |
Declared scope summary |
--harness NAME |
generic, marketing-copilot, or native |
-o, --output PATH |
Output directory or single file |
--plan-only / --suite-only / --remediations-only |
Emit a single artifact |
-v, --version |
Show version |
validate
| Flag | Description |
|---|---|
--known-tools LIST |
Product catalog. With only this flag, every name is treated as bound (legacy suites) |
--bound-tools LIST |
Tools wired to this target. Must be a subset of --known-tools |
--schema NAME |
auto (default), legacy, or native. A file that mixes the two shapes fails |
--target-components LIST |
Allowed target_component or legacy target_agent values |
--suites LIST |
Allowed native suite values. Default: golden,regression,routing,calibration,production-sample |
--agent-has-no-tools |
Agent truly has no tools. Do not combine with tool flags |
suite: smoke is invalid on a native row; smoke lives in metadata.split. OK WITH WARNINGS does not mean the suite is ready to hand off. Any {{...}} left in the file fails, including runtime slots such as {{today_date}}.
Programmatic usage
import agent_eval_planner
result = agent_eval_planner.generate(
input_path="agent.md",
team="Platform Team",
agent_name="analytics",
tools=["funnel_analytics"],
harness="native",
)
with open("evaluation-plan.md", "w", encoding="utf-8") as f:
f.write(result.plan)
errors = agent_eval_planner.validate_suite(
"out/suite.jsonl",
known_tools=["funnel_analytics", "docs_search"],
bound_tools=["funnel_analytics"],
target_components=["analytics"],
schema="native",
)
Publish to PyPI
python -m pip install build twine
python -m build
python -m twine upload dist/*
Development and testing
git clone https://github.com/saulofilho/agent-eval-planner-python.git
cd agent-eval-planner-python
python -m pip install .
python -m unittest discover -s tests -v
Recommended workflow
Agent contract → agent-eval-planner → Plan + suite.jsonl + remediations
↓
Harness / CI execution
↓
Failures + remediations
License
MIT — see LICENSE.
Metadata
Release files for agent-eval-planner 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_eval_planner-0.2.0.tar.gz | 27.4 kB | Without provenance | Details for agent_eval_planner-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_eval_planner-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for agent_eval_planner-0.2.0-py3-none-any.whl |
Total release size: 55.1 kB
Release files / agent_eval_planner-0.2.0.tar.gz
| Download URL | agent_eval_planner-0.2.0.tar.gz |
|---|---|
| Size | 27.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7dc03a446a5b0c40dd72b782580b1f494580a21a6daa08b6e106db31ad7ed00e
|
|
BLAKE2b-256 checksum How to use checksums |
3b828893304c6de7ed85d338015e3437c0003712f00c6886e64398b6a903780d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Release files / agent_eval_planner-0.2.0-py3-none-any.whl
| Download URL | agent_eval_planner-0.2.0-py3-none-any.whl |
|---|---|
| Size | 27.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1533060e62dea34ec9b2d27da377a82bc7c0ff8368caaf8786f3ae14a6c1f8dc
|
|
BLAKE2b-256 checksum How to use checksums |
71d8e5b3cdc00ce201429a4da7f131f7f79564efee826e313b7753ca3189048a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.22
|
Security for agent-eval-planner 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.