Agent Eval 评估器
Agent 能力评估框架（agent-eval CLI）——基于 Agent-Driven 架构，支持课件生成、
代码生成、RAG、对话等多类 Agent 评估。
安装（发布后）
pip install ai-eval-scope # 基础：规则评估（无 LLM 依赖）
pip install "ai-eval-scope[llm]" # + LLM Judge（OpenAI/Anthropic 兼容）
pip install "ai-eval-scope[agent]" # + ExecutionAgent 执行引擎（DeepAgents 底座）
pip install "ai-eval-scope[vision]" # + 视觉评估（playwright 截图渲染）
pip install "ai-eval-scope[datasets]" # + 数据集下载（HuggingFace/ModelScope）
uv tool install "ai-eval-scope[agent]" # uv：全局 CLI 工具安装
uvx --from "ai-eval-scope[agent]" agent-eval --help # 一次性运行，不落盘
包名
ai-eval-scope，import 名agent_eval，CLI 命令agent-eval（命令名≠包名）。 Python ≥ 3.11。LLM 与凭证配置见 CLI 使用教程。
开发
cd evaluator
uv sync --group dev # 开发依赖组（PEP 735，含 llm/vision 测试依赖）
uv run playwright install chromium # 下载 Chromium 浏览器二进制（~150MB，装到用户缓存）
uv run agent-eval --help
配置
交互式向导（推荐）
uv run agent-eval models login
向导流程：
- 选择提供商：
anthropic/openai/deepseek/custom（自定义 OpenAI 兼容端点） - API Base URL：按提供商给默认值（anthropic 缺省
https://api.moonshot.cn/anthropic，即 Kimi 的 Anthropic 协议端点；openai 缺省https://api.openai.com/v1；deepseek 缺省https://api.deepseek.com/v1） - API Key：隐藏输入
- 逐角色确认模型：
text— 文本 LLM Judge（默认必配）vision— 视觉评估模型（用视觉规则集时需要）agent— 执行侧模型（run/pipeline驱动被测 Agent 用；未配置时自动回退text）
查看与验证
uv run agent-eval models list # 查看（API Key 脱敏显示 前4…后4）
uv run agent-eval models test # 对每个已配角色真实调用一次，打印时延
uv run agent-eval models logout # 删除配置（含 key）
运行
uv run agent-eval eval \
--package-dir workspace/packages/<包目录>/ \
--package courseware
验证是否真正执行：产物中出现 *.png 截图、评估耗时显著增加（视觉渲染 + 视觉 LLM 调用），
且日志无「视觉跳过」「Chromium 启动失败」字样。缓存命中时会跳过视觉重算，调试时加
--no-cache 强制重评。
1 maintainer
Metadata
1 maintainer
Release files for ai-eval-scope 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| ai_eval_scope-0.4.0.tar.gz | 1.4 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ai_eval_scope-0.4.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 2.5 MB
Release files / ai_eval_scope-0.4.0.tar.gz
| Download URL | ai_eval_scope-0.4.0.tar.gz |
|---|---|
| Size | 1.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3c4a50e5c024ffd8dd5516d5915e0d4759a1eed15f311979b4b0cdda3fe5471d
|
|
BLAKE2b-256 checksum How to use checksums |
edaafe18128c63b4aedc0d819cd472657f904284ff1d60cfb9131ea64b4c083a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / ai_eval_scope-0.4.0-py3-none-any.whl
| Download URL | ai_eval_scope-0.4.0-py3-none-any.whl |
|---|---|
| Size | 1.1 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
31e9a9b3abe0271563d5827943d43514c749a89853b668b94926d0274f8c1920
|
|
BLAKE2b-256 checksum How to use checksums |
7d273b68ec14b86b65cf509ed744423feb6ed195f409790024009d788762c5d8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.13 {"installer":{"name":"uv","version":"0.12.13","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Debian GNU/Linux","version":"12","id":"bookworm","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for ai-eval-scope 0.4.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
