skilleval 0.1.0

Find the cheapest LLM that gets your task 100% right

SkillEval

English | 中文

Find the cheapest LLM that gets your task 100% right.

SkillEval is a CLI tool that automates LLM evaluation for deterministic tasks. It runs your task across multiple models in parallel, compares outputs against expected results, and recommends the most cost-effective option.

Quick Start

pip install -e .

# Set at least one provider API key
export DASHSCOPE_API_KEY="sk-..."   # Qwen (Alibaba DashScope)

# Create a task folder
skilleval init my-task

# Add your input files, expected output, and skill prompt
# Then run the evaluation
skilleval run my-task/

# Machine-readable output
skilleval run my-task/ --json | jq '.recommendation'

Supported Providers

Provider	Platform	Env Variable
Qwen	Alibaba Cloud / DashScope	DASHSCOPE_API_KEY
GLM	Zhipu AI / BigModel	ZHIPU_API_KEY
MiniMax	MiniMax	MINIMAX_API_KEY

Evaluation Modes

• Mode 1 (run) — You write the prompt (skill), SkillEval tests it across models.
• Mode 2 (matrix) — One model writes the prompt, another executes it. Tests all creator x executor combinations.
• Mode 3 (chain) — A meta-skill guides prompt creation, then another model executes it. Full pipeline evaluation.

Additional Features

• Ad-hoc endpoints — Use any OpenAI-compatible API without editing the catalog: --endpoint, --api-key, --model-name.
• Skill linting (lint) — Validate Claude Code skill structure (frontmatter, phases, references, code blocks).
• Skill testing (skill-test) — Test a skill's core prompt logic against expected outputs.
• Run comparison (compare) — Diff two runs to detect improvements or regressions.
• HTML reports (report --html) — Generate self-contained HTML reports for sharing.
• JSON output (--json) — Machine-readable JSON on run, matrix, chain, catalog, and report commands for piping into other tools.
• Verbose logging (-v / -vv) — -v for INFO, -vv for DEBUG. Logs go to stderr so they don't interfere with --json output.
• Auto-confirm (--yes / -y) — Skip the confirmation prompt on chain (replaces the old --confirm flag).
• Config validation — Warns on unknown keys in config.yaml and validates comparator names at load time.
• Circuit breaker — Automatically skips a provider after 5 consecutive failures, avoiding wasted time and cost.
• Ctrl+C handling — Saves partial results on interrupt so you never lose a half-finished run.
• Friendly errors — No raw tracebacks by default; use -vv to see full stack traces when debugging.
• Progress bar — Now shows elapsed time and ETA alongside the completion percentage.

Documentation

See the User Manual (中文) for detailed setup instructions, configuration options, comparator reference, and walkthroughs.

Development

pip install -e ".[dev,docs]"
pytest
ruff check src/ tests/

See CONTRIBUTING.md (中文) for full contributor guidelines.

License

MIT