pytest-skill-engineering

Test-Driven Skill Engineering for GitHub Copilot

Test MCP servers, CLI tools, Agent Skills, and custom agents using the real GitHub Copilot coding agent. Write tests as prompts, run them against actual Copilot sessions, and get AI-powered insights on what to fix.

Why?

Your MCP server passes all unit tests. Then a user tries it in GitHub Copilot and:

• Copilot picks the wrong tool
• Passes garbage parameters
• Can't recover from errors
• Ignores your skill's instructions

Why? Because you tested the code, not the AI interface.

For LLMs, your API isn't functions and types — it's tool descriptions, Agent Skills, custom agent instructions, and schemas. These are what GitHub Copilot actually sees. Traditional tests can't validate them.

The key insight: your test is a prompt. You write what a user would say ("What's my checking balance?"), and Copilot figures out how to use your tools. If it can't, your AI interface needs work.

What This Tests

pytest-skill-engineering validates the full skill engineering stack that ships with your MCP server:

• MCP Server Tools — Can Copilot discover and call your tools correctly?
• Agent Skills (agentskills.io spec-compliant) — Does domain knowledge improve performance?
• Custom Agents (.agent.md files) — Do your specialist instructions trigger proper custom agent dispatch?
• MCP Prompt Templates — Do server-side templates produce the right behavior?
• CLI Tools — Can Copilot use command-line interfaces effectively?

Plus A/B testing, multi-step workflows with explicit context, and AI-powered reports that tell you exactly what to fix.

How It Works

Write tests as prompts. Run them with the real GitHub Copilot coding agent. Assert on what happened:

A system prompt configures behavior (CopilotEval.instructions or a custom agent's body). A prompt is the user task passed to copilot_eval.

from pytest_skill_engineering.copilot import CopilotEval


async def test_balance_query(copilot_eval):
    agent = CopilotEval(
        skill_directories=["skills/banking-advisor"],
        max_turns=10,
    )
    result = await copilot_eval(agent, "What's my checking balance?")

    assert result.success
    assert result.tool_was_called("get_balance")

The workflow:

1. Write a test — a prompt that describes what a user would say
2. Run it — GitHub Copilot tries to use your tools
3. Fix the interface — improve tool descriptions, skills, or agent instructions until it passes
4. AI analysis tells you what to optimize — cost, redundant calls, better system prompts

If a test fails, your AI interface needs work, not your code.

Agent Skills — First-Class Support

pytest-skill-engineering provides full Agent Skills spec compliance:

• Compatibility field — Mark required tools, models, or platforms
• Metadata — Title, description, version, attribution
• Allowed-tools — Restrict which tools the agent can use
• Scripts & Assets — Package Python scripts, prompts, and resources
• Eval Bridge — Import evals from evals/evals.json, export grading results

Agent Skills are loaded natively when testing with CopilotEval — exactly as users experience them.

AI-Powered Reports

AI analyzes your results and tells you what to fix: which configuration to deploy, how to improve tool descriptions, where to cut costs. See a sample report →

Quick Start

You need Python 3.11+, uv, the GitHub CLI, and a GitHub account with Copilot access.

# Add the package to an existing Python project
uv add pytest-skill-engineering

# Authenticate once, then create and verify the starter
gh auth login --hostname github.com
uv run pytest-skill-engineering init
uv run pytest-skill-engineering doctor

# Run one real Copilot eval
uv run python -m pytest tests/test_copilot_eval.py -v

init creates tests/test_copilot_eval.py, adds explicit pytest settings to pyproject.toml, and records gpt-5.6-sol cost rates in pricing.toml. It refuses to overwrite an existing starter or conflicting report configuration and preserves existing model pricing. The starter invokes the bundled Todo MCP server and may consume Copilot premium requests.

After the run, open:

• aitest-reports/report.html — rendered results and required AI analysis
• aitest-reports/results.json — raw execution evidence

See the complete quickstart or run the standalone examples/quickstart project.

Features

• MCP Server Testing — Test tools, prompt templates, and bundled skills with real Copilot sessions
• Agent Skills — Full agentskills.io spec compliance (compatibility, metadata, allowed-tools, evals bridge)
• Custom Agents — Test .agent.md files and validate custom agent dispatch
• CLI Tool Testing — Verify Copilot can use command-line interfaces
• Plugin Testing — Load complete plugin directories (plugin.json, .github/, .claude/ layouts) with auto-discovery
• A/B Testing — Compare instructions, skills, custom agent versions, or tool configurations
• Eval Leaderboard — Auto-ranked by pass rate and cost
• Multi-Turn Sessions — Test conversations that build on context
• LLM Assertions — Semantic checks with llm_assert and multi-dimension scoring with llm_score
• Tool Images — Capture tool-returned images for inspection in results and reports
• AI-Powered Reports — Actionable feedback on tool descriptions, system prompts, and costs
• Cost Tracking — Copilot premium request tracking + USD estimation via pricing.toml

Who This Is For

• MCP server authors — Validate that GitHub Copilot can actually use your tools
• Agent Skills authors — Test skills exactly as users experience them in Copilot
• Custom agent builders — Validate .agent.md instructions and custom agent dispatch
• Plugin developers — Test complete GitHub Copilot CLI plugins end-to-end
• Teams shipping Copilot integrations — Catch skill stack regressions in CI/CD

Documentation

📚 Full Documentation

Requirements

• Python 3.11+
• pytest 9.0+
• GitHub Copilot subscription (required)

Acknowledgments

Inspired by agent-benchmark.

License

MIT