tool-scorer 1.7.0

Lightweight tool-call testing for LLM agents. Deterministic, local, zero API cost. Compare expected vs actual tool calls in 3 lines of Python. Supports OpenAI, Anthropic, Gemini.

Toolscore

The pytest of tool-calling — deterministic, local, zero API cost

---

Your agent calls tools — search APIs, databases, bookings, file ops. A prompt tweak or a model upgrade can silently change which tools it calls, with which arguments, in which order. Observability platforms tell you how your agent behaves in production; Toolscore fails your CI build before the regression ships. No LLM judge required, no cloud, no per-test API bill — just deterministic scores, snapshot baselines, and readable diffs, the way pytest does it.

from toolscore import expect, ANY, Regex

expect(agent).on("book me a flight to NYC") \
    .calls("search_flights", origin=ANY, destination="NYC") \
    .then_calls("book_flight", flight_id=Regex(r"FL-\d+")) \
    .does_not_call("cancel_booking") \
    .with_score(0.9) \
    .run()

60-Second Quickstart

pip install tool-scorer
toolscore init          # detects your framework, scaffolds a passing pytest suite
pytest                  # first run RECORDS your agent's tool calls as snapshots
toolscore approve --all # review, then approve them as the baseline
pytest                  # every run after this REPLAYS — and fails on drift

That's the whole loop. No hand-written expected-call files, no YAML. Your agent's own behavior becomes the regression test.

Snapshot Testing — Jest for Agents

Stop hand-writing expected tool calls. Record them once, approve them, replay them forever.

def test_books_a_flight(toolscore_snapshot):
    toolscore_snapshot(my_agent("book a flight to NYC"))
    # First run: records a pending snapshot and warns.
    # After `toolscore approve`: replays against the baseline, fails on drift.

The fixture ships with the package — no plugin install, no registration. Snapshots are plain JSON files under .toolscore/snapshots/, named after the pytest node id, so they review cleanly in PRs.

The workflow:

1. Record — the first pytest run captures your agent's tool calls into unapproved snapshots (a terminal summary tells you: toolscore: 1 snapshot created (pending approval)).
2. Approve — review with toolscore snapshots show <name>, then toolscore approve --all (or approve by name).
3. Replay — every subsequent run evaluates the agent against the approved baseline. Drift fails the test with a full expected-vs-actual diff.

Intentional behavior change? Re-record:

pytest --toolscore-update     # overwrite + re-approve baselines

CI is strict by design: snapshots are never created or auto-approved in CI — a missing or pending snapshot fails the build (downgrade to a warning with --toolscore-allow-pending for staged rollouts). You can also record outside pytest with toolscore record -- <any command> or from an existing trace file with toolscore record --from-trace trace.json --name my_snap.

MCP Scorecard — Grade Any MCP Server

The first standard testing tool for MCP servers. Point it at any server — it auto-generates happy-path and edge-case scenarios from each tool's schema, executes them, lints the tool definitions, and prints an A–F grade:

toolscore mcp test "python my_server.py"

# or straight from your Claude Desktop config, zero install:
uvx tool-scorer mcp test --config claude_desktop_config.json --server my-server

╭─────────────────────────────────────╮
│ MCP Scorecard: fake-mcp 0.1.0       │
│ Grade F   Score 47%                 │
│ happy 43%  |  edge 20%  |  lint 85% │
╰─────────────────────────────────────╯
                 Tools
┏━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Tool       ┃ Scenarios ┃ Avg latency ┃
┡━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ add        │       4/6 │      0.1 ms │
│ flaky      │       0/5 │      0.0 ms │
│ bad_schema │       0/1 │      0.0 ms │
└────────────┴───────────┴─────────────┘

Export a Markdown scorecard for your server's README with --report md --output SCORECARD.md. Here is the real output for the deliberately broken demo server above:

# MCP Scorecard: fake-mcp 0.1.0

**Grade: F** · Score 47%

- Happy-path pass rate: 43%
- Edge-case resilience: 20%
- Lint score: 85% (1 errors, 2 warnings)

## Tools

| Tool | Scenarios | Avg latency |
| --- | --- | --- |
| `add` | 4/6 | 0.1 ms |
| `flaky` | 0/5 | 0.0 ms |
| `bad_schema` | 0/1 | 0.0 ms |

## Lint

- warning · `flaky`: properties defined but no 'required' list declared
- warning · `bad_schema`: missing description
- **error** · `bad_schema`: missing or empty inputSchema

Gate CI on quality with --fail-under B (exit code 1 below the bar). toolscore mcp list and toolscore mcp lint are also available standalone.

Fluent Assertions and a Plain Score

Prefer a score over a chain? The core API is three lines:

from toolscore import evaluate

result = evaluate(
    expected=[
        {"tool": "get_weather", "args": {"city": "NYC"}},
        {"tool": "send_email", "args": {"to": "[e-mail]"}},
    ],
    actual=[
        {"tool": "get_weather", "args": {"city": "New York"}},
        {"tool": "send_email", "args": {"to": "[e-mail]"}},
    ],
)

print(result.score)              # 0.85 — weighted composite
print(result.selection_accuracy) # 1.0  — right tools picked
print(result.argument_f1)        # 0.5  — argument match quality

One-liner for any test framework — assert_tools(expected, actual, min_score=0.9). End-to-end in one call:

from toolscore import test_agent

test_agent(
    agent=my_agent,                 # any callable: prompt in, response out
    input="What's the weather in NYC?",
    expected=[{"tool": "get_weather", "args": {"city": "NYC"}}],
    min_score=0.9,
)

Async agents are first-class: await test_agent_async(...), or await expect(my_async_agent).on(prompt).calls(...).run_async().

Omit args in an expected call (or use .calls("tool") with no kwargs) to assert the tool was called without checking its arguments. An explicit "args": {} means "expect zero arguments".

Native Everywhere — Zero Glue

Pass raw responses straight into evaluate(), expect(), test_agent(), or the snapshot fixture. Toolscore auto-detects the format — no manual extraction:

Source	Auto-detected	Explicit helper
OpenAI (Chat Completions, legacy function_call)	Yes	from_openai
Anthropic (tool_use blocks)	Yes	from_anthropic
Google Gemini (functionCall parts)	Yes	from_gemini
LangGraph (state / message lists)	Yes	from_langgraph
Pydantic AI (run results)	Yes	from_pydantic_ai
OpenAI Agents SDK (run results)	Yes	from_openai_agents
Claude Agent SDK (message lists)	Yes	from_claude_agent_sdk
CrewAI (experimental)	Yes	from_crewai
MCP (JSON-RPC 2.0 traces)	Yes	file-based format="mcp"
LangChain / custom trace files	Yes	file-based format="auto"

response = client.chat.completions.create(model="gpt-4o", messages=[...], tools=[...])
result = evaluate(expected=[...], actual=response)   # just works

Matchers — Flexible Where It Matters

Exact equality is the default; matchers loosen exactly the arguments you choose:

Matcher	Matches	Example
ANY	anything	calls("search", q=ANY)
Regex(pattern)	full string match	Regex(r"FL-\d+")
Approx(value, rel, abs)	numbers within tolerance	Approx(40.71, rel=1e-2)
Contains(item)	membership in str/list/dict	Contains("metric")
OneOf(*values)	any of the candidates	OneOf("NYC", "New York")
IsType(*types)	isinstance check (bool-safe)	IsType(int)

from toolscore import evaluate, Approx, Contains, IsType, OneOf

evaluate(
    expected=[{"tool": "get_weather", "args": {
        "city": OneOf("NYC", "New York"),
        "units": Contains("metric"),
        "lat": Approx(40.71, rel=1e-2),
        "days": IsType(int),
    }}],
    actual=[{"tool": "get_weather", "args": {
        "city": "NYC", "units": ["metric", "extended"], "lat": 40.7128, "days": 5,
    }}],
)

Matchers work everywhere expected args do: evaluate, assert_tools, expect().calls(...), gold files.

Failures You Can Actually Read

When a threshold is missed, Toolscore renders an aligned expected-vs-actual table with per-argument mismatches and targeted tips — in the exception message itself, so it lands directly in your pytest output:

                                   Expected vs Actual Tool Calls
┏━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃   # ┃ Expected                     ┃ Actual                       ┃ Status                       ┃
┡━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│   1 │ search_flights(origin='SFO', │ search_flights(origin='SFO', │ destination: 'NYC' ≠ 'BOS'   │
│     │ destination='NYC')           │ destination='BOS')           │                              │
├─────┼──────────────────────────────┼──────────────────────────────┼──────────────────────────────┤
│   2 │ book_flight(flight_id='FL-1… │ cancel_booking(booking_id='… │ tool: 'book_flight' ≠        │
│     │                              │                              │ 'cancel_booking'             │
└─────┴──────────────────────────────┴──────────────────────────────┴──────────────────────────────┘
score 0.47 < 0.90 required  ·  selection 0.50  ·  args 0.40  ·  sequence 0.50

Tips:
  • Use --llm-judge flag to catch semantic equivalence
  • Check that your agent has access to all required tools
  • Verify tool names match exactly (case-sensitive)

(That is real output from a deliberately failing assert_tools — color in a TTY, plain text in CI logs.)

The composite result.score weighs selection accuracy (40%), argument F1 (30%), sequence accuracy (20%), and redundancy (10%); pass weights={...} to re-balance (weights are renormalized to sum to 1.0).

Optional: LLM Judge for Every Provider

When search_web vs web_search is a semantic question, opt into an LLM judge — via OpenAI, Anthropic, Gemini, or any OpenAI-compatible endpoint (Ollama, vLLM, Groq):

toolscore eval gold.json trace.json --llm-judge                                  # OpenAI default
toolscore eval gold.json trace.json --llm-judge --llm-model claude-3-5-haiku-latest
toolscore eval gold.json trace.json --llm-judge --llm-model llama3.1 \
    --llm-base-url http://localhost:11434/v1                                     # local Ollama

from toolscore import evaluate_trace, JudgeConfig

result = evaluate_trace("gold.json", "trace.json",
                        judge=JudgeConfig(model="gemini-2.0-flash"))

The provider is inferred from the model name. Install extras as needed: tool-scorer[llm] (OpenAI/compatible), [anthropic], [gemini]. Everything else in Toolscore stays deterministic and offline.

CI/CD

toolscore init writes a GitHub Actions workflow that replays your approved snapshots on every push. Or use the official action directly:

# Gold-standard evaluation with a threshold
- uses: yotambraun/toolscore@v1
  with:
    gold-file: tests/gold_standard.json
    trace-file: tests/agent_trace.json
    threshold: '0.90'

# MCP scorecard mode — grade your MCP server on every PR
- uses: yotambraun/toolscore@v1
  with:
    mcp-command: 'uvx my-mcp-server'
    mcp-fail-under: 'B'

Baseline regression checks catch slow degradation:

toolscore eval gold.json trace.json --save-baseline baseline.json   # once
toolscore regression baseline.json new_trace.json --gold-file gold.json
# exit codes: 0 = PASS, 1 = regression detected, 2 = error

When to Use Toolscore vs. the Platforms

Toolscore is the pytest of tool-calling: it runs in your test suite, deterministically, for free. Observability and eval platforms watch your agent in production. Use both.

You want to...	Use
Fail the CI build when tool calls drift, deterministically, $0 per run	Toolscore
Grade and lint an MCP server	Toolscore (toolscore mcp test)
Score production traces across many quality dimensions (hallucination, toxicity, RAG)	DeepEval, MLflow
Trace, monitor, and debug agents in production	LangSmith, Arize Phoenix
Evaluate RAG retrieval/faithfulness	Ragas
Safety-focused evaluation harnesses	Inspect AI

Toolscore does one thing well: it verifies your agent calls the right tools, with the right arguments, in the right order — before you ship.

Learn More

• Documentation — full API reference and guides
• TUTORIAL.md — step-by-step walkthrough, from first score to CI
• CHANGELOG.md — what's new
• Medium article — the story behind Toolscore

Development

pip install -e ".[dev]"
pytest
ruff check toolscore
mypy toolscore

License

Apache License 2.0 - see LICENSE for details.

Citation

@software{toolscore,
  title = {Toolscore: Lightweight Tool-Call Testing for LLM Agents},
  author = {Yotam Braun},
  year = {2025},
  url = {https://github.com/yotambraun/toolscore}
}