ToolCallBench

Find out which local language model calls tools reliably on your own machine.

ToolCallBench gives a model six tools (read, list, search, edit, patch, run a command) and 20 small tasks on a sample Python project. It checks the results with plain code, not with another model, and writes a report that compares models and quantization levels.

It is tuned for a MacBook M4 with 16 GB of memory running Ollama, and works with any OpenAI-compatible local server (llama.cpp, LM Studio, vLLM).

Install and run

pip install toolcallbench

mkdir my-bench && cd my-bench
toolcallbench init                                  # creates configs/ with example settings

Edit configs/my-models.yaml and list the models you have pulled in Ollama, then:

toolcallbench doctor --probe --config configs/my-models.yaml   # check Ollama, models, tool calling
toolcallbench run --config configs/my-models.yaml              # run the 20 tasks
toolcallbench sessions                                         # list past runs
toolcallbench report --session <id>                            # Markdown + HTML report in ./results/reports

Every setting can also be given as an argument, and arguments override the config file:

toolcallbench run --model qwen3:8b --model gemma4:e4b --repeats 3 --num-ctx 16384 --groups read,edit
toolcallbench run --help

Other commands: toolcallbench list-tasks, toolcallbench validate-tasks, toolcallbench init --with-tasks (copies the tasks and sample repo so you can edit them; a folder with its own tasks/ and fixtures/repo/ uses them instead of the built-in ones). Results are written to ./results.

No model yet? toolcallbench run --backend scripted --repeats 1 runs the whole pipeline without one.

Ollama must be running (ollama serve) with the models pulled (ollama pull qwen3:8b). On a 16 GB Mac, start from the m4-16gb.yaml example config. See docs/USER_GUIDE.md.

Develop

git clone <repo> && cd toolcallbench
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
python -m pytest

What a report tells you

• Pass rate overall and for read, edit, run, chain and failure tasks
• Why runs failed (13 failure types, such as invalid_json, text_tool_call, patch_failed)
• Tokens per second, seconds per task, and calls compared with a reference solution
• Every setting that affects the result (context length, seed, KV cache type, versions)

Status

Version 1.0.0. The harness, tasks, scoring, backends (checked against mock servers) and reports are covered by automated tests. No real model has been benchmarked yet; run it on your Mac to get the first numbers (docs/TEST_PLAN.md lists exactly what was and was not tested).

Documentation

License

MIT