llm-probe 0.1.3

Probe LLM models for tool-calling support, think block behavior, and response quality across providers.

llm-probe

Test whether LLM models actually work before you build on them. Discovers models from cloud and local providers, runs each through tool-calling, system prompt adherence, and response quality checks, then outputs a verified model list your project can consume.

Picking a model for agents or tool-calling workflows? Running Hugging Face models through Ollama, vLLM, or any OpenAI-compatible endpoint? Most models can generate text, but structured tool calls, enum constraints, and system prompt adherence are different — llm-probe shows you exactly where each model holds up and where it falls short for your use case.

Install

# As a CLI tool (recommended)
pipx install llm-probe

# Or with pip
pip install llm-probe

Quick Start

Run from your project directory — results are written to ./verified/ relative to where you run it.

cd ~/dev/projects/my-project

# Set your API key
export VENICE_API_KEY=your-key-here

# Discover available models
llm-probe discover venice

# Test a specific model
llm-probe test venice --model mistral-small-3-2-24b-instruct

# Test all models from a provider
llm-probe test venice

# View results
llm-probe report

Ollama needs no API key but requires a running instance. If it's not on localhost:

llm-probe --config config.yaml discover ollama
llm-probe --config config.yaml test ollama --model qwen3:8b --timeout 180

Providers

Provider	Discovery	Auth	Default URL
venice	/v1/models API	VENICE_API_KEY env var	https://api.venice.ai/api/v1
ollama	/api/tags endpoint	None	http://localhost:11434
llama.cpp	/v1/models API	None	http://<host>:8080/v1
anthropic	/v1/models API	ANTHROPIC_API_KEY env var	https://api.anthropic.com
openai	/v1/models API	OPENAI_API_KEY env var	https://api.openai.com/v1

All providers except Anthropic use the OpenAI-compatible chat completions API. Anthropic uses the Messages API with tool_use content blocks.

llama.cpp requires a config.yaml entry (not a built-in provider name) — see Custom OpenAI-Compatible Providers below. LM Studio, vLLM, and text-gen-webui work the same way. Local servers typically use http, not https, unless you've configured TLS yourself.

Probe accuracy: Results are only meaningful when tested under the same conditions your runtime uses. If your runtime appends a suffix to model names (e.g. model-name-tools), set model_suffix in config so the probe tests the exact same model variant. The tool_call_large test also helps catch models that pass small tool lists but fail under real agent conditions.

Custom OpenAI-Compatible Providers

Any service that speaks the OpenAI chat completions API can be used without writing code. Just add it to config.yaml:

providers:
  lmstudio:
    api_base: http://localhost:1234/v1
  openrouter:
    api_base: https://openrouter.ai/api/v1
    api_key_env: OPENROUTER_API_KEY
  text-gen-webui:
    api_base: http://<host>:5000/v1

Then use it like any built-in provider:

llm-probe discover lmstudio
llm-probe test lmstudio --model my-local-model

Custom providers get automatic model discovery via /models, tool-call testing via /chat/completions, and local timeout defaults if the URL points to a private IP or localhost.

Tests

Each model is tested for specific capabilities. Tests run N times (default 3) with a pass threshold (default 80%) to catch intermittent failures. See TESTING.md for detailed test documentation.

Test	What it checks
tool_call_basic	Send one tool + an obvious prompt. Does the model return a structured tool_calls response (not text)?
tool_call_enum	Send a tool with enum-constrained parameters. Does the model respect the constraint?
tool_call_multi	Send three tools. Does the model pick the correct one?
think_block_leak	Does internal reasoning (<think>, Thinking Process:, etc.) leak into the response content?
empty_response	Does the model produce empty, degenerate (!!!!), or garbage output?
system_prompt	Given a system prompt that overrides user intent, does the model follow it? Tests if the model respects system prompt instructions over user messages.
response_time	Records response latency as metadata. Always passes.
tool_call_large	Send 11 tools (1 target + 10 distractors). Does the model still pick the right one? Tests conditions closer to real agent use.

Run specific tests with --tests:

llm-probe test venice --model mistral-small-3-2-24b-instruct --tests tool_call_basic,think_block_leak

CLI Reference

# Global option: --config / -c (optional, defaults work for all providers)
llm-probe --config config.yaml <command>

# Discover available models from a provider
llm-probe discover <provider>

# Test models
llm-probe test <provider>                          # all models
llm-probe test <provider> --model <name>           # one model
llm-probe test <provider> --tests <test1,test2>    # specific tests
llm-probe test <provider> --runs 5 --threshold 0.8 # consistency testing
llm-probe test <provider> --timeout 180            # for slow local models

# View results
llm-probe report                # all providers
llm-probe report <provider>     # one provider

Example Output

[1/1] Testing mistral-small-3-2-24b-instruct...
  PASS 6/7 tests passed (avg 1358ms)

Results written to verified/venice.yaml

  + New models: mistral-small-3-2-24b-instruct

Re-running compares against previous results and reports regressions:

  REGRESSIONS:
    qwen3-5-9b: pass -> fail
  IMPROVEMENTS:
    llama-3.3-70b: intermittent -> pass

Configuration

Configuration is optional. Defaults exist for all providers. Copy config.yaml.example to config.yaml and edit as needed:

providers:
  venice:
    api_base: https://api.venice.ai/api/v1
    api_key_env: VENICE_API_KEY             # env var name
    model_suffix: "-tools"                  # optional: append to model IDs when calling the API
  ollama:
    api_base: http://<host>:11434            # override for non-localhost
  anthropic:
    api_key_env: ANTHROPIC_API_KEY
    api_key_secret: ~/secrets/anthropic_key # file path (checked if env var is empty)

tests:
  timeout_cloud: 30        # seconds per request for cloud providers
  timeout_local: 180       # seconds per request for local models (CPU inference)
  runs_per_model: 3        # test each model N times for consistency
  consistency_threshold: 0.8  # must pass 80% of runs to be "pass"

output:
  dir: ./verified          # where to write result YAML files

API keys resolve in order: api_key_env (environment variable) > api_key_secret (file path). Providers that don't need keys (Ollama) skip both.

Output Format

Results are written to ./verified/<provider>.yaml:

provider: venice
last_run: "2026-03-31"
models:
  - id: mistral-small-3-2-24b-instruct
    tool_call: pass             # pass, intermittent, or fail
    think_blocks: none          # none, xml, plain_text
    avg_response_ms: 1358
    tests:
      tool_call_basic: {passed: true, pass_rate: "3/3"}
      tool_call_enum: {passed: true, pass_rate: "3/3"}
      tool_call_multi: {passed: true, pass_rate: "3/3"}
      think_block_leak: {passed: true, pass_rate: "3/3"}
      empty_response: {passed: true, pass_rate: "3/3"}
      system_prompt: {passed: false, pass_rate: "0/3"}
      response_time: {passed: true, pass_rate: "3/3"}
    last_tested: "2026-03-31"

Other projects can import these YAML files to build model allowlists for tool-calling features.