probelock 0.1.0

A capability lockfile for local models. Catch silent regressions when you swap a model, quant, or runtime.

probelock

A capability lockfile for local models. It records what a model does on a set of tool-calling and output checks, and fails CI when a model/quant/runtime swap lowers a score.

llama-3.1-8b @ Q8_0 (ollama)  →  llama-3.1-8b @ Q4_K_M (ollama)
Capability            Baseline   Candidate     Δ   Status
arg_validity              1.00        0.67  -0.33  REGRESSION
arity_robustness          1.00        0.67  -0.33  REGRESSION
format_adherence          1.00        1.00  +0.00  ok
needle_in_tools           1.00        0.33  -0.67  REGRESSION
no_hallucinated_tool      1.00        0.67  -0.33  REGRESSION
required_args             1.00        1.00  +0.00  ok
structured_output         1.00        0.33  -0.67  REGRESSION
tool_discrimination       1.00        0.33  -0.67  REGRESSION
tool_permission           1.00        0.67  -0.33  REGRESSION
tool_restraint            1.00        0.67  -0.33  REGRESSION
tool_selection            1.00        0.67  -0.33  REGRESSION

FAIL — capabilities regressed or removed: arg_validity, arity_robustness,
needle_in_tools, no_hallucinated_tool, structured_output, tool_discrimination,
tool_permission, tool_restraint, tool_selection

Here the Q4 quant scores 0.33–0.67 on several capabilities where Q8 scored 1.00. probelock gate exits non-zero when a capability drops past the threshold.

How it differs from promptfoo

promptfoo is a test framework you author. probelock is a lockfile you commit.

1. Probes are derived from your tool schemas. Point it at the OpenAI-style tool definitions your agent already ships, and it generates a fixed, reproducible battery of capability checks. You write no test cases.
2. No LLM judge. Every probe is scored by code: JSON-schema validation, exact match, or a tool-name check. Run it twice on the same model and the numbers match. (promptfoo relies on assertions you write and often on model-graded evals, which vary across runs.)
3. It compares a model against its own baseline, across a model/quant/runtime swap, rather than producing an absolute leaderboard. You only ever compare like with like, on your box, with your tools, so the "benchmarks are gameable/hardware-dependent" objection doesn't apply.

Install & run (only needs uv)

Run it without installing, or install it into the current environment:

uvx probelock --help          # run the latest release
pip install probelock         # or install it

To run an unreleased revision straight from git:

uvx --from git+https://github.com/kelkalot/probelock probelock --help

The examples below use uv run from a checkout of this repo. No model is required for the demo — a deterministic SimulatedClient stands in for two quant levels of the same model:

# from the probelock/ project dir
uv run probelock derive --tools examples/agent_tools.json          # see the probe battery
uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q8.json -o q8.lock
uv run probelock probe  --tools examples/agent_tools.json --simulate fixtures/profile_q4.json -o q4.lock
uv run probelock diff   q8.lock q4.lock
uv run probelock gate   --baseline q8.lock --candidate q4.lock     # exits non-zero

Against a local model, swap --simulate for an OpenAI-compatible endpoint:

uv run probelock probe --tools examples/agent_tools.json \
    --endpoint http://localhost:11434/v1 --model llama3.1:8b-instruct-q4_K_M \
    --quant Q4_K_M --runtime ollama --timeout 120 -o q4.lock

A probe the model rejects (e.g. "model does not support tools") or that times out scores 0 for that capability and the run continues, so a model that can't tool-call still produces a lockfile. An unreachable server, a 404 (wrong model or URL), or a run where every probe fails aborts the run, so a misconfiguration never becomes a poisoned all-zeros baseline.

Providers & frameworks

probelock speaks one protocol — OpenAI /v1/chat/completions with OpenAI-style tools — so anything that exposes it works with --endpoint. For providers that don't (Anthropic, Gemini, …), route through a unified SDK with --via. Every path is deterministic; none of them put an LLM in the loop.

You have…	Use
Ollama, vLLM, llama.cpp server, LM Studio, HF TGI, OpenAI, OpenRouter, Together…	--endpoint <url>/v1 --model <name> (vLLM needs --enable-auto-tool-choice; llama.cpp needs --jinja)
Anthropic / Gemini / Mistral / Bedrock / … (any-llm)	--via anyllm --model anthropic/claude-3-5-sonnet
Any of 100+ providers (litellm SDK)	--via litellm --model anthropic/claude-3-5-sonnet
A running LiteLLM proxy	--endpoint http://litellm:4000/v1 --model <name> (no extra)
In-process HF transformers / MLX (no server)	not yet — add a small Client adapter

pip install 'probelock[anyllm]'   # or 'probelock[litellm]'
probelock probe --tools tools.json --via anyllm --model mistral/mistral-large-latest \
    --samples 5 --temperature 0.7 -o candidate.lock

--via clients reuse the same caching, sampling, and error semantics as --endpoint; they're thin adapters over each SDK's OpenAI-shaped response. Add a new backend by implementing the tiny Client protocol — that's the only seam.

Recorded demo (Ollama)

demo/ has runs against a local Ollama server: a committed qwen3.5:9b baseline vs a gemma3:1b candidate (which does not support tool-calling). See demo/DEMO.md for the transcript, or replay it:

asciinema play demo/probelock-demo.cast   # or: bash demo/demo.sh

The tool-calling capabilities drop 1.00 → 0.00 and the gate exits non-zero. tool_restraint, tool_permission, and no_hallucinated_tool stay 1.00 (a model that can't call tools can't misuse one), and gemma3:1b scores 1.00 on format_adherence vs 0.50 for qwen3.5:9b. The diff is per-capability.

Also committed: qwen3.5:9b vs lfm2.5-thinking:1.2b:

uv run probelock diff demo/qwen3.5-9b.lock demo/lfm2.5-thinking.lock

The 1.2B model matches qwen3.5:9b on tool selection, discrimination, needle_in_tools, arg_validity, required_args, and the three safety probes; structured_output and arity_robustness drop 1.00 → 0.33.

The capabilities (all scored deterministically)

Capability	What it checks	Scorer
tool_selection	Calls the right tool for the task	tool-name match
tool_discrimination	Calls the right tool and no other (picks precisely)	tool-name set
needle_in_tools	Finds the right tool when many (15+) are offered	tool-name match
arg_validity	Emitted args validate against the tool's JSON schema	jsonschema
required_args	All required args present and non-empty	key presence
arity_robustness	Fills every parameter (required + optional) when asked	all-present
structured_output	Emits schema-valid JSON on demand (no tools, no fences)	parse + jsonschema
tool_restraint	Does not call a tool for a task that needs none (over-trigger)	no tool call
tool_permission	Does not call a tool it was explicitly forbidden to use	forbidden tool absent
no_hallucinated_tool	Does not fabricate a call to a tool that wasn't offered	called ⊆ offered
format_adherence	Follows an exact output constraint	exact match

Three are negative probes (a higher score means the bad behavior didn't happen): tool_restraint (over-triggering), tool_permission (calling a forbidden tool), and no_hallucinated_tool (fabricating a tool). All probes are derived from the tool schemas, not hand-authored.

Architecture

tool schemas ──▶ derive probes ──▶ Client ──▶ ResponseMessage ──▶ deterministic scorer ──▶ Lockfile
 (your agent)    (zero authoring)  (model)    (the only model      (no LLM judge)          (commit it)
                                              -touching part)
                                                                    Lockfile + Lockfile ──▶ diff / gate

The only nondeterministic part is the Client; everything else is pure, so the same inputs produce the same lockfile and the same diff. At temperature 0 the client caches identical requests, so the probes that share one request (the tool checks for a given tool) hit the network once. The SimulatedClient crafts correct or incorrect responses that the real scorers grade, so the scoring path runs even with no model present.

Sampling & noisy gates

With one sample per probe, a capability backed by 3 tools quantizes to {0, 0.33, 0.67, 1.0} — a single flip moves it 0.33, far past the default 0.05 gate. So:

• probe --samples N [--temperature T] runs each probe N times and records the pass-rate (raise the temperature for sampling variance).
• gate --confidence 0.95 only fails on a drop that is statistically significant for the recorded trial count (a one-sided two-proportion test). Sub-significant drops are shown as noisy ↓ and do not fail the gate; raise --samples to confirm or clear them.

A total collapse (e.g. 1.00 → 0.00) is significant even at low N; a single-flip 1.00 → 0.67 over 3 trials is noisy until you raise --samples.

In CI

probelock init scaffolds a probelock.tools.json and a .github/workflows/probelock.yml to start from. Commit a baseline lockfile, then gate each candidate:

- run: uvx probelock probe
       --tools tools.json --endpoint $LLM_URL --model $MODEL --samples 5 --temperature 0.7 -o candidate.lock
- run: uvx probelock gate
       --baseline probelock.lock --candidate candidate.lock --max-drop 0.05 --confidence 0.95

A composite GitHub Action (action.yml) wraps those two steps. To show the result on a pull request, render the diff as Markdown (or --format html for a self-contained page):

probelock diff probelock.lock candidate.lock --format markdown >> "$GITHUB_STEP_SUMMARY"

Roadmap

• A --json-mode structured_output probe (response_format) beside the strict prompt path.
• Trend view: compare a capability across more than two lockfiles (a quant ladder Q8→Q5→Q4→Q3).
• In-process backends (HF transformers / MLX) via a small Client adapter, no server required.

License

Apache-2.0 — see LICENSE.

Acknowledgements

Built with Claude Code.