A pool of routed models rots quietly. An upstream drops a model, a key hits an account tier, a reasoning model starts spending its whole token budget on thinking, a gateway route begins answering SSE to a non-streaming request. None of that raises an exception anywhere. The agent just gets an empty reply, or a slower one, or falls through to a worse model — and nobody notices for a week.

poolprobe looks at every model in the pool, tells you why each one is failing, and refuses to delete anything until five separate guards agree it is dead. It grew out of running a live cliproxy pool on a schedule and learning, the hard way, how many "dead model" verdicts were really weather.

The three rules

Everything in the tool obeys these. They are the reason it exists.

1. A verdict is not a death sentence. Every failure is classified into a kind — ALIVE, ALIVE-STARVED, PERMANENT, TRANSIENT, CREDENTIAL, MODALITY, UNKNOWN. Only PERMANENT (the upstream genuinely does not know this model) is ever a prune candidate. A 401 is a dead key, not a dead model; pruning on it deletes a good alias because a credential ran dry.

2. One observation is not a verdict. A measured pool produced 544 scheduled liveness runs over six weeks; 39% went red, and every single red was transient. poolprobe burst-confirms by default — five attempts in one run, best answer wins — before it will call anything dead.

3. A number without evidence is not reported. Throughput comes from the gateway's own usage block, or it is marked as a lower bound (>=), or it renders as -. Never estimated from SSE chunk counts and dressed up as a measurement.

Install

pip install poolprobe                 # stdlib engine, generic gateway lane
pip install 'poolprobe[cliproxy]'     # + PyYAML, to read a cliproxy-style config

No PyPI release yet? Install straight from the checkout:

git clone https://gitlab.com/ameobius-ai/poolprobe
cd poolprobe && pip install '.[cliproxy]'

The engine has zero runtime dependencies. PyYAML is optional and only needed to attribute models to providers and to plan config edits from a gateway YAML.

Quickstart

# 1. Is the gateway even up, and what does it publish? (no chat request sent)
poolprobe ping -u http://127.0.0.1:8317/v1 -k "$KEY"

# 2. Probe the whole pool, burst-confirmed, and keep a self-contained HTML report
poolprobe sweep -u http://127.0.0.1:8317/v1 -k "$KEY" --html out.html

# 3. Give it memory: fail streaks across runs are what make `prune` safe
poolprobe sweep -u ... -k "$KEY" --state ~/.cache/poolprobe.json

# 4. Only list what passed all five guards — nothing on disk changes without --apply
poolprobe prune -u ... -k "$KEY" --state ~/.cache/poolprobe.json \
                --adapter cliproxy --config ~/.config/gateway.yaml

# 5. Validate agent fallback chains against the lane rules that actually read them
poolprobe chains ~/.hermes/config.yaml --cross-check -u ... -k "$KEY"

-u also reads POOLPROBE_BASE_URL; -k reads POOLPROBE_API_KEY then OPENAI_API_KEY; --state reads POOLPROBE_STATE. The bearer key is masked in every argv echo — reports are meant to be pasted into issues.

What a sweep looks like

poolprobe 1.0.0  ·  sweep
http://127.0.0.1:8317/v1 · generic adapter (read-only)

5 models  ·  4 alive  ·  0 starved  ·  1 dead  ·  0 prunable  ·  median 5.43s  ·  run #1

VERDICT       MODEL                PROVIDER   LAT      BURST  STREAK  KIND        NOTE
------------  -------------------  ---------  -------  -----  ------  ----------  --------------------
OK            gpt-5.4              -            2.74s  ✓         ok1  -           alive
OK            gemini-3.8-flash     -            6.12s  ✓         ok1  -           alive
OK            grok-4.6             -           27.3s   ✓         ok1  -           alive
ERR           suno-music           -            0.13s  ✗          x1  CREDENTIAL  {"error":{"message"…
OK            deepseek-v4-flash    -            4.73s  ✓         ok1  -           alive

prunable: none — nothing met all five guards (dead, PERMANENT, streak, unprotected, chat modality)

what to do
  CREDENTIAL (1): credential or account level: rotate/top up the key, never prune the model

grok-4.6 answering OK here is the whole point: a prior single-shot check had written it off as dead. The burst said otherwise.

Commands

Common options: --adapter {auto,cliproxy,generic}, --config PATH, --threads N, --timeout S, --max-tokens N, --prompt TEXT, --state PATH, --streak-min N, --protect MODEL (repeatable), --json PATH, --html PATH, --color {auto,always,never}, -q/--quiet.

--max-tokens is a floor, not a cap: a reasoning model spends it on thinking and answers EMPTY-LENGTH, which poolprobe reads as alive but starved, never as dead.

Exit codes

A cron contract. Red is a finding, not noise.

The five prune guards

prune will not name a model unless all five agree. A report that says "prunable" while the prune cycle deletes something else is the worst outcome for a tool like this, so the guards live in one function, checked in one place:

1. dead verdict — EMPTY, ERR, or TIMEOUT (OK/EMPTY-LENGTH never qualify);
2. kind PERMANENT — the upstream does not know this model;
3. fail streak — dead for at least --streak-min consecutive runs (needs --state);
4. not protected — absent from --protect (a namespaced alias's base name matches too);
5. not a generation modality — image/video/tts/music models legitimately fail a chat probe.

Without --state there is no memory, every streak reads 0, and guard 3 can never pass — so prune lists candidates but changes nothing. That is the safe default.

Reading a report

Verdict — what the probe saw:

OK · EMPTY (200, empty body) · EMPTY-LENGTH (empty because the token budget ran out — alive) · ERR (non-2xx) · TIMEOUT.

Kind — what it means:

Three renderers, one report: the terminal table (ANSI, honours NO_COLOR / FORCE_COLOR / TERM=dumb / isatty), a machine-readable --json, and a self-contained --html you can attach to a post. They cannot disagree about what a run recorded.

Library use

from poolprobe import Gateway, sweep, taxonomy

gw = Gateway("http://127.0.0.1:8317/v1", api_key="local")
for result in sweep(gw, ["gpt-5.4", "glm-5.3-flash"], attempts=5):
    print(result.model, result.verdict, result.kind, result.advice)

Development

python -m poolprobe selfcheck   # the whole offline contract; CI runs exactly this

selfcheck needs no gateway, no network and no secrets. If it passes, the taxonomy, the burst "best truth wins" logic, the five prune guards, the argv redaction and all three renderers are behaving.

License

MIT — see LICENSE.