llm-handover
Find the AI models your code uses, see when each one retires, and test the replacement before you switch.
Guide and a 79-second demo: https://www.volansaquilae.com/llm-handover/
llm-handover is a command-line tool with no dependencies beyond Python 3.10 or newer. It
does three things, separately or in one run:
- Finds every model ID in your code: every ID in its retirement dataset, plus anything shaped like an OpenAI, Anthropic or Google model ID, so newer models show up too.
- Checks each one against retirement dates: on the providers' own APIs and on Amazon Bedrock, Google Cloud and Microsoft Foundry, from a dataset that ships with the tool. No API keys and no network are needed for this.
- Replays your real prompts on the old and the new model and shows what changed: broken JSON, dropped tool calls, new refusals, missing keywords, cut-off answers.
$ llm-handover scan
llm-handover 0.1.0 | dataset 2026-11-18 (bundled) | as of 2026-11-20
Scanned 3 files in . (listed with git).
Found 5 model ID(s) in 5 place(s): 3 with dataset records, 2 without.
RETIRED claude-3-haiku-20240307 [fails]
Claude API: retired on 2026-04-20 (214 days ago) -> claude-haiku-4-5-20251001
Used in: app/classify.ts:5
RETIRED gemini-2.0-flash [fails]
Gemini API: retired on 2026-06-01 (172 days ago) -> gemini-3.6-flash
Vertex AI (Gemini Enterprise Agent Platform): retired on 2026-06-01 (172 days ago) -> gemini-3.5-flash-lite
Used in: app/config.yaml:2
WITHIN 30 DAYS gpt-5 [fails]
OpenAI API: retires on 2026-12-11 (21 days left) -> gpt-6.1-sol
Azure OpenAI (Microsoft Foundry) [gpt-5 (version 2025-08-07)]: retires on 2027-02-09 (81 days left)
Used in: app/support.py:4
No dates on the chosen platforms:
claude-instant-1.2 (not in the dataset): app/classify.ts:4 (low confidence)
gpt-6.1-sol (not in the dataset): app/config.yaml:4
Result: FAILED. 3 model ID(s) at or above --fail-on 30d: claude-3-haiku-20240307, gemini-2.0-flash, gpt-5
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).
Contents
- Install
- Commands
- Finding models:
scan - Looking up dates:
modelsandupdate - Testing a replacement:
replay - All three in one go:
run - The test set format
- What the checks look for
- Providers and API keys
- Output files
- Exit codes
- Continuous integration
- States
- The dataset
- Limits
- How it is tested
- Licence
Install
pipx install llm-handover
Or pip install llm-handover, or uvx llm-handover to run it without installing. From a
copy of this repository: pip install .
Commands
| Command | What it does | Needs keys |
|---|---|---|
llm-handover scan [PATH] |
Steps 1 and 2: find model IDs and check their dates | No |
llm-handover models [QUERY] |
Look up models in the dataset | No |
llm-handover update |
Download the latest dataset | No |
llm-handover replay TESTSET --candidate P:M |
Step 3: compare a new model with an old one | Yes |
llm-handover run [PATH] --tests TESTSET |
Steps 1, 2 and 3 in one go | Yes, for models it tests |
Every command has --help. Models are written as provider:model, where the provider is
openai, anthropic, gemini or openai-compatible: for example openai:gpt-6.1-sol or
anthropic:claude-sonnet-5-5. Only the first colon separates the two, so Bedrock-style IDs
such as openai-compatible:anthropic.claude-3-haiku-20240307-v1:0 work.
Finding models: scan
llm-handover scan # the current folder
llm-handover scan src --fail-on 90d # fail if anything retires within 90 days
llm-handover scan --platform openai-api,bedrock --format json
Which files. In a git work tree, files come from git ls-files --cached --others --exclude-standard, so .gitignore is respected; otherwise the folder is walked. The output
says which was used. Skipped: folders such as .git, node_modules, .venv, __pycache__
and handover-runs; lock files; minified files and source maps; binary files; files over
2 MB (--max-file-size); symbolic links; and documentation files (.md, .mdx, .rst,
.txt, .adoc) unless you pass --include-docs. --exclude GLOB skips more.
What matches. A model ID only matches as a whole token: gpt-5 does not match inside
gpt-5.1 or gpt-5-mini, but it does match inside "openai/gpt-5", "gpt-5:online" or at
the end of a sentence. Bedrock's region prefixes are understood, so us.anthropic.claude-...
finds anthropic.claude-....
Confidence. A match is high confidence when it is quoted ("gpt-5") or assigned to
something named like a model setting (model: gpt-5, OPENAI_MODEL=gpt-5, --model gpt-5),
and low otherwise, as in a comment. Only high-confidence matches can fail a scan. Very short
IDs such as o1 and o3 are only reported with high confidence. --min-confidence high
hides low-confidence matches.
Urgency. Each record gets exactly one label, as of today (UTC) or --as-of:
| Label | Meaning | In --fail-on |
|---|---|---|
| RETIRED | Retired, or the retirement date is today or earlier | retired and up |
| WITHIN 30 DAYS | Retires within 30 days | 30d and up |
| WITHIN 90 DAYS | Retires within 31 to 90 days | 90d and up |
| MAY RETIRE ANY TIME | Only an earliest date was announced, and it has passed | deprecated and up |
| DEPRECATED, NO DATE | Deprecated or legacy, with no date | deprecated and up |
| SCHEDULED | Retires in more than 90 days | any |
| NO DATE | Listed, with no retirement planned | never |
--fail-on takes never, retired, 30d (the default), 90d, deprecated or any. A
model's most urgent record across the chosen platforms decides; --platform limits the
platforms to the ones you use (openai-api, claude-api, gemini-api, bedrock,
vertex-ai, google-cloud, azure-openai, foundry).
Model IDs with no record on the chosen platforms are listed under "No dates on the chosen platforms" and never fail a scan.
Formats. --format text (the default), json (stable, with a schema field),
markdown (for pull request comments) or github (annotations on the lines, for GitHub
Actions).
Looking up dates: models and update
$ llm-handover models gpt-5-mini
2 record(s), dataset 2026-11-18, as of 2026-11-20:
gpt-5-mini-2025-08-07 [openai-api] WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
gpt-5-mini [openai-api] WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).
llm-handover update downloads the latest dataset from
https://www.volansaquilae.com/downloads/llm-model-deprecation-dates.json into your cache
folder. The download is validated in full before anything is saved, the save is atomic, and an
older dataset never replaces a newer one unless you pass --allow-older. Nothing else in
llm-handover downloads the dataset; scan and models work offline. See
The dataset for which copy is used.
Testing a replacement: replay
export OPENAI_API_KEY=...
llm-handover replay cases.jsonl --baseline openai:gpt-5 --candidate openai:gpt-6.1-sol
For each case in the test set, replay gets the old model's answer, asks the new model the
same thing, and compares the two. The old answer comes from:
- the case's recorded
baseline, when it has one (no call is made); - otherwise a live call to
--baseline; --live-baselinecalls--baselineeven for cases with a recorded answer.
Without --baseline, every case needs a recorded answer, and the command refuses to start
if one is missing. Every problem in the test set, and every missing key, is reported before
the first call is made.
Results go to --out (by default handover-runs/<UTC time>), which must be empty unless you
pass --force; with --force only llm-handover's own result files are replaced. Cases run
4 at a time (--concurrency, 1 to 32). Rate limits and temporary server errors (HTTP 408,
425, 429, 500, 502, 503, 504 and 529), timeouts and network errors are retried 3 times
(--retries) with backoff, and a Retry-After header is honoured up to 60 seconds.
The run passes when no case reaches --fail-on (regression, the default, or warning, or
never) and no case failed. A case fails when it could not be judged, for example because
a call returned an error; that counts against the run, because nothing was proven about it.
A skipped case does not.
Stopping. The first Ctrl-C stops new cases from starting, lets the ones in flight finish,
and writes the reports (exit code 130). A second Ctrl-C quits at once; results.jsonl still
holds every case that finished.
All three in one go: run
llm-handover run --tests cases.jsonl
run scans the code, checks the dates, and tests the replacement for every model that is
urgent enough, using the replacement the provider names. Every case in the test set must say
which model it was written for, in "target". Cases are grouped by target, and each group
gets exactly one outcome:
| Outcome | When |
|---|---|
| tested | The model's most urgent record is at or above --risk (default 90d), and the provider names one replacement model, or you chose one |
| not at risk | Its most urgent record is below --risk |
| not in the dataset | The dataset has no record for it on its provider's own platform |
| needs a choice | It is at risk, but the provider names no single replacement (for example "gemini-3.8-flash or gemini-3.5-flash"), or it is behind an OpenAI-compatible endpoint |
| stopped | A Ctrl-C came before the group started |
Nothing is guessed. A group that needs a choice makes the run fail until you make one:
llm-handover run --tests cases.jsonl \
--candidate gemini:gemini-2.5-flash=gemini:gemini-3.8-flash
--candidate OLD=NEW also lets you test a model that is not at risk, or not in the dataset.
Some details:
- For the
openai,anthropicandgeminiproviders, only records for that provider's own API count. Foropenai-compatible, records on every platform count, or only those you choose with--platform. - When the provider writes a note after the replacement, such as
gpt-6.1-sol (reasoning.mode: pro), the model ID before the bracket is tested and the report says the note was not applied. - When the replacement is itself retired, retiring within 90 days or deprecated, the report says so, so you do not migrate twice.
- When the old model is already retired, it cannot be called, so only cases with a recorded
baselineare compared; the others are skipped. A group where every case was skipped is reported as NOTHING TESTED. - Models found in the code that are at risk but have no test cases are listed. With
--fail-on-untested, they and NOTHING TESTED groups fail the run. --fail-on nevermakes the run always pass (Ctrl-C still exits 130).- The run's own results folder and the test set file are never scanned.
Example output:
openai:gpt-5 -> openai:gpt-6.1-sol: DID NOT PASS
4 ok, 1 changed, 1 regressed, 0 failed, 0 skipped
anthropic:claude-sonnet-4-5-20250929 -> anthropic:claude-sonnet-5-5: passed
6 ok, 0 changed, 0 regressed, 0 failed, 0 skipped
openai:gpt-5.1: not at risk
retires on 2027-04-01 (132 days left)
DID NOT PASS. Details: handover-runs/20261120-091500/report.md
The test set format
A test set is a UTF-8 text file with one JSON object per line. examples/testset.jsonl, in the
source package, has one case of each kind.
{"id": "refund-tool-1", "target": "openai:gpt-5", "messages": [{"role": "system", "content": "You are a support agent."}, {"role": "user", "content": "Please refund order A-1042."}], "tools": [{"name": "refund_order", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}}}], "expect": {"tool": "refund_order"}, "baseline": {"text": "", "tool_calls": [{"name": "refund_order", "arguments": {"order_id": "A-1042"}}]}}
| Field | Required | Meaning |
|---|---|---|
id |
yes | 1 to 100 letters, digits, ., _ or -, unique in the file |
messages |
yes | An optional system message first, then user and assistant messages taking turns, starting and ending with user |
target |
for run |
The model the case was written for, as provider:model |
tools |
no | Tools the model may call: name, optional description, and parameters as a JSON Schema object |
tool_choice |
no | auto, none or required; needs tools |
max_tokens |
no | 1 to 200000, default 1024 |
temperature |
no | 0 to 2; left out, the provider's default is used |
expect |
no | Rules both answers are checked against: json (true or false), contains and not_contains (lists of text), regex, tool (a tool name) |
baseline |
no | The old model's recorded answer: text, and optional tool_calls as {"name", "arguments"} objects |
notes |
no | Free text, ignored |
Unknown fields are errors, so a typo such as "expected" cannot quietly turn a check off.
Every problem in the file is reported at once, with line numbers.
What the checks look for
Each check gives a level: regression (likely to break something), warning (worth a look), info or pass. A case is regressed if any check is a regression, changed if any is a warning, and ok otherwise.
| Check | Regression | Warning | Info |
|---|---|---|---|
empty |
The new answer is empty and the old one was not | ||
expect.* |
The old answer met a rule and the new one does not | Both miss it | The new one now meets it |
json |
The old answer was JSON (an object or a list, optionally in a code fence) and the new one is not, or changed between object and list | ||
json.keys |
Top-level keys disappeared | New top-level keys | |
tool_calls |
The new model called no tool, or different tools in a different order | The new model called a tool where the old one did not | |
tool_args |
Argument names disappeared, or the arguments are not valid JSON | New argument names | Same names, different values |
refusal |
The new model refused where the old one answered | The old one refused, the new one answered | |
cut_off |
The new answer stopped at max_tokens |
||
length |
The answer got more than 50% longer or shorter (--length-threshold) |
||
similarity |
The wording changed a lot (--similarity-threshold, default 0.6) |
A refusal is detected when the provider flags one (OpenAI's refusal field, Anthropic's
refusal stop reason, Gemini's safety finish reasons and blocked prompts), or when the
answer opens with refusal wording such as "I can't help with". The wording check, the length
check and the similarity check are heuristics, and the report says so. Similarity compares
characters, not meaning. Read the regressed cases before you decide.
Providers and API keys
Keys are read from environment variables only. They are never written to any file, and they are removed from any error message before it is shown or saved.
| Provider | Key | Base address (optional) | Request |
|---|---|---|---|
openai |
OPENAI_API_KEY |
OPENAI_BASE_URL, default https://api.openai.com/v1 |
POST /chat/completions |
anthropic |
ANTHROPIC_API_KEY |
ANTHROPIC_BASE_URL, default https://api.anthropic.com/v1 |
POST /messages |
gemini |
GEMINI_API_KEY or GOOGLE_API_KEY |
GEMINI_BASE_URL, default https://generativelanguage.googleapis.com/v1beta |
POST /models/MODEL:generateContent |
openai-compatible |
LLM_HANDOVER_COMPAT_API_KEY (optional) |
LLM_HANDOVER_COMPAT_BASE_URL (required) |
POST /chat/completions |
For openai-compatible, the key is sent as Authorization: Bearer KEY, or in another header
named by LLM_HANDOVER_COMPAT_AUTH_HEADER (for example api-key). That covers gateways and
services that speak the OpenAI Chat Completions format: for example a LiteLLM proxy
(http://localhost:4000/v1), OpenRouter (https://openrouter.ai/api/v1) or Azure OpenAI's
v1 API (https://YOUR-RESOURCE.openai.azure.com/openai/v1 with the api-key header). Check
your provider's documentation for its exact base address. Amazon Bedrock and Google Vertex AI
are not called directly in this version; use them through such a gateway.
Addresses must use https. Plain http is accepted for localhost, 127.0.0.1 and ::1, or
anywhere if you set LLM_HANDOVER_ALLOW_HTTP=1. Redirects are never followed, because
following one would send your key to another address; use the final address instead.
Live calls cost money. A run makes one call to the new model per case, plus one to the old model for each case without a recorded baseline.
Output files
replay writes three files to its results folder:
results.jsonl: one line per case, with both answers, every check, and the case's full state history. Lines are written as cases finish, so a crash keeps what was done, and the file is rewritten in test-set order at the end.summary.json: counts, check totals, tokens, median latency and the run's state history ("schema": "llm-handover/replay-summary/1").report.md: what regressed, changed, failed and was skipped.
run writes summary.json ("schema": "llm-handover/run-summary/1"), report.md and
scan.json (the scan it started from, "schema": "llm-handover/scan/1"), plus one folder per
tested group with the three replay files.
The results hold the models' answers, which may contain whatever your prompts contain. Treat them like the prompts.
Exit codes
| Code | Meaning |
|---|---|
| 0 | Passed |
| 1 | Did not pass: a scan finding at or above --fail-on, a replay regression or failed case, or a run that did not pass |
| 2 | Could not run as asked: a bad option, an invalid file, a missing key, or a file that cannot be read or written |
| 3 | An internal error in llm-handover (please report it; --debug shows the details) |
| 130 | Stopped with Ctrl-C |
Continuous integration
The scan needs no keys, so it can run on every pull request and once a week, because dates change even when your code doesn't. In GitHub Actions:
name: Model retirements
on:
pull_request:
schedule:
- cron: "0 7 * * 1"
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-python@v7
with:
python-version: "3.12"
- run: pip install llm-handover
- run: llm-handover update || echo "Using the bundled dates"
- run: llm-handover scan --format github --fail-on 30d
With --format github, findings at or above --fail-on become error annotations, and other
models that are retired, retiring within 90 days or deprecated become warnings, each on its
file and line.
The source package also has action.yml, a composite GitHub Action that runs the same scan
and also writes a table to the job summary. Its inputs: path (default .), fail-on
(default 30d), platform, min-confidence (default low), update-dataset (default
true) and python-version (default 3.12).
Any other CI works the same way: pipx run llm-handover scan --fail-on 30d. To test
replacements in CI, run llm-handover run --tests cases.jsonl with the keys as secrets, and
keep the results folder as a build artefact.
States
Every run, test case and results file moves through a fixed table of states, enforced at run time; any move not in its table raises an error instead of happening. The history of each one is saved in the results, so every result shows how it got there.
| Machine | States and moves |
|---|---|
A run (replay, run, and each tested group inside run) |
created → validated → running → completed; created, validated or running → aborted |
| A test case | pending → baseline → candidate → comparing → done; baseline, candidate or comparing → failed; pending → skipped |
| A results file | open → finished (rewritten in order) or closed (the run did not finish normally) |
The only signal shared between threads is the stop request: set once by the first Ctrl-C, never cleared.
The dataset
The dates come from the LLM model deprecation and retirement dates dataset by Volans Aquilae: one record per model ID and platform, with status, dates, the named replacement and the source page each record was checked against. It is published at https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ under the Creative Commons Attribution 4.0 licence (CC BY 4.0), and a copy ships inside the package.
Which copy is used, in this order:
- a file you pass with
--data; - the copy saved by
llm-handover update, if it is valid and not older than the bundled one; - the bundled copy.
The cache folder is ~/.cache/llm-handover on Linux (or under $XDG_CACHE_HOME),
~/Library/Caches/llm-handover on macOS and %LOCALAPPDATA%\llm-handover\Cache on Windows;
LLM_HANDOVER_CACHE_DIR overrides it. A warning is shown when the dataset in use is more
than 14 days old.
Providers change dates and replacements without much notice. Treat the dataset as a strong
hint and check the provider's page (each record's source_url) before acting on a date.
Limits
- The dataset covers text and reasoning models from OpenAI, Anthropic and Google, on their own APIs, Amazon Bedrock, Google Cloud and Microsoft Foundry. Models outside it are found by their shape but have no dates.
- Finding models by their shape can list strings that are not model IDs, such as an HTML anchor named like a model. Those are only listed, never failed on.
- A model ID built at run time, for example from two strings joined together, cannot be found.
- Test cases are text conversations with optional tool definitions. Tool results, images, audio, files and streaming are not supported in this version.
- Some models reject some settings: reasoning models may refuse a
temperature, for example. The call then fails and the case shows the provider's error, which is often exactly what you need to know before switching. - Models that think before answering count those tokens against
max_tokenson some providers. If new answers come back cut off or empty, raisemax_tokensin the cases. - GitHub shows a limited number of annotations per step, so the action writes at most 10 per model and 50 in all; the text report lists every place.
How it is tested
The test suite covers every command end to end against a local server that imitates the OpenAI, Anthropic and Gemini APIs, through the same HTTP code that talks to the real ones, including authentication headers, retries, rate limits, redirects, oversized responses, timeouts and Ctrl-C. It runs on Python 3.10 to 3.14. Calls to the real provider APIs are not part of the automated tests, because they need paid keys; the request and response formats follow each provider's public API documentation. If something differs on a live API, please open an issue with the error message.
pip install -e . pytest ruff mypy
ruff check . && ruff format --check . && mypy && pytest
Licence
The code is licensed under the Apache License 2.0 (the LICENSE file). The bundled dataset
is © Volans Aquilae, licensed under CC BY 4.0; the NOTICE file has the attribution to keep
if you redistribute it.
Metadata
Release files for llm-handover 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| llm_handover-0.1.0.tar.gz | 120.8 kB | Without provenance | Details for llm_handover-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| llm_handover-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for llm_handover-0.1.0-py3-none-any.whl |
Total release size: 208.4 kB
Release files / llm_handover-0.1.0.tar.gz
| Download URL | llm_handover-0.1.0.tar.gz |
|---|---|
| Size | 120.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ddb5ce32dcafb55e95dbd6d341e19bb8da40814c95c08b0448da659fc2720c83
|
|
BLAKE2b-256 checksum How to use checksums |
d6a19dc62667b88c1815f3f482e7842cd75e9673d5cf8a0f2030e1b3e94590f7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Release files / llm_handover-0.1.0-py3-none-any.whl
| Download URL | llm_handover-0.1.0-py3-none-any.whl |
|---|---|
| Size | 87.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b2132d585b1b17108e6255b8fc024efbcbe6ba3660264417c82812d8afe703f1
|
|
BLAKE2b-256 checksum How to use checksums |
90ece02e4fba7e56007a14e9e206b29ef359cbe4ee23f2d1f8ac0a7f20cc58f1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.10
|
Security for llm-handover 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.