Skip to main content

llm-handover

Find the AI models your code uses, see when each one retires, and test the replacement before you switch.

Guide and a 79-second demo: https://www.volansaquilae.com/llm-handover/

llm-handover is a command-line tool with no dependencies beyond Python 3.10 or newer. It does three things, separately or in one run:

  1. Finds every model ID in your code: every ID in its retirement dataset, plus anything shaped like an OpenAI, Anthropic or Google model ID, so newer models show up too.
  2. Checks each one against retirement dates: on the providers' own APIs and on Amazon Bedrock, Google Cloud and Microsoft Foundry, from a dataset that ships with the tool. No API keys and no network are needed for this.
  3. Replays your real prompts on the old and the new model and shows what changed: broken JSON, dropped tool calls, new refusals, missing keywords, cut-off answers.
$ llm-handover scan
llm-handover 0.1.0 | dataset 2026-11-18 (bundled) | as of 2026-11-20
Scanned 3 files in . (listed with git).
Found 5 model ID(s) in 5 place(s): 3 with dataset records, 2 without.

RETIRED  claude-3-haiku-20240307  [fails]
  Claude API: retired on 2026-04-20 (214 days ago) -> claude-haiku-4-5-20251001
  Used in: app/classify.ts:5

RETIRED  gemini-2.0-flash  [fails]
  Gemini API: retired on 2026-06-01 (172 days ago) -> gemini-3.6-flash
  Vertex AI (Gemini Enterprise Agent Platform): retired on 2026-06-01 (172 days ago) -> gemini-3.5-flash-lite
  Used in: app/config.yaml:2

WITHIN 30 DAYS  gpt-5  [fails]
  OpenAI API: retires on 2026-12-11 (21 days left) -> gpt-6.1-sol
  Azure OpenAI (Microsoft Foundry) [gpt-5 (version 2025-08-07)]: retires on 2027-02-09 (81 days left)
  Used in: app/support.py:4

No dates on the chosen platforms:
  claude-instant-1.2 (not in the dataset): app/classify.ts:4 (low confidence)
  gpt-6.1-sol (not in the dataset): app/config.yaml:4

Result: FAILED. 3 model ID(s) at or above --fail-on 30d: claude-3-haiku-20240307, gemini-2.0-flash, gpt-5
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).

Contents

Install

pipx install llm-handover

Or pip install llm-handover, or uvx llm-handover to run it without installing. From a copy of this repository: pip install .

Commands

Command What it does Needs keys
llm-handover scan [PATH] Steps 1 and 2: find model IDs and check their dates No
llm-handover models [QUERY] Look up models in the dataset No
llm-handover update Download the latest dataset No
llm-handover replay TESTSET --candidate P:M Step 3: compare a new model with an old one Yes
llm-handover run [PATH] --tests TESTSET Steps 1, 2 and 3 in one go Yes, for models it tests

Every command has --help. Models are written as provider:model, where the provider is openai, anthropic, gemini or openai-compatible: for example openai:gpt-6.1-sol or anthropic:claude-sonnet-5-5. Only the first colon separates the two, so Bedrock-style IDs such as openai-compatible:anthropic.claude-3-haiku-20240307-v1:0 work.

Finding models: scan

llm-handover scan                      # the current folder
llm-handover scan src --fail-on 90d    # fail if anything retires within 90 days
llm-handover scan --platform openai-api,bedrock --format json

Which files. In a git work tree, files come from git ls-files --cached --others --exclude-standard, so .gitignore is respected; otherwise the folder is walked. The output says which was used. Skipped: folders such as .git, node_modules, .venv, __pycache__ and handover-runs; lock files; minified files and source maps; binary files; files over 2 MB (--max-file-size); symbolic links; and documentation files (.md, .mdx, .rst, .txt, .adoc) unless you pass --include-docs. --exclude GLOB skips more.

What matches. A model ID only matches as a whole token: gpt-5 does not match inside gpt-5.1 or gpt-5-mini, but it does match inside "openai/gpt-5", "gpt-5:online" or at the end of a sentence. Bedrock's region prefixes are understood, so us.anthropic.claude-... finds anthropic.claude-....

Confidence. A match is high confidence when it is quoted ("gpt-5") or assigned to something named like a model setting (model: gpt-5, OPENAI_MODEL=gpt-5, --model gpt-5), and low otherwise, as in a comment. Only high-confidence matches can fail a scan. Very short IDs such as o1 and o3 are only reported with high confidence. --min-confidence high hides low-confidence matches.

Urgency. Each record gets exactly one label, as of today (UTC) or --as-of:

Label Meaning In --fail-on
RETIRED Retired, or the retirement date is today or earlier retired and up
WITHIN 30 DAYS Retires within 30 days 30d and up
WITHIN 90 DAYS Retires within 31 to 90 days 90d and up
MAY RETIRE ANY TIME Only an earliest date was announced, and it has passed deprecated and up
DEPRECATED, NO DATE Deprecated or legacy, with no date deprecated and up
SCHEDULED Retires in more than 90 days any
NO DATE Listed, with no retirement planned never

--fail-on takes never, retired, 30d (the default), 90d, deprecated or any. A model's most urgent record across the chosen platforms decides; --platform limits the platforms to the ones you use (openai-api, claude-api, gemini-api, bedrock, vertex-ai, google-cloud, azure-openai, foundry).

Model IDs with no record on the chosen platforms are listed under "No dates on the chosen platforms" and never fail a scan.

Formats. --format text (the default), json (stable, with a schema field), markdown (for pull request comments) or github (annotations on the lines, for GitHub Actions).

Looking up dates: models and update

$ llm-handover models gpt-5-mini
2 record(s), dataset 2026-11-18, as of 2026-11-20:
  gpt-5-mini-2025-08-07  [openai-api]  WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
  gpt-5-mini  [openai-api]  WITHIN 30 DAYS: retires on 2026-12-11 (21 days left) -> gpt-6.1-terra
Dates: Volans Aquilae, https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ (CC BY 4.0).

llm-handover update downloads the latest dataset from https://www.volansaquilae.com/downloads/llm-model-deprecation-dates.json into your cache folder. The download is validated in full before anything is saved, the save is atomic, and an older dataset never replaces a newer one unless you pass --allow-older. Nothing else in llm-handover downloads the dataset; scan and models work offline. See The dataset for which copy is used.

Testing a replacement: replay

export OPENAI_API_KEY=...
llm-handover replay cases.jsonl --baseline openai:gpt-5 --candidate openai:gpt-6.1-sol

For each case in the test set, replay gets the old model's answer, asks the new model the same thing, and compares the two. The old answer comes from:

  • the case's recorded baseline, when it has one (no call is made);
  • otherwise a live call to --baseline;
  • --live-baseline calls --baseline even for cases with a recorded answer.

Without --baseline, every case needs a recorded answer, and the command refuses to start if one is missing. Every problem in the test set, and every missing key, is reported before the first call is made.

Results go to --out (by default handover-runs/<UTC time>), which must be empty unless you pass --force; with --force only llm-handover's own result files are replaced. Cases run 4 at a time (--concurrency, 1 to 32). Rate limits and temporary server errors (HTTP 408, 425, 429, 500, 502, 503, 504 and 529), timeouts and network errors are retried 3 times (--retries) with backoff, and a Retry-After header is honoured up to 60 seconds.

The run passes when no case reaches --fail-on (regression, the default, or warning, or never) and no case failed. A case fails when it could not be judged, for example because a call returned an error; that counts against the run, because nothing was proven about it. A skipped case does not.

Stopping. The first Ctrl-C stops new cases from starting, lets the ones in flight finish, and writes the reports (exit code 130). A second Ctrl-C quits at once; results.jsonl still holds every case that finished.

All three in one go: run

llm-handover run --tests cases.jsonl

run scans the code, checks the dates, and tests the replacement for every model that is urgent enough, using the replacement the provider names. Every case in the test set must say which model it was written for, in "target". Cases are grouped by target, and each group gets exactly one outcome:

Outcome When
tested The model's most urgent record is at or above --risk (default 90d), and the provider names one replacement model, or you chose one
not at risk Its most urgent record is below --risk
not in the dataset The dataset has no record for it on its provider's own platform
needs a choice It is at risk, but the provider names no single replacement (for example "gemini-3.8-flash or gemini-3.5-flash"), or it is behind an OpenAI-compatible endpoint
stopped A Ctrl-C came before the group started

Nothing is guessed. A group that needs a choice makes the run fail until you make one:

llm-handover run --tests cases.jsonl \
  --candidate gemini:gemini-2.5-flash=gemini:gemini-3.8-flash

--candidate OLD=NEW also lets you test a model that is not at risk, or not in the dataset.

Some details:

  • For the openai, anthropic and gemini providers, only records for that provider's own API count. For openai-compatible, records on every platform count, or only those you choose with --platform.
  • When the provider writes a note after the replacement, such as gpt-6.1-sol (reasoning.mode: pro), the model ID before the bracket is tested and the report says the note was not applied.
  • When the replacement is itself retired, retiring within 90 days or deprecated, the report says so, so you do not migrate twice.
  • When the old model is already retired, it cannot be called, so only cases with a recorded baseline are compared; the others are skipped. A group where every case was skipped is reported as NOTHING TESTED.
  • Models found in the code that are at risk but have no test cases are listed. With --fail-on-untested, they and NOTHING TESTED groups fail the run.
  • --fail-on never makes the run always pass (Ctrl-C still exits 130).
  • The run's own results folder and the test set file are never scanned.

Example output:

openai:gpt-5 -> openai:gpt-6.1-sol: DID NOT PASS
  4 ok, 1 changed, 1 regressed, 0 failed, 0 skipped
anthropic:claude-sonnet-4-5-20250929 -> anthropic:claude-sonnet-5-5: passed
  6 ok, 0 changed, 0 regressed, 0 failed, 0 skipped
openai:gpt-5.1: not at risk
  retires on 2027-04-01 (132 days left)
DID NOT PASS. Details: handover-runs/20261120-091500/report.md

The test set format

A test set is a UTF-8 text file with one JSON object per line. examples/testset.jsonl, in the source package, has one case of each kind.

{"id": "refund-tool-1", "target": "openai:gpt-5", "messages": [{"role": "system", "content": "You are a support agent."}, {"role": "user", "content": "Please refund order A-1042."}], "tools": [{"name": "refund_order", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}}}], "expect": {"tool": "refund_order"}, "baseline": {"text": "", "tool_calls": [{"name": "refund_order", "arguments": {"order_id": "A-1042"}}]}}
Field Required Meaning
id yes 1 to 100 letters, digits, ., _ or -, unique in the file
messages yes An optional system message first, then user and assistant messages taking turns, starting and ending with user
target for run The model the case was written for, as provider:model
tools no Tools the model may call: name, optional description, and parameters as a JSON Schema object
tool_choice no auto, none or required; needs tools
max_tokens no 1 to 200000, default 1024
temperature no 0 to 2; left out, the provider's default is used
expect no Rules both answers are checked against: json (true or false), contains and not_contains (lists of text), regex, tool (a tool name)
baseline no The old model's recorded answer: text, and optional tool_calls as {"name", "arguments"} objects
notes no Free text, ignored

Unknown fields are errors, so a typo such as "expected" cannot quietly turn a check off. Every problem in the file is reported at once, with line numbers.

What the checks look for

Each check gives a level: regression (likely to break something), warning (worth a look), info or pass. A case is regressed if any check is a regression, changed if any is a warning, and ok otherwise.

Check Regression Warning Info
empty The new answer is empty and the old one was not
expect.* The old answer met a rule and the new one does not Both miss it The new one now meets it
json The old answer was JSON (an object or a list, optionally in a code fence) and the new one is not, or changed between object and list
json.keys Top-level keys disappeared New top-level keys
tool_calls The new model called no tool, or different tools in a different order The new model called a tool where the old one did not
tool_args Argument names disappeared, or the arguments are not valid JSON New argument names Same names, different values
refusal The new model refused where the old one answered The old one refused, the new one answered
cut_off The new answer stopped at max_tokens
length The answer got more than 50% longer or shorter (--length-threshold)
similarity The wording changed a lot (--similarity-threshold, default 0.6)

A refusal is detected when the provider flags one (OpenAI's refusal field, Anthropic's refusal stop reason, Gemini's safety finish reasons and blocked prompts), or when the answer opens with refusal wording such as "I can't help with". The wording check, the length check and the similarity check are heuristics, and the report says so. Similarity compares characters, not meaning. Read the regressed cases before you decide.

Providers and API keys

Keys are read from environment variables only. They are never written to any file, and they are removed from any error message before it is shown or saved.

Provider Key Base address (optional) Request
openai OPENAI_API_KEY OPENAI_BASE_URL, default https://api.openai.com/v1 POST /chat/completions
anthropic ANTHROPIC_API_KEY ANTHROPIC_BASE_URL, default https://api.anthropic.com/v1 POST /messages
gemini GEMINI_API_KEY or GOOGLE_API_KEY GEMINI_BASE_URL, default https://generativelanguage.googleapis.com/v1beta POST /models/MODEL:generateContent
openai-compatible LLM_HANDOVER_COMPAT_API_KEY (optional) LLM_HANDOVER_COMPAT_BASE_URL (required) POST /chat/completions

For openai-compatible, the key is sent as Authorization: Bearer KEY, or in another header named by LLM_HANDOVER_COMPAT_AUTH_HEADER (for example api-key). That covers gateways and services that speak the OpenAI Chat Completions format: for example a LiteLLM proxy (http://localhost:4000/v1), OpenRouter (https://openrouter.ai/api/v1) or Azure OpenAI's v1 API (https://YOUR-RESOURCE.openai.azure.com/openai/v1 with the api-key header). Check your provider's documentation for its exact base address. Amazon Bedrock and Google Vertex AI are not called directly in this version; use them through such a gateway.

Addresses must use https. Plain http is accepted for localhost, 127.0.0.1 and ::1, or anywhere if you set LLM_HANDOVER_ALLOW_HTTP=1. Redirects are never followed, because following one would send your key to another address; use the final address instead.

Live calls cost money. A run makes one call to the new model per case, plus one to the old model for each case without a recorded baseline.

Output files

replay writes three files to its results folder:

  • results.jsonl: one line per case, with both answers, every check, and the case's full state history. Lines are written as cases finish, so a crash keeps what was done, and the file is rewritten in test-set order at the end.
  • summary.json: counts, check totals, tokens, median latency and the run's state history ("schema": "llm-handover/replay-summary/1").
  • report.md: what regressed, changed, failed and was skipped.

run writes summary.json ("schema": "llm-handover/run-summary/1"), report.md and scan.json (the scan it started from, "schema": "llm-handover/scan/1"), plus one folder per tested group with the three replay files.

The results hold the models' answers, which may contain whatever your prompts contain. Treat them like the prompts.

Exit codes

Code Meaning
0 Passed
1 Did not pass: a scan finding at or above --fail-on, a replay regression or failed case, or a run that did not pass
2 Could not run as asked: a bad option, an invalid file, a missing key, or a file that cannot be read or written
3 An internal error in llm-handover (please report it; --debug shows the details)
130 Stopped with Ctrl-C

Continuous integration

The scan needs no keys, so it can run on every pull request and once a week, because dates change even when your code doesn't. In GitHub Actions:

name: Model retirements
on:
  pull_request:
  schedule:
    - cron: "0 7 * * 1"
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
      - run: pip install llm-handover
      - run: llm-handover update || echo "Using the bundled dates"
      - run: llm-handover scan --format github --fail-on 30d

With --format github, findings at or above --fail-on become error annotations, and other models that are retired, retiring within 90 days or deprecated become warnings, each on its file and line.

The source package also has action.yml, a composite GitHub Action that runs the same scan and also writes a table to the job summary. Its inputs: path (default .), fail-on (default 30d), platform, min-confidence (default low), update-dataset (default true) and python-version (default 3.12).

Any other CI works the same way: pipx run llm-handover scan --fail-on 30d. To test replacements in CI, run llm-handover run --tests cases.jsonl with the keys as secrets, and keep the results folder as a build artefact.

States

Every run, test case and results file moves through a fixed table of states, enforced at run time; any move not in its table raises an error instead of happening. The history of each one is saved in the results, so every result shows how it got there.

Machine States and moves
A run (replay, run, and each tested group inside run) created → validated → running → completed; created, validated or running → aborted
A test case pending → baseline → candidate → comparing → done; baseline, candidate or comparing → failed; pending → skipped
A results file open → finished (rewritten in order) or closed (the run did not finish normally)

The only signal shared between threads is the stop request: set once by the first Ctrl-C, never cleared.

The dataset

The dates come from the LLM model deprecation and retirement dates dataset by Volans Aquilae: one record per model ID and platform, with status, dates, the named replacement and the source page each record was checked against. It is published at https://www.volansaquilae.com/guides/llm-model-deprecation-dates/ under the Creative Commons Attribution 4.0 licence (CC BY 4.0), and a copy ships inside the package.

Which copy is used, in this order:

  1. a file you pass with --data;
  2. the copy saved by llm-handover update, if it is valid and not older than the bundled one;
  3. the bundled copy.

The cache folder is ~/.cache/llm-handover on Linux (or under $XDG_CACHE_HOME), ~/Library/Caches/llm-handover on macOS and %LOCALAPPDATA%\llm-handover\Cache on Windows; LLM_HANDOVER_CACHE_DIR overrides it. A warning is shown when the dataset in use is more than 14 days old.

Providers change dates and replacements without much notice. Treat the dataset as a strong hint and check the provider's page (each record's source_url) before acting on a date.

Limits

  • The dataset covers text and reasoning models from OpenAI, Anthropic and Google, on their own APIs, Amazon Bedrock, Google Cloud and Microsoft Foundry. Models outside it are found by their shape but have no dates.
  • Finding models by their shape can list strings that are not model IDs, such as an HTML anchor named like a model. Those are only listed, never failed on.
  • A model ID built at run time, for example from two strings joined together, cannot be found.
  • Test cases are text conversations with optional tool definitions. Tool results, images, audio, files and streaming are not supported in this version.
  • Some models reject some settings: reasoning models may refuse a temperature, for example. The call then fails and the case shows the provider's error, which is often exactly what you need to know before switching.
  • Models that think before answering count those tokens against max_tokens on some providers. If new answers come back cut off or empty, raise max_tokens in the cases.
  • GitHub shows a limited number of annotations per step, so the action writes at most 10 per model and 50 in all; the text report lists every place.

How it is tested

The test suite covers every command end to end against a local server that imitates the OpenAI, Anthropic and Gemini APIs, through the same HTTP code that talks to the real ones, including authentication headers, retries, rate limits, redirects, oversized responses, timeouts and Ctrl-C. It runs on Python 3.10 to 3.14. Calls to the real provider APIs are not part of the automated tests, because they need paid keys; the request and response formats follow each provider's public API documentation. If something differs on a live API, please open an issue with the error message.

pip install -e . pytest ruff mypy
ruff check . && ruff format --check . && mypy && pytest

Licence

The code is licensed under the Apache License 2.0 (the LICENSE file). The bundled dataset is © Volans Aquilae, licensed under CC BY 4.0; the NOTICE file has the attribution to keep if you redistribute it.

Metadata

Release files for llm-handover 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for llm-handover 0.1.0
File Size Uploaded Provenance Actions
llm_handover-0.1.0.tar.gz 120.8 kB Without provenance Details for llm_handover-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for llm-handover 0.1.0
File Interpreter ABI Platform Actions
llm_handover-0.1.0-py3-none-any.whl Python 3 none any Details for llm_handover-0.1.0-py3-none-any.whl

Total release size: 208.4 kB

Release files / llm_handover-0.1.0.tar.gz

Download URL llm_handover-0.1.0.tar.gz
Size 120.8 kB
Tags Source
SHA-256 checksum
How to use checksums
ddb5ce32dcafb55e95dbd6d341e19bb8da40814c95c08b0448da659fc2720c83
BLAKE2b-256 checksum
How to use checksums
d6a19dc62667b88c1815f3f482e7842cd75e9673d5cf8a0f2030e1b3e94590f7
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Release files / llm_handover-0.1.0-py3-none-any.whl

Download URL llm_handover-0.1.0-py3-none-any.whl
Size 87.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b2132d585b1b17108e6255b8fc024efbcbe6ba3660264417c82812d8afe703f1
BLAKE2b-256 checksum
How to use checksums
90ece02e4fba7e56007a14e9e206b29ef359cbe4ee23f2d1f8ac0a7f20cc58f1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.10

Security for llm-handover 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page