ModelBump

Dependabot for LLM models. ModelBump finds the AI models your code depends on and warns you before a provider retires them. Later versions will also test the old model against its replacement, repair prompts that regress, and open the upgrade pull request for you.

Status: v0.1.0 ships the first stage, Scan, as a command-line tool and a GitHub Action. The other stages are still being built; see the roadmap.

The problem

Apps built on hosted LLMs (OpenAI, Anthropic, Google, ...) depend on specific model versions, and providers retire those versions on a schedule. When a model is retired, an app either breaks outright or keeps running while quietly producing worse answers, because prompts tuned for the old model don't transfer cleanly to the new one.

A 2026 study of 22,555 migration commits across 17,703 open-source repositories found that:

• 82% of migrations happened after the old model was already shut down (our own re-check on a larger dataset finds a smaller share, but still more than half),
• model identifiers were hard-coded in 94% of migrating apps, and
• only 4% of migrations came with any evaluation suite.

Source: H. L. Kim, "When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications", arXiv:2609.31288, 2026.

Install and scan

Requires Python 3.12 or newer.

pip install modelbump
modelbump scan .

Or run it once without installing, with uv:

uvx modelbump scan .

Each line of the report gives the file, line and column, the model ID, its status (OK, RETIRING_SOON, DEPRECATED, RETIRED or UNKNOWN), the shutdown date with the suggested replacement, and where the ID sits (a call argument, an assignment, a config value, a comment, ...).

Useful options:

Run modelbump scan --help for the rest.

GitHub Action

name: ModelBump
on:
  pull_request:
  schedule:
    - cron: "0 6 * * 1"  # every Monday, to catch new retirement notices

permissions:
  contents: read
  security-events: write  # to upload findings to code scanning

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: imhammad/modelbump@v0.1.0
        with:
          fail-within: "30"
          skip-comments: "true"

All inputs are listed in the scanner documentation.

How well the scanner works

We ran the scanner on 100 real projects, each at the commit just before it moved off a retired model, and labelled a sample of 250 findings by hand (evaluation details):

Replayed on those 100 projects, the scanner flagged 96 before they migrated. The default is tuned not to miss anything; the two flags trade some misses for a much quieter report.

The plan

Development

Requires uv.

git clone https://github.com/imhammad/modelbump.git
cd modelbump
uv sync
uv run modelbump --version
uv run pytest

Roadmap

• Phase 0: Foundations: packaging, CI, pre-commit, branch protection
• Phase 1: Migration dataset + model retirement registry
• Phase 2: Scanner + GitHub Action (v0.1.0)
• Phase 3: Model runner with caching and cost budgets
• Phase 4: Statistical comparison engine
• Phase 5: Regression suite synthesis + LLM-judge calibration
• Phase 6: Prompt repair
• Phase 7: End-to-end modelbump bump + v1.0.0

Research

ModelBump is also a research project: an empirical study of how open-source LLM apps migrate off retired models (paper in preparation), and later a paper on the tool itself. Research notes, protocols and results are in the research section of the docs.

License

Apache License 2.0