since-cutoff

English | 简体中文

Your coding agent learned your libraries before they changed. since-cutoff finds exactly which APIs of your dependency versions it gets wrong, and fixes them with a small AGENTS.md note that is checked by a type checker, not by another LLM.

The problem

Every model has a training cutoff. Your lockfile does not. A few real examples for a model with a July 2025 cutoff (Claude Sonnet 4.5) and current releases, found by since-cutoff scan:

For that sample project, 7 of 9 dependencies had changed their public API after the cutoff (the static diff flags 483 changes; many are internals, which the task writer skips). An agent that learned the old API writes code that fails at import or call time, or, worse, still runs because the old path is only deprecated.

Documentation tools paste whole docs into the context and hope. since-cutoff measures which of those changes your model actually gets wrong, writes only the notes that are needed, and proves on held-out tasks that the notes fix them.

Features

• scan: for every dependency, the version your model saw at its training cutoff vs. the one you pin, and a static diff of what broke in between (no model calls, no API key).
• run: probes the model with short tasks that need the changed APIs and scores its code with a type checker against both versions: stale, wrong, deprecated or correct.
• Verified fixes: one-line AGENTS.md / CLAUDE.md notes, kept only if their example type-checks against your exact version, and re-tested on held-out tasks.
• Works where you are: Claude Code plugin and skill, or any of Anthropic, OpenAI, OpenRouter, DeepSeek, Ollama and OpenAI-compatible servers.
• Every lockfile: uv, Poetry, PDM, pylock, Pipenv, requirements files, or a .venv.
• Safe and reproducible: never runs package or model-written code; everything is cached; full JSON and Markdown reports.

Quick start

# list API changes since your model's cutoff (fast, no model calls)
uvx since-cutoff scan

# probe the model, write verified notes, and apply them to AGENTS.md
uvx since-cutoff run --apply

Or install it with pipx install since-cutoff (or pip install since-cutoff) and run since-cutoff. Run it from your project root (anything with uv.lock, poetry.lock, pdm.lock, pylock.toml, Pipfile.lock, requirements*.txt, pyproject.toml or a .venv).

In Claude Code

/plugin marketplace add MohammadHijjawi97/since-cutoff
/plugin install since-cutoff@since-cutoff

Then ask Claude to "check which of our dependencies you are out of date on", or run /since-cutoff:since-cutoff. The skill runs the CLI; the measuring itself is done by a fresh, tool-less copy of the model, so the agent cannot grade itself.

A real run

Two Claude models on the 9-dependency sample project in examples/agent-app, with Claude Opus 4.6 writing the tasks and notes:

The stronger model is not safer: Opus 4.6 confidently wrote APIs that were removed after its cutoff, including anthropic.HUMAN_PROMPT with client.completions. Stale code from both runs, each valid for the version the model learned and broken for the pinned one: messages.create(temperature=...) (anthropic 1.8), hf_hub_download(resume_download=...), local_dir_use_symlinks=..., force_filename=... and proxies=... (huggingface-hub 2.0), and client.beta.vector_stores (openai 3.x).

The notes it wrote (excerpt, verbatim):

<!-- since-cutoff:start -->
## Library changes after the model's training cutoff

**anthropic 1.8.0**
- `temperature=...` was removed from `messages.create()` in anthropic 1.8.0. Omit the `temperature` parameter entirely; there is no replacement.

**huggingface-hub 2.0.0**
- `hf_hub_download(..., resume_download=True)`: The `resume_download` parameter was removed in huggingface-hub 2.0.0. Omit it; downloads resume automatically.

**openai 3.19.2**
- `client.beta.vector_stores` is removed in openai 3.19.2. Use `client.vector_stores` instead.
<!-- since-cutoff:end -->

Small samples, two models, one project: treat it as a demonstration, not a benchmark. The full report (every task, answer and type-checker error) is what since-cutoff run writes to .since-cutoff/report.md. To reproduce: cd examples/agent-app && since-cutoff run --model claude-code:claude-haiku-4-5 --task-model claude-code:claude-opus-4-6.

Models

Training cutoffs come from models.dev (a snapshot is bundled for offline use). since-cutoff models sonnet lists them; --cutoff 2025-07 overrides.

How it works

flowchart LR
  L[lockfile] --> V[version at the model's cutoff<br/>vs your version]
  V --> D[static API diff<br/>griffe]
  D --> T[short tasks that need<br/>the changed API]
  T --> M[model answers<br/>no tools, no docs]
  M --> C[basedpyright against<br/>BOTH versions]
  C --> N[notes, verified<br/>by the type checker]
  N --> H[held-out tasks<br/>with vs without notes]

Everything is scored by a type checker against the exact package versions, each in an isolated environment with that package's own runtime dependencies. No LLM judges anything, and every number traces back to results.json. Details: docs/how-it-works.md.

What it runs, sends and fetches

• Fetches package metadata and wheels from PyPI and model cutoffs from models.dev (a snapshot is bundled for offline use).
• Sends prompts only to the model provider you choose (run only; scan sends nothing). Prompts contain package names, versions, public signatures and docstrings of the changed APIs, the generated tasks and, for notes, the model's own answer. Never your source code.
• Runs basedpyright locally on the model's answers. It never executes them.
• Writes .since-cutoff/ in your project, its cache (since-cutoff cache path) and, with --apply, one marked block in AGENTS.md/CLAUDE.md. No telemetry.

Safe by design

• Never executes code. Package code is read statically (griffe with inspection off; only .py/.pyi files are extracted, with path and size checks). Model-written code is only type-checked.
• Writes almost nothing. Only .since-cutoff/ (which ignores itself in git) and, with --apply, one marked block in AGENTS.md/CLAUDE.md. Everything else in that file is left byte-for-byte unchanged.
• Stays on PyPI. Git, path, workspace and private-index dependencies are never looked up on public PyPI by name.
• Local and cached. No telemetry. PyPI data, diffs, tasks and answers are cached, so re-runs are free and reproducible (--fresh asks the model again).

Use in CI

since-cutoff run --quick --fail-on-stale --json > since-cutoff.json

Exit codes: 0 ok, 1 error (including "no model answer could be scored"), 2 usage error, 3 stale API use found with --fail-on-stale.

Limitations

• Python only for now. TypeScript (.d.ts diffs, tsc) is next.
• A type checker sees wrong names, wrong parameters and PEP 702 deprecations. It cannot see behaviour changes behind an unchanged signature, or deprecations that only warn at run time.
• Probes cover a ranked sample of the breaking changes (symbols your code already uses first), not all of them.
• "The version the model saw" is the newest release on or before the cutoff date. Models know recent releases less well, so real staleness can start earlier.
• Held-out tasks are paraphrases of the same change: they show that a note fixes that change, not that the model got better in general.

Related work

• Context7 and similar tools retrieve current docs at answer time. since-cutoff is complementary: it measures what is actually wrong and keeps a small, verified note in the repo.
• cutoff probes a library you maintain; postcut pastes changelogs since the cutoff.
• Built on griffe, basedpyright, models.dev and rich.

Contributing

Issues and pull requests are welcome; see CONTRIBUTING.md. The offline test suite runs the whole pipeline with a toy library and a scripted model, so no API key is needed.

Citation

If you use since-cutoff in research, please cite it (see CITATION.cff).

License

MIT © Mohammad Hijjawi