Tiltmeter

An early-warning system for the third-party APIs your AI agents depend on.

Volcanologists bolt tiltmeters to the ground to sense tiny deformations long before an eruption. Tiltmeter does the same for the APIs under your agents. It learns the shape of their JSON responses from real traffic, notices when a vendor renames a field, starts sending null or turns a number into a string, and tells you, your CI and the agents themselves before anything goes on tilt.

Typed code crashes loudly when an API changes. Agents don't. They read the new response, fill the gaps with guesses, or retry until the budget is gone.

Highlights

• Agents can check before they act. An MCP server lets Claude Code, Cursor, LangChain or any MCP client call has_drifted before trusting an API, and get_schema to see which fields to expect.
• No spec required. Tiltmeter learns from real responses, so it works with any JSON API, documented or not.
• Breaking or harmless, at a glance. Every change is classified as breaking, warning or info, and a removed field next to a similar new one is flagged as a probable rename.
• Explanations on demand. With an Anthropic API key, Claude explains what changed, the likely cause and what to do about it.
• A CI gate with alerts. Exit codes fail the build on breaking drift. Slack and webhook alerts take one line of config.
• Local and private. One SQLite file and no account. Credentials are expanded at request time and never stored. Response bodies are not stored or sent to the LLM, only their shape.

Quickstart

pip install tiltmeter        # or: uv tool install tiltmeter
tiltmeter init               # writes tiltmeter.yaml with a public example API
tiltmeter check              # the first check records the baseline

Run tiltmeter check again from cron or CI, or leave tiltmeter watch running. This is what you get when a vendor renames phone and turns balance into a string:

$ tiltmeter check
✗ billing_customers  200 · 13 ms  BREAKING · 3 changes
    breaking  type_changed   $.data[*].balance       integer → string
    breaking  field_removed  $.data[*].phone         string → (absent)  possibly renamed to $.data[*].phone_number
    info      field_added    $.data[*].phone_number  (absent) → string  possibly replaces $.data[*].phone
1 endpoint checked · 1 breaking
$ echo $?
2

The getting started guide runs that exact scenario against a fake API on your machine in about five minutes.

Give your agents a tiltmeter

claude mcp add tiltmeter -- tiltmeter serve --config /absolute/path/to/tiltmeter.yaml

Then add a line like this to your agent's instructions:

Before you rely on a third-party API, call has_drifted for it. If it drifted, read the summary and adapt before trusting the response.

Setups for Cursor, Claude Desktop and LangChain agents are in the agents guide.

What counts as drift

Override any of them globally or per endpoint with severity_overrides.

Use it in CI

The baseline lives in .tiltmeter/, so keep that folder between runs. The CI guide has ready-to-copy jobs for GitHub Actions, GitLab CI and cron.

How it works

1. Observe. A check calls the endpoint and turns the JSON body into a shape: for every path, which JSON types appeared, how many objects contained each field and, for fields that look categorical (active, past_due, USD), which values were seen.
2. Compare. The baseline is the merge of the shapes from the last 20 successful checks. Tiltmeter reports only what contradicts it: a missing optional field is fine, a field that was in every object and is now gone is not.
3. Report once. The new shape joins the baseline, so the check that sees a change reports it, and from then on it is the new normal.

The pipeline is a small LangGraph graph that only sequences the steps. The diff itself is plain, deterministic Python with no LLM in the loop. How it works covers the details.

Configuration

endpoints:
  - id: stripe_customers
    url: https://api.stripe.com/v1/customers
    params:
      limit: 10
    headers:
      Authorization: Bearer ${STRIPE_API_KEY}   # expanded from the environment at request time
    ignore_paths:
      - $.data[*].metadata                      # free-form, changes all the time

notify:
  - type: slack
    webhook_url: ${SLACK_WEBHOOK_URL}

Every option is described in the configuration reference, and examples/ has a config you can run as is.

How it compares

• OpenAPI diff tools such as oasdiff compare two versions of a spec. They are the right tool when the vendor publishes and maintains one, and they cannot see changes the vendor did not document.
• Contract testing such as Pact needs the provider to run your contracts, which third-party vendors don't do.
• Uptime and synthetic monitors tell you an API is up. You write and maintain the response assertions yourself.
• Tiltmeter learns the contract from real responses, so it needs nothing from the vendor and catches undocumented changes. It also hands what it learns to your agents.

Privacy and security

• ${VAR} placeholders are expanded right before a request or notification is sent. The values are never written to disk, sent to the LLM or kept in the LangGraph state, and they are scrubbed from error messages.
• The history stores shapes, status codes, latencies and changes, never response bodies unless you set store_samples: true.
• Agents can only check the endpoints in your config. They cannot make Tiltmeter call an arbitrary URL.

SECURITY.md has the full threat model and how to report a vulnerability.

Roadmap

• Auto-fix pull requests. When a vendor ships a breaking change, an agent built on LangChain Deep Agents finds the affected code in your repository and opens a PR.
• Spec and changelog awareness. Diff vendor OpenAPI specs and changelogs next to observed responses.
• Hosted checks and a team dashboard.

Documentation

• Getting started: a five-minute tour on your machine.
• Configuration reference: every option, placeholders and ignore patterns.
• Agents and MCP: Claude Code, Cursor, Claude Desktop and LangChain.
• CI and scheduling: GitHub Actions, GitLab CI and cron.
• How it works: shapes, baselines, the diff rules and the pipeline.
• FAQ and troubleshooting.

Contributing

Bug reports, ideas and pull requests are welcome. CONTRIBUTING.md explains how to set up the project and what a good change looks like. Everyone taking part is expected to follow the code of conduct.

License

MIT