Skip to main content
Backstop — Hard limits. Real control.

In-process token budget enforcement, circuit breaking, and concurrency control for OpenAI and Anthropic SDK clients.

CI Release: v0.8.0 License: MIT Python 3.10+


The Problem

A runaway agent loop will keep calling the LLM until something stops it. Without an in-process guardrail, the first thing that stops it is your credit card limit.

Your options today:

Approach Latency Complexity In-process
Hope the agent stops 0 ms none n/a
Proxy gateway (LiteLLM, etc.) +network hop high ✗
Backstop 0.07 ms p50 one wrap call ✓

30-Second Keyless Proof

No API key needed. This runs entirely offline:

pip install "backstop-ai"
backstop verify

Expected output:

# Backstop Verify — 30-Second Keyless Proof

Mode: offline (100% local mock transport, zero network, zero API keys)
Provider: openai (gpt-4o-mini simulation)

| Metric            | Result             | Notes                                     |
|-------------------|--------------------|-------------------------------------------|
| Allowed calls     | 2                  | Completed within budget (500 tokens)      |
| Blocked calls     | 8                  | Pre-empted before network dispatch        |
| Tokens reserved   | 500                | Enforced budget ceiling                   |
| Tokens saved      | 2,000              | 8 runaway calls prevented                 |
| Exception surfaced| BudgetExceededError| Caught cleanly (subclasses openai.OpenAIError) |
| Wall-clock overhead| 0.08 ms           | Control-path p99 mediation latency        |

## Mechanism Checks
- [PASS] config valid: BackstopConfig() constructed with sane bounds.
- [PASS] wrap pipeline: Backstop.wrap(client) served a mock request end-to-end.
- [PASS] budget block: 8/10 runaway calls blocked at 500-token budget (saved 2,000 tokens).
- [PASS] overhead: control-path overhead p99 = 0.080 ms (direct p99 0.261 ms). Sub-millisecond-class in-process control path.
- [PASS] cache hit: second identical request served from cache (cache_hits=1).
- [PASS] per-agent isolation: agent A blocked 5/5 at its own cap while agent B kept serving — budgets are independent.
- [PASS] hierarchical budgets: parent exhaustion blocks child; sibling unaffected; most-restrictive-wins enforced.
- [PASS] shadow mode: budget exhausted but 0/10 requests blocked; would_block recorded=10 (enabled!=enforced: observations without denial).

Summary: 8 passed, 0 warn, 0 failed, 0 skipped
Status: VERIFIED (real wrap enforcement active)

Everything above the ## Mechanism Checks line is exact except the overhead figure, which is a wall-clock measurement and varies with your host. The check count and pass/fail verdicts are stable. Exit code is 0.

Two flags worth knowing before you rely on verify

  • --offline does nothing. It is accepted and defaults to off, but the runner only ever looks at --live. Passing --offline together with --live still performs the live probe. Omit --live if you want no network.
  • --live resolves the key per provider. --provider anthropic reads ANTHROPIC_API_KEY and --provider openai reads OPENAI_API_KEY; an explicit --api-key-env always wins. Anthropic is probed with x-api-key + anthropic-version, OpenAI with Authorization: Bearer. If --base-url points at a host other than the provider's default, verify prints a warning to stderr naming the destination, because your provider credential is about to be sent there. The probe is a GET /models: it proves the key exists, not that it has scope, quota, or entitlement for a given model.

See the runaway loop demo:

backstop demo
| Calls completed | 10 | 3 | -7 (-70.0%) |
| Calls blocked   | 0  | 7 | +7 (blocked in-process) |
| Tokens consumed | 250| 75| -175 (-70.0%) |

Install

0.8.0 is published — on PyPI as backstop-ai, on npm as backstop-ai, and as the v0.8.0 GitHub Release. pip install gives you that release; the default branch carries unreleased work past it, so the source install below tracks main rather than v0.8.0.

pip install "backstop-ai"             # OpenAI only
pip install "backstop-ai[anthropic]"  # OpenAI + Anthropic

The npm package of the same name is a partial TypeScript port, not the Python distribution. See TypeScript below.

From source

git clone https://github.com/RavaniRoshan/backstop.git
cd backstop
pip install -e ".[anthropic]"

Usage

from openai import OpenAI
from backstop import Backstop, BackstopConfig
from backstop.exceptions import BudgetExceededError

client = Backstop.wrap(
    OpenAI(),
    budget=50_000,          # token ceiling for this session
    config=BackstopConfig(
        initial_concurrency=4,
    ),
)

try:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "Hello"}],
    )
except BudgetExceededError:
    print("Budget hit — no runaway loop possible.")

That's the whole integration. No call-site changes beyond the wrap, and the default path adds no network hop — see What It Does Not Do for the two places where that is not the whole story.

Per-agent isolation

from anthropic import Anthropic
from backstop import Backstop

agent_a = Backstop.wrap(Anthropic(), budget=10_000)
agent_b = Backstop.wrap(Anthropic(), budget=10_000)
# agent_a exhausting its budget does not affect agent_b

Catching budget errors

from backstop.exceptions import BudgetExceededError

try:
    response = client.chat.completions.create(...)
except BudgetExceededError as exc:
    # BudgetExceededError is also a subclass of openai.OpenAIError /
    # anthropic.AnthropicError, so existing error handling still works.
    log.warning("Budget ceiling reached", extra={"tokens_used": exc.tokens_used})

What It Enforces

Feature Description
Token budgets Hard ceiling on total tokens per session/agent
Per-agent isolation Each wrapped client has its own independent budget
Hierarchical budgets Parent budget caps all child agents
Circuit breaking Trips on repeated provider errors
Concurrency control Limits parallel in-flight requests
Priority admission Queues and prioritises across critical / default / background; see below
Retry + backoff Configurable exponential backoff with jitter
Shadow mode Observe-without-enforce for safe rollout
Spend ledger Opt-in. One priced, attributed record per completed request, to a JSONL file or an in-memory ring; see below and docs/ledger.md
Chargeback export backstop ledger export writes a finance-grade CSV that joins to your own revenue data
Runaway-spend detection Velocity, drift, retry amplification and context growth, per attribution key. Shadow by default: reports, never blocks
Dollar budgets A second ceiling, in dollars, on top of the token one. A request is admitted only if it fits under both; see docs/usd-budgets.md

Priority admission

There are exactly three priorities — critical, default and background (backstop.Priority). There is no normal.

The gate queues and prioritises; it does not shed. When the concurrency limit is reached, a request waits for a slot, and a critical request is selected ahead of waiting default and background ones. It does not cancel or discard the lower-priority waiters.

To stop a starved ticket waiting forever, starvation_after_seconds (default 1.0) releases the oldest ticket in any queue once it has waited that long, even if a higher priority is also waiting. Set it to 0 to disable starvation protection, and set queue_timeout to bound how long any request will wait before it gets an error instead.

from backstop import Backstop, BackstopConfig

client = Backstop.wrap(
    OpenAI(),
    budget=50_000,
    config=BackstopConfig(starvation_after_seconds=1.0, queue_timeout=10.0),
)

Dollar budgets

A token ceiling says how much work a session may do. A dollar ceiling says how much it may cost, which is the question a team with a budget actually asked, and the one a token ceiling cannot answer: two models that take the same number of tokens can differ in price twenty times.

from decimal import Decimal

client = Backstop.wrap(
    OpenAI(),
    budget=50_000,                # tokens — unchanged
    budget_usd=Decimal("5.00"),   # dollars
)

It is a second ceiling, not a replacement: a request is admitted only if it fits under both. The money is Decimal end to end — a float is refused, because a float cannot represent most fractions of a dollar exactly, and a ceiling a cent out is a ceiling that lets the wrong request through.

A dollar ceiling needs every request's cost measured, so it turns the spend ledger on. The cost of a dollar ceiling is the cost of the ledger, and the price is stated: docs/usd-budgets.md.

Spend ledger

Opt-in, off by default, and the same one-wrap-call adoption: you add a config field, not a call site. Every completed provider request then produces one priced, attributed record.

from backstop import Backstop, BackstopConfig, attribution

client = Backstop.wrap(
    OpenAI(),
    budget=500_000,
    config=BackstopConfig(ledger_enabled=True, ledger_path="ledger.jsonl"),
)

with attribution(team="payments", feature="checkout-v2"):
    client.chat.completions.create(...)

See the charge-back it produces, with no key and no network:

backstop ledger demo
| team | feature | request_count | input_tokens | output_tokens | total_usd | unpriced_requests | unpriced_components |
| --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
| payments | checkout-v2 | 388 | 8532297 | 278259 | 26.21 | 0 | cache_write |
| payments | refunds | 96 | 520529 | 101931 | 3.40 | 0 | (none) |
| support | triage | 240 | 516754 | 87934 | 1.76 | 0 | (none) |
| search | query-rewrite | 1140 | 741077 | 165354 | 1.65 | 0 | (none) |
| (unattributed) | (unattributed) | 69 | 396310 | 39882 | 1.54 | 0 | (none) |
| search | vendor-preview | 34 | 82143 | 11300 | 0.00 | 34 | (none) |
| **Total** | all | 1967 | 10789110 | 684660 | **34.56** | 34 | cache_write |

That traffic is synthetic and the run is deterministic to the dollar; the prices are real, computed with Decimal from a bundled rate card. Read your own file with backstop ledger show --path ledger.jsonl, or write the CSV with backstop ledger export --path ledger.jsonl --out chargeback.csv --group-by team,feature. The honesty columns — unpriced_requests, estimated_requests, unpriced_components — sit next to the money on purpose: an unattributed dollar is the thing finance most needs to see, so it renders as (unattributed) and never as a blank cell.

It is not free, and the number is measured. With the ledger off (the default) the whole per-request cost is one truth test and no event is built. With it on, a request roughly doubles in cost: +147 µs p50 (+98%) on a 150 µs request, measured on Python 3.12.3/Linux over 7 alternating runs. The request path still never opens a file or a socket. Full numbers, the reproduction script, and the known limitations — including a dated rate card and a silent cache-write under-count on Anthropic's 1-hour TTL — are in docs/ledger.md.


What It Does Not Do

  • The default integration path is not a proxy. When you Backstop.wrap() an SDK client, no network traffic passes through Backstop — the transport is injected in-process and requests go to the provider directly. The package also ships an optional OpenAI-compatible reverse proxy (backstop serve, needs the fastapi extra) for non-Python or hard-to-bypass deployments. That is a different mode with a real network hop; it is not what wrap() does.
  • Not a hosted control plane. There is no central Backstop server and no shared policy service. Multi-tenant routing is in-process only, and each replica holds its own budget unless you opt into the redis extra. There is no central key store, but there is key-adjacent machinery: virtual_keys maps a caller-supplied key to a tenant id, and secret_provider resolves keys to provider secrets at call time. That is indirection over keys you already hold, not custody of them.
  • Not a multi-provider router. Use LiteLLM or similar if you need fallback across providers. Backstop's fallback chain stays within one provider.
  • Not observability storage — but not nothing either. Metrics are still yours to ship: Backstop emits them and you send them to your own backend, with no retention, no query layer and no collector of its own. What changed is that Backstop now has one kind of durable local storage of its own: the opt-in spend ledger writes an append-only NDJSON file, or keeps a bounded in-memory ring, on the machine that made the requests. That is a file, not a system. It is what makes the charge-back above possible, and it is bounded by the disk you gave it and by a buffer that refuses rather than grows. Still absent:
    • No cloud. Nothing is uploaded. There is no Backstop service, no collector, no dashboard that reads your ledger.
    • No multi-tenancy of the ledger. One file per process, whatever path that process chose. Rotation, archival, retention policy and querying are yours. (virtual_keys and per-tenant budgets are about enforcement, not about partitioning storage.)
    • No forecasting of spend. backstop forecast projects budget exhaustion from a measured burn rate, and the transport can tighten the concurrency limit when the rate projects exhaustion inside a horizon. What it does not do is model what the next month will cost: the ledger is a record of what happened, not a model of what will.
    • No cross-customer benchmarks. Nothing aggregates across deployments and there is no published "normal cost per request" for you to compare against. The demo's numbers are a synthetic scenario, not a reference.
    • No framework-native attribution. team / feature / customer come from a ContextVar you scope yourself, or from virtual_keys; nothing reads a LangChain/LlamaIndex/CrewAI run for you. Attribution is exactly what your call sites declare, and an undeclared call site is a (unattributed) row rather than a guess.
    • No enforcement on spend. The detector reports; it cannot block, cancel or kill, and spend avoided is not in the ledger at all — it stays on the enforcement surface (budget_exceeded counters, the audit log's deny records), where it already was.
  • Not LangGraph/CrewAI-native. Works with those frameworks only at the SDK level, not via framework-specific hooks.

SDK Compatibility

Provider SDK range Python Status
OpenAI >=2.37,<4 3.10–3.12 ✅ Supported
Anthropic >=0.98,<2 3.10–3.12 ✅ Supported

CI runs the full matrix (openai 2.37/3.14 × anthropic 0.99/1.6 × python 3.10/3.11/3.12).

Run backstop doctor to check your installed versions:

backstop doctor

What doctor is, precisely: a wrap-and-import smoke test. It constructs mock clients, wraps them, and confirms the HTTP-family detection resolves. It does not send a request through the wrapped transport, so it cannot prove enforcement works — backstop verify is the command that does that.

httpx2 is not a declared dependency; it arrives only as a dependency of newer openai / anthropic majors. doctor handles its absence: on an install resolving to the older SDK family it reports httpx2 not installed; the SDKs in use are on the httpx family and carries on, and still exits 0. On an install where it is present, the httpx2 mock-transport and compat_for checks run as before.

See docs/compatibility.md for the full version matrix and unsupported version notes.


Architecture

Backstop replaces the SDK's internal HTTP transport with a controlled pipeline:

SDK client → BackstopTransport → budget/circuit/concurrency check → original transport → provider

No monkey-patching. No import hooks. Backstop detects which HTTP library the wrapped SDK actually uses — httpx on older SDK majors, httpx2 on openai>=3 / anthropic>=1 — and builds its transport from that same module, because the two are mutually incompatible. No configuration needed; see src/backstop/_httpcompat.py.

One correction worth stating plainly: Backstop does not leave the SDK's retry logic in place. Wrapping sets the wrapped client to max_retries=0, and Backstop's own transport performs the retries, with Backstop's own backoff and circuit state. That is deliberate — the guardrail has to be able to see and stop each individual attempt. The SDK's authentication and request-construction logic does still run above the transport, which is why Backstop's errors propagate to your code as the provider's own error subclasses.

See docs/architecture.md for the full design.


Benchmarks

Control-path overhead, from the committed snapshot docs/benchmark-results-2026-10-20.md — 2,000 requests through a local httpx.MockTransport, no network, at two payload sizes:

Payload Direct p50 Backstop p50 Overhead p50 Overhead p99
94 B 0.1084 ms 0.2056 ms 0.0972 ms 0.1318 ms
4096 B 0.1184 ms 0.2196 ms 0.1012 ms 0.1105 ms

Two payload sizes, deliberately. The 94-byte one is the body the previously published 0.07 ms figure used; the 4 KB one is closer to a real prompt. The overhead is body-proportional — the guardrail reads, parses and inspects every request — so a snapshot that reported only the small body made the guardrail look cheaper than it is.

These are not the old 0.07 ms, and that is the point. The old snapshot recorded the date, the seed, the transport and the request count, and not the host, the OS, the Python version or the SDK versions — so it could not be reproduced on another machine or compared with anything. The new one records all of it. A number nobody can reproduce is not a better number for being smaller.

The conditions of this run: Linux x86_64 (WSL2), CPython 3.12.3, httpx 0.28.1, openai 2.46.0, anthropic 0.116.0, prometheus-client 0.26.0, local mock transport, no network, 2,000 requests per payload. The snapshot also breaks down what the overhead is made of — the body parse, the gauges, the counters, and the ledger's share, which is off by default.

Reproduce the measurement yourself:

backstop benchmark

That prints the same table for your host. Two things about comparability, because the seed only fixes the input sequence, not the outcomes:

  • Latencies are wall-clock and will never match a snapshot.
  • The burst and steady-state counts reproduce exactly. The error-storm and budget-hit counts do not reliably. Both scenarios turn on retry backoff and circuit cooldown, which are wall-clock timers. For example, the snapshot's error-storm row (15 provider calls, 11 successes, 39 circuit-blocked) does not reproduce on a current tree, which reports 12 / 8 / 42 instead. Compare those two rows as indicative, never as a diff.

The same measurement is also available standalone, with no CLI, as PYTHONPATH=src python3 benchmarks/local_overhead.py --requests 1000.

The overhead check inside backstop verify passes below a 5 ms p99 threshold — it proves the control path stayed in the sub-millisecond class, not that it hit any particular figure.

See docs/benchmarks.md.


Documentation

Page What it covers
docs/quickstart.md Install to first guarded request, in six steps
docs/install.md Every install path, including air-gapped
docs/architecture.md How the transport, the ledger and the gate fit together
docs/usd-budgets.md A ceiling in dollars, and what it costs
docs/ledger.md The spend ledger, the charge-back export and the price catalog
docs/detection.md The four runaway-spend detectors
docs/alerts.md Budget alerts and webhooks
docs/frameworks.md The framework adapters
docs/dashboard.md The built-in dashboard
docs/benchmarks.md How the overhead numbers are measured
docs/benchmark-results-2026-10-20.md The current committed overhead snapshot, with its conditions
docs/benchmark-results-2026-07-20.md The superseded snapshot, kept for history
docs/compatibility.md The SDK version matrix, and what is not supported
docs/sdk-matrix.md Which SDK versions CI actually proves
docs/concurrency.md The GIL ceiling, and when to reach for a proxy
docs/threat-model.md Assets, trust boundaries and the privacy posture
observability/prometheus-alerts.yml Ready-made Prometheus alert rules for the metric surface
usecases/ Five deployments end to end, with walkthroughs

Examples

All 13 files in examples/ are listed below. Everything marked Yes runs with no API key and no network; everything marked No needs a real key.

File Description Runs offline
agent_loop_guard.py Runaway loop stopped by a hard budget Yes
anthropic_budget.py Anthropic SDK, mock transport Yes
budget_blocking_demo.py Raw BackstopTransport over httpx.MockTransport Yes
prometheus_metrics.py Starts the metrics server on :9090 (local, no key) Yes
basic.py Minimal OpenAI wrap No
openai_sync.py Synchronous budget guard No
openai_async.py Async budget guard No
anthropic_sync.py Anthropic sync guard No
anthropic_async.py Anthropic async guard No
fastapi_tenants.py Per-tenant budgets (FastAPI) No
background_priority.py Priority admission No
wedge_basic.py Wedge harness, 3 Anthropic runners No
wedge_openai.py Wedge harness, 3 OpenAI runners No

Only agent_loop_guard.py and anthropic_budget.py carry the # KEYLESS header in their docstring. The header is a documentation convention, not a machine-checked property — this table is the authority.

Run the offline examples:

python examples/agent_loop_guard.py
python examples/anthropic_budget.py
python examples/budget_blocking_demo.py

TypeScript (partial port)

backstop-ai on npm is a partial TypeScript port, not a mirror of the Python package:

  • It targets the OpenAI SDK only. No Anthropic support.
  • It intercepts by patching client.chat.completions.create rather than by injecting a transport, so it is a different interception strategy, not the design documented above.
  • It covers roughly half the Python BackstopConfig surface: budget, circuit, retry, fallback chains, cache (exact and semantic), audit, agent guard, admission, AIMD, quota auto-tuning, forecast, request deadlines, hooks. No Redis shared budget, no OTel, no gateway, no shadow mode, no spend ledger, no dollar budgets, no Wedge.
  • It is in CI: .github/workflows/ci.yml runs its typecheck, build and tests on every push, checks that its lockfile and package metadata agree with the Python distribution's version, and the release tag publishes it to npm with provenance.

Use the Python package when you need the full feature set, Anthropic, or the money features. The npm package is a smaller subset, and its README says exactly which parts those are — treat the Python package as the reference.


Contributing

PRs welcome. Run the test suite before opening one:

pytest tests -q
backstop verify

See CHANGELOG.md for release history.


License

MIT — see LICENSE.txt.

Metadata

Release files for backstop-ai 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for backstop-ai 0.8.0
File Size Uploaded Provenance Actions
backstop_ai-0.8.0.tar.gz 562.9 kB Without provenance Details for backstop_ai-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for backstop-ai 0.8.0
File Interpreter ABI Platform Actions
backstop_ai-0.8.0-py3-none-any.whl Python 3 none any Details for backstop_ai-0.8.0-py3-none-any.whl

Total release size: 871.4 kB

Release files / backstop_ai-0.8.0.tar.gz

Download URL backstop_ai-0.8.0.tar.gz
Size 562.9 kB
Tags Source
SHA-256 checksum
How to use checksums
aa618972bb8a2cf08d62b7d36b7c31a6c478110fa66e7fa3232e1d3ac0e60c2f
BLAKE2b-256 checksum
How to use checksums
bc02382cd812bd58e877aa4d06789b03d641f7b15715b6f10f83f55118643ae5
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Release files / backstop_ai-0.8.0-py3-none-any.whl

Download URL backstop_ai-0.8.0-py3-none-any.whl
Size 308.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
4a6de9cc183d80059ef49e104036610365e778ffaa133eaf2184bf6763f8e4b1
BLAKE2b-256 checksum
How to use checksums
a5b22d91914c9b62dc40c2766c5222ceadbf7df4aa91afe8e61b41b220478785
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.14

Security for backstop-ai 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page