fannypack

Fast, reversible tool calls for LLM agents. Your tools strapped on and within reach: the runtime already has the likely call in flight before the model finishes asking for it, and every call it makes is recorded well enough to be taken back afterwards.

Two claims, both measured (current release: 0.3.0):

1. Latency shaving — 1.2–1.8x off a tool-calling turn, taken from the parts of the latency that are the runtime's fault rather than the model's. Bounded by tool latency: with tools under ~50ms there is nothing to shave (a coding belt measured 1.0–1.17x); the gain grows with slower, uneven, or repeated calls.
2. Selective undo — take back one action and leave every action that never depended on it standing. The runtime says which of three things an action is (nothing to undo, reversible, irreversible) and refuses rather than pretends.

Install

pip install fannypack-agents        # the bare name on PyPI is an unrelated package; the import is `fannypack`

or from source:

git clone https://github.com/nikhilkulkarni1755/fannypack-agents && cd fannypack-agents
pip install -e .                    # one dependency: httpx

Python 3.10+. Provider keys go in ANTHROPIC_API_KEY, OPENAI_API_KEY, or FIREWORKS_API_KEY; Ollama needs no key.

Quickstart: a bank agent

Register tools with a decorator. Each one says what it does to the world, and the ones that can be taken back say how.

import asyncio
from fannypack import Agent, Effect, Pack, providers

pack = Pack()
balances = {"acct_alice_0001": 1000.0, "acct_bob_0002": 250.0}
transfers = []

@pack.tool(effect=Effect.READ_ONLY)                      # a read: cacheable, safe to run early
async def get_balance(account_id: str) -> dict:
    """Balance of one account."""
    return {"account_id": account_id, "balance": balances[account_id]}

@pack.tool(effect=Effect.IDEMPOTENT_WRITE)               # a write; the same reference twice moves money once
async def transfer(from_account: str, to_account: str, amount: float, reference: str) -> dict:
    """Move money between accounts."""
    balances[from_account] -= amount
    balances[to_account] += amount
    transfers.append(reference)
    return {"transfer_id": f"txn_{len(transfers):06d}", "from_account": from_account,
            "to_account": to_account, "amount": amount}

@pack.compensator("transfer")                            # this is what makes transfer reversible
async def reverse_transfer(result: dict) -> dict:
    balances[result["to_account"]] -= result["amount"]
    balances[result["from_account"]] += result["amount"]
    return {"reversed": result["transfer_id"]}

@pack.tool(effect=Effect.NON_IDEMPOTENT_WRITE)           # no compensator: the notes are gone
async def withdraw_cash(account_id: str, amount: float) -> dict:
    """Dispense cash at an ATM."""
    balances[account_id] -= amount
    return {"dispensed": amount}

async def main():
    agent = Agent(pack, providers.from_spec("anthropic:claude-sonnet-5"))
    run = await agent.run("Move 100 from acct_alice_0001 to acct_bob_0002 with reference 'rent-sep', "
                          "then confirm both balances.")
    print(run.answer)
    print(run.metrics.as_dict())

asyncio.run(main())

Schemas come from your type hints; Annotated[str, "..."] adds a parameter description. No base classes, no rewrite of your functions.

Selective undo

Suppose the transfer was a mistake.

txn = next(a for a in run.actions() if a.tool == "transfer")

print(run.plan_undo(txn.id).explain())
# undo a_7f2c19 in run_4b81:
#   compensate a_7f2c19 (transfer) [target] -- compensating via reverse_transfer

await run.undo(txn.id)                      # the money is back; the balance reads that
                                            # followed are untouched -- they never depended on it

And suppose the model had withdrawn cash instead:

cash = next(a for a in run.actions() if a.tool == "withdraw_cash")
await run.undo(cash.id)
# Irreversible: withdraw_cash registers no compensation

There is no force flag. Every action resolves to one of three states — nothing to undo (a read), reversible (a write with a compensator), irreversible (a write without one) — and the plan tells you which before anything runs.

A write whose inverse needs the old state — an edit, an overwrite — gets it from a snapshot the runtime takes just before the write and stores on the action:

@pack.snapshotter("write_file")
def before_write(args: dict) -> dict:
    return {"content": files.get(args["path"])}

@pack.compensator("write_file")
def restore(args: dict, before: dict) -> dict:
    files[args["path"]] = before["content"]
    return {"restored": args["path"]}

What makes this selective is the ledger: it records which later calls used which earlier results (a transfer_id returned by one call and passed to another), so undoing one action cascades only through the calls that actually consumed it. run.why(action_id) shows that chain; run.verify() confirms the history was appended to, never rewritten.

Restart from step N

resumed = await agent.resume(run.id, from_action=some_model_turn.id)

The transcript up to that point is rebuilt from the ledger, and every call the original run completed is served back from the record instead of executed again — reads within their freshness window, writes unconditionally. Twelve of twelve resumed bank transfers across four real models moved the money exactly once.

before[] → model → after[]

The policy's first use was to pre-fire calls into the cache; the model still spent a turn asking for them. Options(prefill=True) goes the rest of the way: the calls a request of this shape always starts with run before the first model call, and go into the transcript as a turn already taken — an assistant message that made them, and their results. The model's first real turn starts from after[], the decisions the table doesn't know.

agent = Agent(pack, provider, ledger=ledger, options=Options(policy=True, prefill=True))
agent.warm()                       # learn before[] from past runs
run = await agent.run("I need $50 rn from acct_alice_0001")
# before[]: get_balance(acct_alice_0001) ran while the prompt was being built
# model:    decides withdraw_cash(50) -- writes are always the model's

Mapping "I need $50 rn" to get_balance is the hard part, and it is two deterministic tiers, no model involved, ~100µs: an exact lookup on the request's shape (its tool-vocabulary words), and, failing that, cosine similarity against the words and character trigrams of every past goal that led to each prescription. "Withdraw 20 dollars", "take out 40 in cash" and "I want to withdraw some cash" all taught the same signature; "I need $50 rn" matches it and fires; "what's the weather" does not. Only read-only calls are ever run this way. A wrong match costs one read and is reported. When the table knows a chain (find_sources → read_source($prev.next) → verify_claim), the whole chain runs before the model reads the prompt.

Prefill is a bet -- reads now against model turns later -- and the policy learns both sides of it, so it declines when history says the reads would cost more than the turns they remove (a fast model with slow tools). Measured: 11.3s → 6.3s on a 3.4s-a-turn model, correctly skipped on a 0.45s one. Two known limits: results must be spliced in as tool results (a prose "already done" paragraph is ignored or re-fetched by every model tested), and it should not be combined with plans on a task defined by a step count -- the model loses the count.

The vocabulary

Latency shaving is these flags, each measurable alone:

Options.baseline() turns everything off — that is how most agent loops run today, and what the numbers below compare against. It is an honest baseline: measured equal to a hand-written sequential SDK loop to within 10ms on every shape tested.

Which flags matter for which shape, measured against peer runtimes (OpenAI Agents SDK, Pydantic AI, LangGraph) on a scripted model:

• Repeated reads — the cache is the largest default-on win, 1.27x at 0.5s tools rising to 1.59x at 2s. No peer runtime has one.
• Uneven fan-out (one slow call among fast ones) — stream-ahead wins by 1.05–1.11x; on a homogeneous fan-out every runtime that runs calls concurrently ties within 20ms, and all three peers do.
• Dependent chains — only plans help, and a plan is worth about one model round-trip per step saved, so its value is set by the model's per-turn time: ~2s on Sonnet, less on faster models. Steps now run while the plan is still streaming (0.4–0.9s of tool time under decode on gpt-oss chains); on a scripted model that is a 5-step chain in 1.15s instead of 1.75s.
• Small belts — routing is inert until the pack is larger than route_k (12); it exists for the 50-tool case.

Findings

Three rounds, four models (claude-sonnet-5, claude-haiku-4-5, gpt-5.1, gpt-oss-120b), a 47–50 tool belt, every run correctness-gated so a configuration cannot get faster by doing less. Full tables in docs/results.md, docs/results-2.md and docs/results-3.md; raw samples and ledgers in bench-results/.

Claim 1 — latency shaving, 1.2–1.8x. On independent-call workloads the runtime removes its share: gpt-5.1 fan_out 13.0s → 7.6s, Sonnet 18.0s → 14.4s, the clearest single mechanism being parallel fan-out taking queued from 12.7s to 0. On strictly dependent chains the ordinary mechanisms are neutral — nothing to overlap — and one reach is what moves them: gpt-5.1 chain_8 13.7s → 7.6s, nine model turns to two or three. Pre-dispatch takes a warm research_chain_3 from 11.3s to 5.8s on gpt-oss at 100% precision.

Claim 2 — selective undo. In the suite, undoing a transfer reverses exactly the transfer; undoing a route change restores the route and leaves the brake applied; undoing a cash withdrawal is refused with the reason. Restart from any model turn replays the record: 12/12 exactly-once.

What does not help, measured. compact_state loses on most workloads. Reach-ahead from the goal text wasted about one read per run and is off by default. Routing hurt on Sonnet chains until the routed prefix was frozen and large enough to prompt-cache. Haiku never uses one-reach plans at all. The runtime's own time is 2–3ms a turn, under 8ms at max; the model is where the seconds are.

Round four, the limits (docs/results-4.md). Prefill as a synthesized turn: 11.3s → 6.3s (1.78x) on a 3.4s-a-turn model, and correctly declined on a 0.45s-a-turn one; as a prose paragraph it never won. Plans hold at 16 dependent steps (1.25x, two plans). A 200-tool belt: the baseline fails the task, routing to 12 completes it in 5.9s (3.9x). The query → before[] mapping: 6/6 on paraphrases, 0 false fires on near-misses. Where it breaks: prefill plus plans on a step-counted task loses the count; a model that calls one tool per turn caps every fan-out gain.

Do models need fine-tuning to call tools faster? No, for round-trips — plans and pre-dispatch get that from frontier models by prompting. What a fine-tune would buy, read off the ledgers with fannypack mine: gpt-oss emits sequential-but-independent calls 48% of the time (plans already fix that), Haiku writes ~15 tokens of prose before each call, gpt-5.1 and Sonnet nothing. docs/research.md has the literature.

Run the experiments

fannypack bench --provider anthropic:claude-sonnet-5 --repeats 3 --json out.json
fannypack report out.json
fannypack compare bench-results/*.json

cd suite && uv sync            # the external suite: a bank, a car, a camera, a research chain
uv run fannypack-suite chain  --provider openai:gpt-5.1 --db /tmp/ledgers
uv run fannypack-suite policy --provider fireworks:accounts/fireworks/models/gpt-oss-120b
uv run fannypack-suite mine     /tmp/ledgers/*.db
uv run fannypack-suite classify /tmp/ledgers/*.db

Elsewhere

• As a library: PackServer gives an agent loop you already have the ledger, the cache and selective undo without adopting Agent; Router, ResultCache, Ledger and plan_undo work on their own.
• Over MCP (pip install -e '.[mcp]'): examples/serve_mcp.py serves a pack to Claude or ChatGPT with fannypack_undo and fannypack_why as tools.
• CLI: fannypack inspect | why | graph | verify | undo | policy | mine | classify.
• Not this: a tracing dashboard, a durable execution engine, an agent graph framework, a replacement for MCP, a sandbox. docs/design.md has the reasoning and the known limits.

If you find it useful, a star on the repo helps other people find it.

Apache-2.0.