promptbudget

Deterministic context-window budgeting for LLM prompts — with a full audit trail of every token decision.

Your system prompt, tool schemas, chat history and retrieved chunks are all competing for the same context window. Most codebases resolve that fight with a while total > limit: messages.pop(0) loop buried in a helper — which silently evicts the one document the answer depended on, and leaves you no way to find out.

promptbudget makes that allocation an explicit, testable, auditable decision.

from promptbudget import Section, Item, pack

result = pack(
    [
        Section("system", SYSTEM_PROMPT, priority=100, strategy="all_or_nothing"),
        Section("tools", tool_schemas, priority=90, min_tokens=2_000),
        Section("history", messages, priority=50, strategy="drop_oldest"),
        Section("retrieval", chunks, priority=40, strategy="lowest_score_first"),
        Section("scratchpad", notes, priority=10, max_tokens=1_000),
    ],
    budget=128_000,
    reserved_output=4_000,
)

send(result.text)

for d in result.dropped():
    log.info("%s[%d] %s: %s", d.section, d.index, d.action.value, d.reason)

retrieval[7] dropped: lowest relevance score, evicted first to fit the section allocation
scratchpad[0] truncated: cut to the tokens remaining in the section allocation

Why this exists

Existing tools each cover a slice: LangChain's trim_messages trims message lists, LLMLingua compresses text, tiktoken counts tokens. None of them arbitrate a fixed budget across several sections with declared guarantees and tell you what they did.

Install

pip install promptbudget                    # zero dependencies
pip install "promptbudget[tiktoken]"        # exact OpenAI-family token counts
pip install "promptbudget[huggingface]"     # Llama, Mistral, Qwen, Gemma
pip install "promptbudget[anthropic]"       # exact Claude counts via the API

Python 3.9+. The core has no runtime dependencies.

Chat models take messages, not strings

pack() returns one string. No provider SDK accepts one — they all take a list of role-tagged messages, and the roles are not decoration: they drive the chat template, and an assistant turn promoted to user is a different prompt.

pack_chat() budgets the conversation and hands the messages back:

from promptbudget import Message, pack_chat

result = pack_chat(
    system="You are a support agent.",
    context=[Message("user", doc, score=hit.score) for doc, hit in retrieved],
    history=conversation,
    budget=8_000,
    reserved_output=1_000,
    tokens_per_message=4,  # providers bill for message framing too
)

client.chat.completions.create(model="gpt-4o", messages=result.to_openai())

context[1] dropped: lowest relevance score, evicted first to fit the section allocation
history[0] dropped: oldest item, evicted first to fit the section allocation

The layout is opinionated on purpose: the system prompt is atomic and outranks everything, retrieved context outranks history (a stale turn is cheaper to lose than the document holding the answer), and history sheds oldest-first.

The last user message is pinned by default. Evicting the question the model is supposed to answer is never the intent, and it is the easiest thing in this whole problem space to do by accident. Pass pin_last_user=False to opt out.

Already have provider-style dicts? They work as-is:

pack_chat(history=[{"role": "user", "content": "hi"}], budget=4_000)

pack_chat does not reimplement the allocator — it maps messages onto sections, calls the same pack(), and maps the survivors back. Every guarantee below holds for chat packing too, and the property tests assert them twice.

Core ideas

Sections compete; priority decides who wins

Sections are served highest priority first. Ties break on name, so identical inputs always produce an identical prompt — the same reason you pin dependency versions.

Rendering follows your list order, not priority order. Priority decides who gets tokens; you decide what the prompt reads like.

Guarantees are kept or the call fails

Section("tools", tool_schemas, priority=90, min_tokens=2_000)

min_tokens is reserved before any lower-priority section is served — a greedy high-priority section cannot eat tokens promised downstream. If the minimums collectively don't fit, you get BudgetTooSmallError rather than a prompt that quietly violates one:

BudgetTooSmallError: Guaranteed minimums require 6000 tokens but only 4000 are
available (budget=8000, reserved_output=4000). Lower a min_tokens value, raise
the budget, or reduce reserved_output.

max_tokens is the mirror image: a cap, so a runaway history can't crowd out retrieval even when the window is empty.

Reserve room for the reply

pack(sections, budget=128_000, reserved_output=4_000)

Forgetting the model's own output is the most common way to get a context-length error from a prompt that "obviously fits". Reserved tokens are never offered to a section.

Strategies decide what gets shed

Pin anything that must survive:

Item("The user's actual question", pinned=True)

Pinned items are never dropped or cut. If a pinned item genuinely cannot fit, you get SectionOverflowError — a loud failure beats a mangled prompt.

Bring your own by subclassing Strategy:

from promptbudget import Strategy, StrategyResult, register_strategy


class DeduplicateThenTruncate(Strategy):
    name = "dedupe"

    def apply(
        self, section, items, allocated, tokenizer, separator
    ): ...  # return StrategyResult(items, decisions)


register_strategy("dedupe", DeduplicateThenTruncate())

Any tokenizer, or none

from promptbudget import TiktokenTokenizer, HuggingFaceTokenizer, AnthropicTokenizer, pack

pack(sections, budget=128_000, tokenizer=TiktokenTokenizer("gpt-4o"))
pack(sections, budget=32_000, tokenizer=HuggingFaceTokenizer("mistralai/Mistral-7B-v0.1"))
pack(sections, budget=200_000, tokenizer=AnthropicTokenizer(client=anthropic.Anthropic()))
pack(sections, budget=8_000, tokenizer=lambda text: len(text) // 4)  # any str -> int
pack(sections, budget=8_000)  # built-in heuristic

The allocator never imports a tokenizer directly — it talks to the Tokenizer interface, so HuggingFace, a provider SDK, or your own estimator all drop straight in.

The result explains itself

result.text  # the fitted prompt
result.used_tokens  # what it actually costs
result.remaining_tokens  # headroom left
result.complete  # True when nothing was lost
result.dropped()  # every item that lost content, and why
result.section("rag")  # per-section accounting
result.to_json(indent=2)  # structured logging, ready to ship

Because packing is a pure function returning a plain value, you can assert on it:

def test_the_user_question_always_survives():
    result = pack(sections, budget=4_000)
    assert "user_question" not in {d.section for d in result.dropped()}

Guarantees

Verified by property-based tests across hundreds of generated inputs (tests/test_properties.py):

• The output never exceeds the budget.
• Reserved output tokens are never consumed.
• max_tokens caps are never breached.
• Identical inputs produce byte-identical prompts.
• Every item receives exactly one decision — nothing vanishes unrecorded.
• complete never lies about dropped content.
• Chat packing preserves roles, keeps messages in conversation order, and never invents a message that was not supplied.

Development

git clone https://github.com/mathewOracle/promptbudget && cd promptbudget
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest && ruff check . && mypy

Contributions welcome — see CONTRIBUTING.md. New strategies are especially welcome, and shouldn't require touching the allocator at all.

See CHANGELOG.md for release history and RELEASING.md for how versions get published.

License

MIT — see LICENSE.