barber-llm 0.1.1

Query-aware context trimming for LLM requests. Your context could use a trim.

barber

Query-aware context trimming for LLM requests.

Your context could use a trim. 33% off the top.

The 30-second pitch

Retrieved context is mostly irrelevant to any single question. Your RAG stack fetches ten paragraphs, the question needs two, and you pay for all ten on every request.

barber embeds the chunks and the question, keeps what matters, and drops the rest. Nothing is rewritten or summarized: chunks survive verbatim or vanish. No model calls at trim time, no required dependencies, deterministic output.

In the published benchmark that locked barber's defaults, that meant 31.8 to 34.1% of context tokens gone with answer quality within noise of full context (numbers below).

Install

pip install barber-llm

The PyPI name is barber-llm; the import name is barber. Zero required dependencies. Extras when you want them:

Extra	Pulls in	Gives you
barber-llm[semantic]	sentence-transformers	semantic scoring, paraphrase-safe
barber-llm[tokens]	tiktoken	exact token counts in TrimResult
barber-llm[eval]	datasets, openai, tiktoken	the barber-eval benchmark harness

Quickstart

Zero-dependency lexical mode, runnable as pasted:

from barber import trim

context = "\n\n".join(f"Passage {i}: facts about topic {i}." for i in range(40))  # your RAG block
messages = [{"role": "user", "content": context},
            {"role": "user", "content": "What do the passages say about topic 7?"}]

result = trim(messages)   # keep=0.6, lexical fallback, zero deps
print(result.tokens_saved, result.chunks_dropped, result.changed)
# result.messages is the same conversation, fewer tokens; send it to your LLM

tokens_saved is signed. Each dropped run costs about 20 tokens of marker, so on blocks with many small, scattered chunks the markers can cost more than the drops save. A negative number is barber telling you this shape is not worth trimming, so skip it or raise keep.

Semantic mode, the configuration the benchmark shipped with:

from barber import trim, embedders

embed = embedders.sentence_transformers("llm-semantic-router/mmbert-embed-32k-2d-matryoshka")
result = trim(messages, keep=0.6, embedder=embed)

The lexical fallback is deterministic and dependency-free, but it matches words, not meaning. For production traffic use the semantic embedder above (32K context window) or BAAI/bge-small-en-v1.5 as the lightweight alternative; the two tied on quality in the published runs. There is also embedders.endpoint(base_url, model, api_key) for any OpenAI compatible /v1/embeddings server (vLLM, TEI), which needs the openai package.

Multi-turn pipelines use the transform form. Pass a shared cache and a block is decided once, then replayed byte-identically on every later turn, so your provider prompt cache stays warm:

from barber import make_transform, Cache

cache = Cache()                          # bounded LRU, safe for a long-running process
name, fn = make_transform(embedder=embed, keep=0.6, cache=cache)
messages, changed = fn(messages)         # call this every turn

A plain dict works too, but it never evicts: one entry per distinct context block for the life of the process. Cache bounds it and is thread-safe, so a threaded server can share one. Two things to size against:

• An entry holds one whole trimmed block, so the bound is a count, not a byte budget. At maxsize=4096 with large RAG blocks that is hundreds of megabytes per process. Lower it if your blocks are big.
• Keep maxsize above your live-conversation count. Evicting a block means the next turn decides it again against the then-current question, which is the prefix churn the cache exists to prevent.

When barber does nothing

trim() returns changed=False and leaves your messages untouched unless some message meets every one of these:

Requirement	Why
Role is user, tool, or function	System and assistant messages are never touched.
Not the latest user message	That message is the question, and the question is never trimmed.
content is a plain string	Content-parts lists ([{"type": "text", ...}]) are skipped.
At least 800 characters	Anything shorter is not retrieved context.
At least 4 chunks	Below that there is nothing to choose between.

The common surprise is the single-message shape. This is a no-op, because the only message present is the question:

messages = [{"role": "user", "content": f"Context:\n{docs}\n\nQuestion: {q}"}]

Give the context its own message and barber has something to work with:

messages = [{"role": "user", "content": docs},
            {"role": "user", "content": q}]

Most APIs accept consecutive user messages. If yours does not, put the context in a tool or function message, which is where retrieved context usually arrives anyway.

The benchmark

barber's defaults were locked by a paired A/B on HotpotQA (distractor config): answer each question with full context and with trimmed context, then have a blind judge grade both answers against the gold reference.

	Medium (~6K tok)	Large (~14K tok)
Tokens saved	31.8%	34.1%
Answer-paragraph retention	100%	100%
Full-context accuracy	97.2%	94.8%
Trimmed-context accuracy	96.0%	95.9%

About 350 judged pairs, MiniMax M3 as the blind judge, answers graded against gold references, keep=0.6. On large contexts trimmed beat full: selection removes the distractors models trip over.

Full write-up: the benchmark post. Full protocol: docs/methodology.md. Reproduce it:

pip install "barber-llm[eval]" && barber-eval --n 200 --keep 0.6 --size large

How it works

1. Query. The latest user message is the question. It is never trimmed.

2. Candidate blocks. Large user, tool, and function messages (800+ chars, 4+ chunks). System and assistant messages are never candidates.

3. Chunk. Split each block on blank lines, --- rules, headings, and [n] citation markers, the boundaries RAG concatenations already have. Falls back to sentences for a single wall of prose.

4. Score. Embed every chunk plus the question, cosine each chunk against the question, keep the top keep fraction.

5. Guards. Pinned chunks bypass the budget entirely, the first and last chunk always survive, and a relevance floor drops pure noise even when the budget would keep it. Details below.

6. Marker. Each dropped run collapses into one line, so the model knows the cut happened and doesn't go looking for missing text:

   [… 4 passage(s) omitted as not relevant to this question — the remaining context is sufficient …]
   

   The assertive wording is deliberate and benchmark-locked: it won our marker ablation. The working theory is that a neutral "lower-relevance passages omitted" invites the model to hedge, while this one tells it to proceed. Re-run the ablation yourself with barber-eval --marker neutral.

Decisions are memoized on the block hash alone, never the query. The first turn to see a block decides it; every later turn replays the decision byte-identically. Your history prefix stays stable and your provider prompt cache keeps hitting.

What barber does not do

• No summarization. Chunks survive verbatim or vanish. A summary is a rewrite, and rewrites can silently invent or lose facts.
• No letter tricks. Dropping vowels, truncating words, gzip-then-base64: in the same benchmark, every character-level scheme cost MORE tokens, not fewer. Letter removal measured 1.34x to 1.69x the tokens; classic compression 3.2x to 3.7x. Tokenizers are already compressors; fighting them backfires. The numbers.
• No history compaction. Providers do that natively now, and re-writing old turns busts their prompt caches. barber targets fresh retrieved context only.
• No model calls at trim time. Scoring is an embedding pass (or pure lexical math). Nothing is sent anywhere unless you opt into the endpoint() embedder.

Guards, or why your answers survive

The failure mode of context pruning is silent: drop the one chunk that held the answer and nothing errors, the model just answers worse. barber ships with every guard on:

• Deontic and PII pinning. Chunks with constraint language ("must", "never", "do not", "don't", "shall not", "prohibited", "required", "only if") or sensitive-data markers (PII, HIPAA, PCI, SSN, password, secret, API key) are never dropped.
• Rare-query-entity pinning. A query term that appears in only one or two chunks of a block is a strong "this chunk answers the question" signal. Those chunks are never dropped, which is what protects multi-hop questions.
• Lead and tail keep. The first and last chunk of every block survive: headers, conclusions, and the lost-in-the-middle mitigation.
• Relevance floor (0.35). A chunk scoring far below the block's top chunk is noise even if the budget has room for it.
• Never touched at all: the latest user message, system messages, assistant messages, non-string content, any message under 800 chars, any block under 4 chunks. See when barber does nothing.

In the benchmark these guards held answer-paragraph retention at 100% in both published sizes (table above).

Evaluate it on your data

Published numbers are an existence proof, not a guarantee for your traffic. The harness that produced them ships in the package:

pip install "barber-llm[eval]"
export JUDGE_API_KEY=sk-...     # any OpenAI compatible judge; MiniMax by default

barber-eval --baseline --n 25   # sanity-check generator + judge first
barber-eval --n 200 --keep 0.6 --size large

The number to gate on is the regression rate: how often trimming turned a right answer wrong. Hold it under 1 to 2%, then take the most aggressive keep that stays under your bar. Tokens saved is the payoff, not the gate. The exact judge prompt ships in barber/eval/JUDGE_PROMPT.md, including a reference-free variant for chat data without gold answers. Judge and generator are swappable via JUDGE_* and GEN_* env vars (details).

Credits

• The MiniMax team: MiniMax M3 was the generator and the blind judge for the entire eval, about 2,000 model calls on well under $20 of credit.
• The vLLM Semantic Router team: their mmbert-embed-32k-2d-matryoshka is the recommended embedder, picked for its 32K window.

---

Want this tier-aware, quality-monitored in production, and billed only on verified savings? That's Nadir.