barber-llm 0.1.0

Query-aware context trimming for LLM requests. Your context could use a trim.

barber

Query-aware context trimming for LLM requests.

Your context could use a trim. 33% off the top.

The 30-second pitch

Retrieved context is mostly irrelevant to any single question. Your RAG stack fetches ten paragraphs, the question needs two, and you pay for all ten on every request.

barber embeds the chunks and the question, keeps what matters, and drops the rest. Nothing is rewritten or summarized: chunks survive verbatim or vanish. No model calls at trim time, no required dependencies, deterministic output.

In the published benchmark that locked barber's defaults, that meant 31.8 to 34.1% of context tokens gone with answer quality within noise of full context (numbers below).

Install

pip install barber-llm

The PyPI name is barber-llm; the import name is barber. Zero required dependencies. Extras when you want them:

Extra	Pulls in	Gives you
barber-llm[semantic]	sentence-transformers	semantic scoring, paraphrase-safe
barber-llm[tokens]	tiktoken	exact token counts in TrimResult
barber-llm[eval]	datasets, openai, tiktoken	the barber-eval benchmark harness

Quickstart

Zero-dependency lexical mode:

from barber import trim

result = trim(messages)                  # OpenAI-style list[dict], keep=0.6
send_to_your_llm(result.messages)        # same conversation, fewer tokens
print(result.tokens_saved, result.chunks_dropped, result.changed)

Semantic mode, the configuration the benchmark shipped with:

from barber import trim, embedders

embed = embedders.sentence_transformers("llm-semantic-router/mmbert-embed-32k-2d-matryoshka")
result = trim(messages, keep=0.6, embedder=embed)

The lexical fallback is deterministic and dependency-free, but it matches words, not meaning. For production traffic use the semantic embedder above (32K context window) or BAAI/bge-small-en-v1.5 as the lightweight alternative; the two tied on quality in the published runs. There is also embedders.endpoint(base_url, model, api_key) for any OpenAI compatible /v1/embeddings server (vLLM, TEI), which needs the openai package.

Multi-turn pipelines use the transform form. Pass a shared cache and a block is decided once, then replayed byte-identically on every later turn, so your provider prompt cache stays warm:

from barber import make_transform

name, fn = make_transform(embedder=embed, keep=0.6, cache=shared_dict)
messages, changed = fn(messages)         # call this every turn

The benchmark

barber's defaults were locked by a paired A/B on HotpotQA (distractor config): answer each question with full context and with trimmed context, then have a blind judge grade both answers against the gold reference.

	Medium (~6K tok)	Large (~14K tok)
Tokens saved	31.8%	34.1%
Answer-paragraph retention	100%	100%
Full-context accuracy	97.2%	94.8%
Trimmed-context accuracy	96.0%	95.9%

About 350 judged pairs, MiniMax M3 as the blind judge, answers graded against gold references, keep=0.6. On large contexts trimmed beat full: selection removes the distractors models trip over.

Full write-up: the benchmark post. Full protocol: docs/methodology.md. Reproduce it:

pip install "barber-llm[eval]" && barber-eval --n 200 --keep 0.6 --size large

How it works

1. Query. The latest user message is the question. It is never trimmed.

2. Candidate blocks. Large user, tool, and function messages (800+ chars, 4+ chunks). System and assistant messages are never candidates.

3. Chunk. Split each block on blank lines, --- rules, headings, and [n] citation markers, the boundaries RAG concatenations already have. Falls back to sentences for a single wall of prose.

4. Score. Embed every chunk plus the question, cosine each chunk against the question, keep the top keep fraction.

5. Guards. Pinned chunks bypass the budget entirely, the first and last chunk always survive, and a relevance floor drops pure noise even when the budget would keep it. Details below.

6. Marker. Each dropped run collapses into one line, so the model knows the cut happened and doesn't go looking for missing text:

   [… 4 passage(s) omitted as not relevant to this question — the remaining context is sufficient …]
   

   The assertive wording is deliberate and benchmark-locked. A neutral "lower-relevance passages omitted" primes refusals; this one tells the model to proceed. Re-run that ablation yourself with barber-eval --marker neutral.

Decisions are memoized on the block hash alone, never the query. The first turn to see a block decides it; every later turn replays the decision byte-identically. Your history prefix stays stable and your provider prompt cache keeps hitting.

What barber does not do

• No summarization. Chunks survive verbatim or vanish. A summary is a rewrite, and rewrites can silently invent or lose facts.
• No letter tricks. Dropping vowels, truncating words, gzip-then-base64: in the same benchmark, every character-level scheme cost MORE tokens, not fewer. Letter removal measured 1.34x to 1.69x the tokens; classic compression 3.2x to 3.7x. Tokenizers already are compressors; fighting them backfires. The numbers.
• No history compaction. Providers do that natively now, and re-writing old turns busts their prompt caches. barber targets fresh retrieved context only.
• No model calls at trim time. Scoring is an embedding pass (or pure lexical math). Nothing is sent anywhere unless you opt into the endpoint() embedder.

Guards, or why your answers survive

The failure mode of context pruning is silent: drop the one chunk that held the answer and nothing errors, the model just answers worse. barber ships with every guard on:

• Deontic and PII pinning. Chunks with constraint language ("must", "never", "do not", "prohibited", "required", "only if") or sensitive-data markers (PII, HIPAA, PCI, SSN, password, secret, API key) are never dropped.
• Rare-query-entity pinning. A query term that appears in only one or two chunks of a block is a strong "this chunk answers the question" signal. Those chunks are never dropped, which is what protects multi-hop questions.
• Lead and tail keep. The first and last chunk of every block survive: headers, conclusions, and the lost-in-the-middle mitigation.
• Relevance floor (0.35). A chunk scoring far below the block's top chunk is noise even if the budget has room for it.
• Never touched at all: the latest user message, system messages, assistant messages, any message under 800 chars, any block under 4 chunks.

In the benchmark these guards held answer-paragraph retention at 100% in both published sizes (table above).

Evaluate it on your data

Published numbers are an existence proof, not a guarantee for your traffic. The harness that produced them ships in the package:

pip install "barber-llm[eval]"
export JUDGE_API_KEY=sk-...     # any OpenAI compatible judge; MiniMax by default

barber-eval --baseline --n 25   # sanity-check generator + judge first
barber-eval --n 200 --keep 0.6 --size large

The number to gate on is the regression rate: how often trimming turned a right answer wrong. Hold it under 1 to 2%, then take the most aggressive keep that stays under your bar. Tokens saved is the payoff, not the gate. The exact judge prompt ships in barber/eval/JUDGE_PROMPT.md, including a reference-free variant for chat data without gold answers. Judge and generator are swappable via JUDGE_* and GEN_* env vars (details).

Credits

• The MiniMax team: MiniMax M3 was the generator and the blind judge for the entire eval, about 2,000 model calls on well under $20 of credit.
• The vLLM Semantic Router team: their mmbert-embed-32k-2d-matryoshka is the recommended embedder, picked for its 32K window.

---

Want this tier-aware, quality-monitored in production, and billed only on verified savings? That's Nadir.