laya-compactor

Keep the evidence. Cut the noise.

Query-aware context compaction for RAG and agents. Score an entire retrieval batch in one local forward pass, then keep only the documents worth sending to your LLM.

Quick start · How it works · Benchmarks · Integrations · Limitations

▶ Watch the 18-second demo with sound · H.264 MP4 · 1080p

Measured on SQuAD: the same exact-match score as full context with 973 average input tokens instead of 3,214. On HotpotQA, it retained 94.5% of answer-bearing documents while using 32% fewer input tokens than full context. See the full methodology and tradeoffs.

Why laya-compactor?

RAG pipelines often retrieve more text than the final model needs. Sending all of it raises token cost and can bury the useful evidence; truncating from the head or tail ignores the query.

laya-compactor makes the cut deliberately:

• Query-aware: each document is scored against the question that triggered retrieval.
• One shared forward pass: the whole batch is scored with laya's predict_batch, rather than one model request per document.
• Local after download: scoring runs with the open-source laya checkpoint; there is no per-compaction API call.
• Verbatim output: documents are either kept whole or removed. Their text is never rewritten or summarized.
• Auditable cuts: every removed document includes its score and a concrete reason, such as a low relevance score or an exhausted token budget.

The project is in early development and publishes its losses alongside its wins. Read Status and limitations before using it in a production path.

Quick start

Install from the repository with Python 3.10 or newer:

git clone https://github.com/Gjusev/laya-compactor.git
cd laya-compactor
uv venv
uv pip install -e .

Compact a retrieval batch to a token budget:

from laya_compactor import compact

query = "Who designed the Eiffel Tower?"
retrieved_docs = [
    "Gustave Eiffel's company designed and built the Eiffel Tower.",
    "The tower was completed in Paris in 1889.",
    "The Berlin Wall fell in 1989.",
]

result = compact(query, retrieved_docs, budget=20)

result.kept   # ScoredDoc objects, most relevant first, text unchanged
result.cut    # CutDoc objects with a human-readable reason
result.stats  # token counts, kept/cut totals and savings percentage

The first run downloads the laya checkpoint from Hugging Face; later runs use the local cache. If the budget must match a provider exactly, pass that provider's tokenizer through token_counter. The dependency-free default is a deterministic word-count proxy.

laya-compact \
  --budget 4000 \
  --query "Who designed the Eiffel Tower?" \
  docs.jsonl

--min-score FLOAT
--rubric default|v2_question_first|v3_needle

How it works

1. Score the batch. Each (query, document) pair receives a continuous score against a four-level rubric: 0 irrelevant, 1 background, 2 relevant and 3 essential. All pairs share one predict_batch call.
2. Rank by relevance. Documents are ordered from highest to lowest score; equal scores preserve the original retrieval order.
3. Apply the policy. Documents below min_score are removed first. The remaining documents are kept verbatim until the token budget is full.
4. Explain every cut. The result records whether each document was removed because of its score or because it no longer fit the budget.

retrieved docs + query
          │
          ▼
  one predict_batch call
          │
          ▼
 relevance-ranked docs ──► min_score ──► token budget
          │                                  │
          └──────── kept verbatim ◄──────────┘
                                             └── cut + reason

compact() also accepts an injected agent, any object exposing predict_batch(states, questions), plus a custom token_counter and rubric variant. This keeps the selection logic testable without loading the real checkpoint.

Measured results

The reported evaluation runs 200 questions from each dataset with the same BM25 retrieval, generator, judge, prompts and 1,000-token budget for all three budgeted policies. Full is the unbudgeted reference. The generator and judge were glm-5.3-flash through Z.ai's OpenAI-compatible API at temperature 0.

The multi-hop result is the important warning: deleting context can remove a bridge document even when it looks secondary in isolation. laya-compactor kept more answer-bearing documents than either truncation baseline, but it still lost three exact-match points against full context on HotpotQA.

Reproduce the evaluation

make smoke  # 3 questions per dataset, no LLM calls
make eval   # 200 questions per dataset; requires OPENAI_API_KEY

OPENAI_BASE_URL can point the harness at another OpenAI-compatible provider. The full run writes rows.jsonl, summary.json and table.md under eval/results/. Metrics include exact match, judge win rate, gold-document retention, token counts, p50 compaction latency and computed cost.

The datasets are HotpotQA distractor validation for multi-hop retrieval and SQuAD validation for single-hop retrieval. Subsets use a fixed seed with a streaming shuffle; pin the datasets package version for bit-exact reruns.

Integrations

Python Core API and JSONL CLI

LangChain Query-aware document compressor

LlamaIndex Node postprocessor

Install the optional integration dependencies:

uv pip install -e ".[integrations]"

from langchain.retrievers import ContextualCompressionRetriever
from laya_compactor.integrations.langchain import LayaCompactor

retriever = ContextualCompressionRetriever(
    base_retriever=your_retriever,
    base_compressor=LayaCompactor(budget=1500),
)

results = retriever.invoke("Who designed the Eiffel Tower?")
# Kept Documents include their score in metadata["laya_score"].

from laya_compactor.integrations.llamaindex import LayaCompactorPostprocessor

query_engine = index.as_query_engine(
    node_postprocessors=[LayaCompactorPostprocessor(budget=1500)],
)

Both wrappers accept the core options: budget, min_score, agent and token_counter. Surviving items remain unchanged and are returned in relevance order.

Rubric sensitivity

Relevance models react to wording, so the repository ships three rubric phrasings and a 100-row, hand-labeled mini-dataset. Each row contains a query, document, gold label from 0 to 3 and rationale.

Measured with the default English laya checkpoint on CPU:

The default remains the default because the question-first phrasing lost 15 percentage points of label agreement on the same data. Re-run the study on the packaged dataset or your own JSONL file:

python -m laya_compactor.sensitivity
python -m laya_compactor.sensitivity my_rows.jsonl

The raw checked-in report is at eval/sensitivity_report.json.

Status and limitations

• Early development: the public API and rubric variants may still change.
• Token counts are configurable: the library default uses a word-count proxy. The benchmark uses tiktoken with cl100k_base for every policy, so the comparison is consistent even though a provider's exact count may differ.
• CPU latency is material: measured p50 compaction time was 6.3 seconds on HotpotQA and 10.0 seconds on SQuAD on the evaluation desktop. Hardware and batch shape affect this substantially.
• Scores are estimates: a document cut at 0.9 may still matter. Tune min_score for the cost of a false negative in your application.
• English first: v0.1 uses laya's default English checkpoint.
• Multi-hop remains hard: isolated relevance scoring can undervalue bridge documents. The HotpotQA benchmark measures that failure mode explicitly.

Development

uv venv
uv pip install -e ".[dev]"
pytest          # fast, offline unit tests; laya is mocked
pytest -m slow  # opt in to loading the real checkpoint

Useful project resources:

• examples/compact_demo.py — runnable eight-document demo.
• docs/how-it-works-animation.html — dependency-free animated explainer.
• docs/comparison-fast-jev.md — architectural comparison with fast-jev-compaction.
• docs/eval-summary.svg — visual benchmark summary used above.
• docs/social-preview.png — repository social preview artwork.

The project follows the same delete, do not rewrite principle as fast-jev-compaction, but targets a different unit of work: retrieved documents instead of agent transcripts. laya-compactor is built on the Apache-2.0-licensed laya decision engine.

License

Apache 2.0. See LICENSE.