contextpilot 0.3.1

Efficient Retrieval-Augmented Generation with Accuracy-Preserving Context Reuse

ContextPilot: Efficient Long Context Inference with Context Reuse

---

| Documentation | Examples | Benchmarks |

News

• [2026/01] ContextPilot has been accepted to MLSys 2026 🎉! See you in Bellevue, WA, USA.
• [2026/01] Code is released!

About

ContextPilot is a fast optimization system on context engineering layer for agentic workloads:

1. High Throughput: Boosting prefill throughput and prefix cache hit ratio with intelligent context reuse.
2. Accuracy Preserved: Accuracy loss is negligible and even improved!
3. Strong Compatibility: Strong compatibility with existing popular RAG libraries (PageIndex), Agentic memory layer (Mem0), KV cache optimization engine (LMCache), and Inference engines (vLLM and SGLang). Both single-node and multi-node deployment!
4. Widely Tested: Tested with a wide range of RAG and Agentic AI applications.

Target Workloads

1. Trending Topic QA with Retrieval — Search and generation for breaking news and hot topics beyond model knowledge
2. Closed-Domain Long-Context QA — Retrieval-augmented QA over specialized corpora (novels, financial reports, legal documents)
3. Multi-Turn Conversations with Long-Term Memory — Persistent context across sessions (e.g. Mem0)

Benchmark and Performance

System Performance

ContextPilot on DeepSeek-R1 maintains accuracy compared to SGLang, achieving 64.68% vs 64.15% F1 on MultihopRAG and 41.08% vs 40.20% F1 on NarrativeQA.

Accuracy on MT-RAG Benchmark

Method	Qwen3-4B	Llama3.1-8B	Qwen3-30B-A3B
LMCache	62.56	68.46	75.12
CacheBlend	50.33	56.52	X
RadixCache	62.56	68.46	75.12
ContextPilot	64.27	68.12	75.81

ContextPilot delivers 4-13x improvements in cache hit rates and 1.5-3.5x reductions in prefill latency for large-batch RAG workloads, while maintaining or improving accuracy.

Furthermore, ContextPilot has been tested to reduce input token costs by around 36% with GPT-5.2.

See Benchmarks in the documentation for GPU vs CPU performance analysis and detailed benchmark methodology.

Getting Started

Installation

Requirements: Python >= 3.10

pip install contextpilot

Or install from source:

git clone https://github.com/SecretSettler/ContextPilot.git
cd ContextPilot
pip install -e .

More detailed installation instructions are available in the docs.

Documentation

Check out the ContextPilot documentation for comprehensive guides.

Examples

Go hands-on with our examples, demonstrating how to address different use cases with ContextPilot.

Contributing

We welcome and value all contributions! Please feel free to submit issues and pull requests.

Citation

We will include the paper citation soon!