hot-context 0.1.0b1

LRU-based intelligent context manager for LLM agents. Keeps hot context, evicts cold.

🔥 hot-context

LRU-based intelligent context manager for LLM agents. Keeps hot context, evicts cold.

Most conversation managers use a naive sliding window — drop the oldest messages when context fills up. hot-context treats your context window like an intelligent cache: every chunk is scored by relevance, recency, and access frequency, and only the highest-value context survives.

---

Key Features

• Relevance-aware trimming — trims tool results to only the lines the LLM actually used, instead of evicting entire entries
• Adaptive token calibration — learns the real chars-per-token ratio from actual LLM feedback, self-corrects over time
• Entity echo feedback loop — uses the LLM's own response as a proxy for attention weights, zero extra API calls
• Turn-aware eviction — never evicts during a turn (LLM needs full context to reason), only trims between turns
• Framework-agnostic core with drop-in adapters for Strands, LangChain, and OpenAI

---

How It Works

Turn N:
  User query → Tools return data → LLM sees FULL context → Answers
                                                              ↓
                                          Entity echo: what did the LLM use?
                                          Trimmer: keep only relevant lines from tool results
                                          Calibrator: learn real token ratio from Bedrock
                                                              ↓
Turn N+1:
  Context has trimmed tool results (5k instead of 80k) + all assistant answers preserved

Scoring formula (no LLM needed):

score = α·similarity(chunk, query) + β·recency(chunk) + γ·access_frequency(chunk) - δ·size_penalty(chunk)

• Similarity: Jaccard overlap on extracted entities (BM25-inspired, zero API calls)
• Recency: Exponential decay by turn distance
• Access frequency: Time-decayed count of how often the LLM actually used this chunk
• Size penalty: Large tool results are penalized proportionally to budget

Topic shift detection: When the user changes topics, similarity weight increases and frequency weight decreases — preventing stale popular chunks from dominating.

---

Results

Tested against Strands SlidingWindowConversationManager on a real multi-turn agent with MCP tools (SQL queries, knowledge base lookups):

Metric	SlidingWindow	hot-context
Q3 (heavy query)	CRASHED — context overflow	✅ Answered (2831 chars)
Peak context Q2	Unbounded (168k+)	105k (trimmed)
Peak context Q3	Overflow (>200k)	113k (controlled)
Queries completed	2.5 of 3	3 of 3

Peak context went DOWN from Q1→Q2 (128k→105k) — the trimmer removed irrelevant lines from Q1's tool result.

---

Quick Start

Installation

# Core — no dependencies
pip install hot-context

# With AWS Strands adapter
pip install hot-context[strands]

# With embedding-based similarity (recommended for production)
pip install hot-context[embeddings]

# Everything
pip install hot-context[all]

Strands (drop-in replacement)

from strands import Agent
from strands.models.bedrock import BedrockModel
from hot_context.adapters.strands import StrandsHotContextManager
from hot_context import ScoringWeights

agent = Agent(
    model=BedrockModel(model_id="us.anthropic.claude-sonnet-4-5-20250929-v1:0"),
    conversation_manager=StrandsHotContextManager(
        token_budget=100_000,
        weights=ScoringWeights.tool_agent(),
    ),
)

Framework-agnostic core

from hot_context import HotContextManager, ScoringWeights

manager = HotContextManager(
    token_budget=100_000,
    weights=ScoringWeights.tool_agent(),
    pin_last_n=2,
)

# Before each query
manager.on_query(user_input)

# After LLM responds
manager.on_response(str(result))

Adaptive calibration (Strands)

# Token estimator self-calibrates from actual Bedrock token counts.
# After each LLM call, feed back the reported input_tokens:
cm = agent.conversation_manager
cm.calibrate(agent.messages, actual_input_tokens=87450)
# Future estimates are now more accurate — no manual tuning needed.

Weight Presets

Use a built-in preset or define your own custom weights:

ScoringWeights.tool_agent()     # high similarity — for agents with many tool calls
ScoringWeights.conversational() # high recency + frequency — for multi-turn chat
ScoringWeights.research()       # high similarity + frequency — for RAG/research agents

# Custom weights — all four components must sum to 1.0
ScoringWeights(
    similarity=0.5,    # how much query-chunk relevance matters
    recency=0.2,       # how much to favor recent turns
    access_freq=0.2,   # how much to favor chunks the LLM has used before
    size_penalty=0.1,  # how much to penalize large chunks
)

Token Budget

token_budget is the maximum tokens reserved for conversation history. It should be sized as:

token_budget = model_context_limit - expected_query_tokens - system_prompt_tokens - tool_definition_tokens

For example, with Model's 200k context window:

HotContextManager(
    token_budget=100_000,  # leaves headroom for prompts, tools, and the response
)

Setting it too high risks context overflow; too low and hot-context will aggressively evict useful history.

pin_last_n

pin_last_n protects the most recent N turns from eviction, ensuring the LLM always has immediate conversational context regardless of score:

HotContextManager(
    pin_last_n=2,  # last 2 turns are never evicted (default)
)

Increase this if your agent relies heavily on recent tool results that shouldn't be trimmed mid-conversation.

Custom Entity Patterns

import re
from hot_context import HotContextManager

manager = HotContextManager(
    extra_patterns=[
        ("order_id", re.compile(r"\b\d{3}-\d{7}-\d{7}\b")),
        ("asin", re.compile(r"\bB[A-Z0-9]{9}\b")),
    ]
)

---

Architecture

┌──────────────────────────────────────────────────────────────┐
│                    HotContextManager                         │
│                                                              │
│  TokenEstimator ──→ EntityExtractor ──→ ContextScorer        │
│       ↑                                      │               │
│  calibrate()                            score + rank         │
│  (from LLM)                                  │               │
│                                              ↓               │
│  Trimmer ←── FeedbackLoop ←── LLM Response ←── Evictor      │
│  (line-level    (entity echo)                (entry-level)   │
│   relevance)                                                 │
└──────────────────────────────────────────────────────────────┘

Adapters: Strands | LangChain | OpenAI

---

Why Not Just Summarize?

Approach	Token cost	Latency	Quality
Sliding window	Free	None	Poor — drops relevant old context
LLM summarization	High	+500ms/turn	Good — but expensive at scale
hot-context	Free	~1ms/turn	Good — keeps what matters, trims what doesn't

---

Components

File	Purpose
manager.py	Core orchestrator: on_query() → score + evict, on_response() → feedback
scorer.py	Scoring formula with topic shift detection and pluggable similarity
token_estimator.py	Adaptive chars-per-token calibration from real LLM feedback
trimmer.py	Line-level relevance trimming of tool results
chunker.py	Smart chunking respecting tool-use/tool-result pairing
entity_extractor.py	Regex-based entity extraction (dates, IDs, numbers, custom patterns)
feedback.py	Entity echo feedback loop — updates access logs from LLM response
types.py	ContextEntry, AccessRecord, ScoringWeights with presets
adapters/strands.py	Drop-in ConversationManager for Strands agents
adapters/langchain.py	BaseChatMemory adapter for LangChain
adapters/openai.py	Message list manager for OpenAI Chat API

---

Roadmap

P2 — Benchmarks & Paper

• Benchmark harness — synthetic multi-turn conversations with known answers. Measure token savings + answer quality (ROUGE/BERTScore on trimmed vs full context)
• Standard benchmarks — LongBench, SCROLLS, QuALITY, NarrativeQA for long-context evaluation
• Ablation studies — trimmer-only vs scorer-only vs full pipeline vs baselines (MemGPT, LLMLingua, attention sink)
• Token savings curves — plot peak context vs conversation length across strategies

---

Contributing

Issues and PRs welcome at github.com/Ankit28P/hot-context.

License

Apache 2.0 — see LICENSE.