agent-context-manager

A lightweight Python library for managing LLM context windows in AI agents. Prevents context overflow, reduces token costs, and maintains conversation coherence.

The Problem

AI agents face a critical challenge: context windows fill up fast. When they overflow:

• Costs explode - token usage grows exponentially
• Performance degrades - LLMs struggle with long contexts ("lost in the middle")
• Coherence breaks - agents forget important context while keeping noise

Current solutions are either too complex (require LLM calls for summarization) or too naive (just truncate old messages).

The Solution

agent-context-manager provides intelligent context compression without requiring additional LLM calls:

• Token-aware management - Track usage, warn before overflow
• Multiple compression strategies - Choose what fits your use case
• Framework agnostic - Works with any LLM provider
• Zero LLM dependencies - No API calls needed for compression

Installation

pip install agent-context-manager

Quick Start

from agent_context_manager import ContextManager, SlidingWindowStrategy

# Create a context manager with 8K token limit
manager = ContextManager(
    max_tokens=8000,
    strategy=SlidingWindowStrategy(keep_system=True, keep_recent=10)
)

# Add messages as your agent works
manager.add_message({"role": "system", "content": "You are a helpful assistant."})
manager.add_message({"role": "user", "content": "Hello!"})
manager.add_message({"role": "assistant", "content": "Hi there!"})

# Get compressed context when needed
context = manager.get_context()

# Check token usage
print(f"Tokens used: {manager.token_count}/{manager.max_tokens}")

Compression Strategies

1. Sliding Window (Default)

Keeps the most recent N messages, always preserving system messages.

from agent_context_manager import SlidingWindowStrategy

strategy = SlidingWindowStrategy(
    keep_system=True,      # Always keep system messages
    keep_recent=20,        # Keep last 20 messages
    keep_first_user=True   # Keep the original user request
)

2. Importance Scoring

Scores messages by relevance and keeps the most important ones.

from agent_context_manager import ImportanceStrategy

strategy = ImportanceStrategy(
    system_weight=1.0,     # System messages always kept
    user_weight=0.8,       # User messages high priority
    assistant_weight=0.6,  # Assistant messages medium priority
    tool_weight=0.4,       # Tool results lower priority
    recency_decay=0.95     # Recent messages score higher
)

3. Semantic Deduplication

Removes near-duplicate messages to reduce redundancy.

from agent_context_manager import DeduplicationStrategy

strategy = DeduplicationStrategy(
    similarity_threshold=0.85,  # Remove if >85% similar
    keep_latest=True            # Keep the most recent of duplicates
)

4. Hybrid (Recommended for Production)

Combines multiple strategies for best results.

from agent_context_manager import HybridStrategy

strategy = HybridStrategy([
    DeduplicationStrategy(similarity_threshold=0.9),
    ImportanceStrategy(recency_decay=0.95),
    SlidingWindowStrategy(keep_recent=50)
])

Token Counting

Built-in token counting for popular models:

from agent_context_manager import ContextManager

# Auto-detect tokenizer based on model
manager = ContextManager(max_tokens=8000, model="gpt-4")
manager = ContextManager(max_tokens=100000, model="claude-3")

# Or use a custom tokenizer
manager = ContextManager(
    max_tokens=8000,
    tokenizer=my_custom_tokenizer
)

Overflow Handling

from agent_context_manager import ContextManager, OverflowPolicy

manager = ContextManager(
    max_tokens=8000,
    overflow_policy=OverflowPolicy.COMPRESS,  # Auto-compress when near limit
    overflow_threshold=0.9  # Compress at 90% capacity
)

# Or get warnings instead
manager = ContextManager(
    max_tokens=8000,
    overflow_policy=OverflowPolicy.WARN
)

# Check status
if manager.is_near_overflow():
    print(f"Warning: {manager.usage_percent}% of context used")

Memory Blocks (Structured Context)

Organize context into logical blocks with size limits:

from agent_context_manager import ContextManager, MemoryBlock

manager = ContextManager(max_tokens=8000)

# Define memory blocks
manager.add_block(MemoryBlock(
    name="system",
    max_tokens=500,
    priority=1.0,  # Highest priority, never compressed
    content="You are a helpful coding assistant."
))

manager.add_block(MemoryBlock(
    name="user_profile",
    max_tokens=200,
    priority=0.9,
    content="User prefers Python, uses VS Code."
))

manager.add_block(MemoryBlock(
    name="conversation",
    max_tokens=7000,
    priority=0.5,  # Can be compressed if needed
    strategy=SlidingWindowStrategy(keep_recent=30)
))

# Update blocks as needed
manager.update_block("user_profile", "User prefers Python, uses VS Code, timezone: PST")

Integration Examples

With OpenAI

from openai import OpenAI
from agent_context_manager import ContextManager

client = OpenAI()
manager = ContextManager(max_tokens=8000, model="gpt-4")

manager.add_message({"role": "system", "content": "You are helpful."})

while True:
    user_input = input("You: ")
    manager.add_message({"role": "user", "content": user_input})
    
    response = client.chat.completions.create(
        model="gpt-4",
        messages=manager.get_context()  # Auto-compressed if needed
    )
    
    assistant_message = response.choices[0].message.content
    manager.add_message({"role": "assistant", "content": assistant_message})
    print(f"Assistant: {assistant_message}")

With Anthropic

from anthropic import Anthropic
from agent_context_manager import ContextManager

client = Anthropic()
manager = ContextManager(max_tokens=100000, model="claude-3")

# Same pattern works with any provider

With LangChain

from langchain.memory import ConversationBufferMemory
from agent_context_manager import ContextManager, LangChainAdapter

manager = ContextManager(max_tokens=8000)
memory = LangChainAdapter(manager)  # Drop-in replacement

API Reference

ContextManager

Method	Description
add_message(msg)	Add a message to context
get_context()	Get compressed context as message list
token_count	Current token count
usage_percent	Percentage of context used
is_near_overflow()	Check if approaching limit
compress()	Manually trigger compression
clear()	Clear all messages

Strategies

Strategy	Best For
SlidingWindowStrategy	Simple agents, chatbots
ImportanceStrategy	Complex agents with tool use
DeduplicationStrategy	Repetitive workflows
HybridStrategy	Production systems

Contributing

Contributions welcome! Please read CONTRIBUTING.md first.

License

MIT License - see LICENSE file.