token-reducer 0.2.0

Intelligent token reduction library for LLM applications with context-aware compression

Token Reducer

Intelligent token reduction library for LLM applications with context-aware compression

Overview

Token Reducer is a Python library designed to reduce token counts in text and code inputs for Large Language Model (LLM) applications while preserving semantic meaning, logical structure, and task-relevant information. Achieve 50-70% token reduction without distorting facts or breaking code logic.

Key Features

• 🎯 Context-Aware Compression: Task-specific strategies for summarization, RAG, extraction, reasoning, and more
• 📊 Multi-Level Compression: Choose between light (5-15%), moderate (20-40%), or aggressive (50-70%) reduction
• 🔄 Multi-Pass Pipeline: Specialized passes for optimal compression (normalize → prune → compress → summarize → repack)
• 💻 Text & Code Support: Domain-specific compression for both natural language and source code
• 🔌 Tokenizer Agnostic: Works with OpenAI, Anthropic, HuggingFace, and custom tokenizers
• 🛡️ Fail-Safe Mode: Automatic quality validation with semantic similarity checking
• ⚡ High Performance: <100ms per 1000 tokens for text, <200ms for code
• 📴 Offline Operation: No cloud dependencies, works standalone

Installation

Basic Installation

pip install token-reducer

With Optional Dependencies

# For HuggingFace tokenizers
pip install token-reducer[transformers]

# For Anthropic tokenizers
pip install token-reducer[anthropic]

# For NLP features (entity extraction, sentence segmentation)
pip install token-reducer[nlp]

# For semantic similarity checking
pip install token-reducer[similarity]

# For code compression
pip install token-reducer[code]

# Install everything
pip install token-reducer[all]

Quick Start

Text Compression

from token_reducer import compress_text, TaskContext, CompressionLevel

# Compress text for summarization task
result = compress_text(
    text="Your long article text here...",
    task=TaskContext.SUMMARIZATION,
    level=CompressionLevel.MODERATE,
    tokenizer="gpt-4"
)

print(f"Original: {result.original_tokens} tokens")
print(f"Compressed: {result.compressed_tokens} tokens")
print(f"Reduction: {result.reduction_percentage}%")
print(f"\nCompressed text:\n{result.compressed_text}")

Code Compression

from token_reducer import compress_code, TaskContext, CompressionLevel

# Compress code for LLM context
result = compress_code(
    code="""
    def calculate_total_price(items, tax_rate=0.1):
        # Calculate subtotal
        subtotal = sum(item['price'] for item in items)
        # Apply tax
        tax = subtotal * tax_rate
        # Return total
        return subtotal + tax
    """,
    task=TaskContext.CODE_COMPLETION,
    level=CompressionLevel.AGGRESSIVE,
    language="python"
)

print(f"Compressed code:\n{result.compressed_code}")

Advanced Configuration

from token_reducer import CompressionConfig, compress_text

config = CompressionConfig(
    task=TaskContext.RAG,
    level=CompressionLevel.MODERATE,
    tokenizer="claude-3",
    preserve_entities=True,
    preserve_numbers=True,
    quality_threshold=0.90,
    enable_fail_safe=True
)

result = compress_text(text, config=config)

Compression Strategies

Task Types

Token Reducer adapts compression strategies based on your use case:

• SUMMARIZATION: Preserves causal links and chronological order
• RAG: Optimizes for retrieval context (entities, facts, key phrases)
• EXTRACTION: Keeps only fields relevant to extraction target
• REASONING: Preserves premises, key details, and logical connections
• TRANSLATION: Bypasses compression entirely
• CODE_COMPLETION: Preserves function signatures and interfaces
• DEBUGGING: Maintains variable names and error-relevant context
• QUESTION_ANSWERING: Preserves facts and entities for potential questions

Compression Levels

Level	Token Reduction	Semantic Similarity	Use Case
Light	5-15%	>98%	Maximum safety, minimal loss
Moderate	20-40%	>90%	Balanced compression and quality
Aggressive	50-70%	>80%	Maximum savings, acceptable loss

How It Works

Text Compression Pipeline

1. Normalize: Fix spacing, remove HTML, standardize quotes
2. Prune: Remove duplicates, redundancy, verbose explanations
3. Compress: Extract entities/facts, compact phrasing, reduce adjectives
4. Summarize: Apply task-specific tightening
5. Repack: Shorten sentences, optimize structure

Code Compression Pipeline

1. Remove Noise: Strip comments, blank lines, logging, debug prints
2. Rename Identifiers: Shorten variable/function/class names
3. Remove Unused: Eliminate dead code, unused imports/functions
4. Optimize Expressions: Simplify boolean/arithmetic expressions
5. Summarize Functions: Replace bodies with summaries (aggressive mode)
6. Reformat: Minimize whitespace, compact structure

Performance

• Text: <100ms per 1000 tokens (excluding quality checks)
• Code: <200ms per 1000 tokens (excluding quality checks)
• Quality Check: <50ms for semantic similarity validation

Use Cases

Reduce LLM API Costs

# Before: 10,000 tokens × $0.03/1K = $0.30 per request
# After (60% reduction): 4,000 tokens × $0.03/1K = $0.12 per request
# Savings: 60% cost reduction

Fit More Context in Token Limits

# Compress multiple documents to fit in context window
from token_reducer import batch_compress_text

results = batch_compress_text(
    texts=[doc1, doc2, doc3, doc4, doc5],
    task=TaskContext.RAG,
    level=CompressionLevel.MODERATE,
    parallel=True
)

# Combine compressed documents within token limit
combined = "\n\n".join(r.compressed_text for r in results)

RAG Pipeline Optimization

# Compress retrieved documents before sending to LLM
retrieved_docs = vector_store.similarity_search(query, k=10)

compressed_docs = [
    compress_text(
        doc.page_content,
        task=TaskContext.RAG,
        level=CompressionLevel.MODERATE
    )
    for doc in retrieved_docs
]

# Use compressed docs in prompt
context = "\n\n".join(d.compressed_text for d in compressed_docs)

Documentation

• Repository: https://github.com/UsamaTufail31/token-reducer
• Issues: https://github.com/UsamaTufail31/token-reducer/issues
• Examples: https://github.com/UsamaTufail31/token-reducer/tree/main/examples

Development

Setup Development Environment

# Clone repository
git clone https://github.com/UsamaTufail31/token-reducer.git
cd token-reducer

# Install with development dependencies
pip install -e ".[dev,all]"

# Install pre-commit hooks
pre-commit install

Run Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=token_reducer --cov-report=html

# Run specific test file
pytest tests/test_text_compression.py

Code Quality

# Format code
black src/ tests/

# Sort imports
isort src/ tests/

# Lint code
ruff src/ tests/

# Type check
mypy src/

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Citation

If you use Token Reducer in your research, please cite:

@software{token_reducer,
  title = {Token Reducer: Intelligent Token Reduction for LLM Applications},
  author = {Tufail, Usama},
  year = {2024},
  url = {https://github.com/UsamaTufail31/token-reducer}
}

Acknowledgments

• Inspired by research in context compression and semantic similarity
• Built with modern NLP libraries (spaCy, sentence-transformers, tiktoken)
• Designed for the LLM application development community

Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions

---

Created by Usama Tufail | GitHub