PyStreamPDF 2.2.1

Intelligence engine for PDFs: selective retrieval, structure analysis, and token-efficient RAG

PyStreamPDF

The Intelligence Engine for PDFs

---

The Problem

You're building AI agents that work with PDFs. It works, but it's wasteful and expensive:

The Painful Reality

• 📄 Converting a 100-page document takes 10-30 seconds
• 💰 Your token costs are 10-50x higher than necessary
• 🔍 You process content your AI will never use
• ⏱️ API calls are slow with massive context
• 💾 Storage balloons as you keep full markdown versions

Example: A 500-page manual with traditional RAG:

• Convert all 500 pages → 2-3M tokens → $30-50 per conversation
• Process everything → find 1-5% used → waste 95-99%

---

The Solution: PyStreamPDF

PyStreamPDF finds and converts only what matters in 3 steps:

PyStreamPDF Workflow:
1. Analyze PDF structure (no conversion needed)
2. Search for relevant content (5-10% identified)
3. Convert only selected sections (5-10% processed)
4. Auto-optimize context for your token budget

Result: 10-50x cost reduction with same or better accuracy

Concrete Impact

Metric	Traditional	PyStreamPDF
Processing Time	30 seconds	0.5 seconds
Token Usage	2M tokens	50-150k tokens
Cost per Query	$30-50	$0.30-1.50
Storage	Full document	Indexed metadata
API Latency	Slow	Fast
Accuracy	Hits irrelevant content	Finds only relevant sections

---

What's New in v2.1.0

🎯 Automatic Budget Selection from Queries

PyStreamPDF now auto-detects query complexity and selects the optimal token budget:

from pystreampdf.query_analyzer import auto_select_budget

# Auto-detects and selects budget automatically
budget, analysis = auto_select_budget("What is machine learning?")
# → minimal: 500 tokens, confidence: 65%

budget, analysis = auto_select_budget("Analyze and compare neural networks vs transformers")
# → rich: 2000 tokens, confidence: 78%

# Override if needed
budget, analysis = auto_select_budget(
    "What is X?", 
    force_level="comprehensive"  # Force max: 2750 tokens
)

How it works:

• Simple queries ("What is X?") → 500 tokens
• Moderate queries ("Explain how X works") → 1500 tokens
• Complex queries ("Compare X vs Y") → 2000 tokens
• Deep analysis ("Deep dive into X") → 2750 tokens

🔍 Search Results with Rich Metadata

Search results now include everything you need to evaluate quality:

from pystreampdf.search import SearchFilter

# Search and get rich results
results = navigator.search("machine learning")
# → 6 sections found, 25,300 total words

# View in beautiful table format
print(results.to_cli_table())
# Section Title              | Pages    | Score | Length | Preview
# Chapter 1: Intro ML       | p.1-3    | 95%   | 850w   | Machine learning is a subset...
# Chapter 2: Supervised     | p.4-8    | 88%   | 1200w  | Supervised learning uses...

# Filter by multiple criteria
high_relevance = results.by_relevance(0.75)  # Only >75% matches
specific_chapter = results.by_page_range(10, 50)  # Pages 10-50 only
substantial = results.by_length(min_words=200)  # 200+ words

# Chain filters together
refined = (results
    .by_relevance(0.70)
    .by_page_range(1, 100)
    .by_length(min_words=100)
    .sorted_by_relevance()
    .top(5))

# Export as JSON for programmatic use
json_data = results.to_json()

Result metadata includes:

• Page numbers (start & end)
• Relevance score (0.0-1.0)
• Word count
• Text preview (first 200 chars)
• Matched keywords

⚙️ Filtering Strategies

Control what sections are included with 3 intelligent strategies:

from pystreampdf.config import FilteringConfig

# Strategy 1: STRICT (only highest relevance)
strategy = FilteringConfig.get_strategy("strict")
# Min score: 0.70, Max sections: 3
# Use: Quick lookups, minimal focused output

# Strategy 2: BALANCED (recommended)
strategy = FilteringConfig.get_strategy("balanced")
# Min score: 0.50, Max sections: 5
# Use: Most queries, best precision/recall balance

# Strategy 3: LENIENT (broader coverage)
strategy = FilteringConfig.get_strategy("lenient")
# Min score: 0.30, Max sections: 10
# Use: Comprehensive analysis, broad topics

📊 Updated Token Budget Scale

Expanded 4-slab design for richer context retrieval:

from pystreampdf.config import TokenBudgetConfig

# Available presets (hard limits: 650-3500)
minimal      = TokenBudgetConfig.get_preset("minimal")        # 650 tokens (~500 words)
standard     = TokenBudgetConfig.get_preset("standard")       # 2500 tokens (~1923 words) ⭐ RECOMMENDED
rich         = TokenBudgetConfig.get_preset("rich")           # 3000 tokens (~2307 words)
comprehensive = TokenBudgetConfig.get_preset("comprehensive") # 3500 tokens (~2692 words)

# Use with navigator
context, flow = navigator.retrieve_with_flow(
    "your query",
    max_tokens=standard  # 1500 tokens
)

📈 Pipeline Visualization

See exactly where content is lost at each stage:

context, flow = navigator.retrieve_with_flow("neural networks", max_tokens=1500)

# Visual table showing flow through pipeline
print(flow.to_cli_table())
# Section                    | Raw    | Extract | Index  | Retrieve | Select
# Chapter 1: Intro          | 850w   | [OK]    | [OK]   | [OK]     | [OK]
# Chapter 2: Deep Learning  | 2100w  | [*]     | [*]    | [OK]     | [X]
#
# Legend:
# [OK]  = Passed through
# [*]   = Data loss at this stage
# [--]  = Filtered out (intentional)
# [X]   = Exceeds token budget

# Get summary
print(flow.summary)
# Query: neural networks
# Retrieval loss: 2300 words (31%)
# Budget loss: 850 words (filtered by token limit)

---

Quick Start (2 Minutes)

Install

pip install PyStreamPDF==2.1.0

Basic Usage

import pystreampdf
from pystreampdf.query_analyzer import auto_select_budget

# Open PDF
doc = pystreampdf.open("research_paper.pdf")
index = doc.build_index("/tmp/index.db")
navigator = doc.navigator_with_index(index)

# Auto-select budget based on query
budget, analysis = auto_select_budget("How do transformers work?")
print(f"Selected budget: {budget} tokens ({analysis.query_complexity})")

# Retrieve with auto-selected budget
context, flow = navigator.retrieve_with_flow(
    "How do transformers work?",
    max_tokens=budget  # Auto-selected from query
)

# See what was retrieved
print(flow.to_cli_table())
print(f"Retrieved: {len(context.sections)} sections, {context.total_tokens} tokens")

Search with Filtering

from pystreampdf.search import SearchFilter

# Search
results = navigator.search("machine learning", max_results=20)
print(f"Found {results.count()} matching sections")

# Filter to high-quality, substantial content
high_quality = (results
    .by_relevance(0.75)           # 75%+ match
    .by_length(min_words=200)     # 200+ words
    .sorted_by_relevance())

# Display results
print(high_quality.to_cli_table())

# Export for analysis
json_results = high_quality.to_json()

---

Feature Showcase

1. Intelligent Retrieval

# Find relevant pages without converting everything
results = navigator.search("attention mechanisms")

# Evaluation metrics built-in
print(f"Relevance scores: min={min(r.relevance_score for r in results.results):.2f}, "
      f"max={max(r.relevance_score for r in results.results):.2f}")

# Access rich metadata
for result in results.sorted_by_relevance().results[:5]:
    print(f"{result.section_title} ({result.pages_range()})")
    print(f"  Relevance: {result.relevance_score:.0%}")
    print(f"  Length: {result.word_count} words")
    print(f"  Preview: {result.preview[:100]}...")

2. Query-Aware Budget Selection

from pystreampdf.query_analyzer import QueryAnalyzer

# Analyze any query
analysis = QueryAnalyzer.analyze("What is attention in transformers?")

print(f"Complexity: {analysis.query_complexity}")        # "simple"
print(f"Confidence: {analysis.confidence:.0%}")          # "85%"
print(f"Matched keywords: {analysis.matched_keywords}")  # ["what is", "?"]

# Budget automatically selected based on analysis
if analysis.confidence > 0.8:
    # High confidence - use minimal budget
    budget = 500
else:
    # Lower confidence - use standard budget
    budget = 1500

3. Multi-Criteria Filtering

# Combine multiple filters
results = (navigator.search("deep learning")
    .by_relevance(0.60)              # Only 60%+ matches
    .by_page_range(1, 100)           # First 100 pages only
    .by_length(min_words=100)        # At least 100 words
    .by_section("chapter")           # Only chapters
    .sorted_by_relevance()           # Sort by quality
    .top(10))                        # Top 10 results

# Each filter is fast and composable
print(f"Final results: {results.count()} sections")

4. Pipeline Understanding

# See entire flow from search → selection
context, flow = navigator.retrieve_with_flow(
    "transformer architectures",
    max_tokens=1500
)

# Understand each stage
print("=== Pipeline Analysis ===")
print(f"Raw matched sections: {len(flow.sections)}")
print(f"Extraction loss: {flow.summary.extraction_loss_pct():.1f}%")
print(f"Retrieval loss: {flow.summary.retrieval_loss_pct():.1f}%")
print(f"Budget filtering: {flow.summary.filtering_loss_pct():.1f}%")

# Visual representation
print(flow.to_flow_diagram())

---

Real-World Cost Savings

Processing a 300-page technical manual with GPT-4 for support queries:

Traditional RAG

• Full conversion: 20 seconds
• Per-query tokens: 120,000 (full doc)
• Cost per query: ~$1.80
• Monthly (1,000 queries): ~$1,800

PyStreamPDF v2.1.0

• Structure analysis: 0.5 seconds
• Per-query tokens: 1,500 (auto-selected)
• Cost per query: ~$0.02
• Monthly (1,000 queries): ~$20

Savings: 98% cost reduction ($1,780/month) + 40x faster

---

Use Cases

📚 Document Q&A

"What is the return policy?" → Searches, auto-selects minimal budget (500), retrieves 1-2 sections

📊 Data Extraction

"Extract all metrics from Chapter 4" → Filters by section, uses rich budget (2000), gets comprehensive context

🔍 Research & Analysis

"Compare approaches in sections 3.1 and 3.2" → High relevance filter, rich budget, ranked results

🔒 Compliance & Audit

"Find all references to data retention policy" → Cross-document search, full metadata, audit trail

💬 Chatbot Integration

"Answer customer questions about the manual" → Auto-select by complexity, cache results, minimal tokens

---

Feature Comparison

Feature	Traditional	PyStreamPDF
Token Efficiency	❌ 100% of doc	✅ 5-10% needed
Retrieval Speed	❌ Slow	✅ <50ms
Cost per Query	❌ $1-10	✅ $0.01-1
Search Metadata	❌ None	✅ Full metadata
Filter by Criteria	❌ No	✅ 5 dimensions
Auto Budget Selection	❌ No	✅ Yes, by complexity
Pipeline Visualization	❌ Black box	✅ Full transparency
Large Documents	❌ Memory issues	✅ 1000+ pages
Security Support	❌ Basic	✅ Full encryption/audit
Production Ready	⚠️ Maybe	✅ Yes (94 tests)

---

Documentation

• Auto Budget Selection — Query analysis and automatic token budget selection
• Search Results & Filtering — Rich metadata and multi-criteria filtering
• Filtering Strategies — Control section selection (strict/balanced/lenient)
• Pipeline Visualization — Understand where content is lost
• OCR & Parsing Issues — Handle scanned PDFs and complex documents
• OCR Feature Guide — Built-in optical character recognition

Examples

• examples/auto_budget_selection.py — Query analysis demos
• examples/search_and_filtering.py — Search with rich metadata
• examples/complete_pipeline_flow.py — End-to-end pipeline visualization
• examples/search_demo.py — Interactive search examples

---

Installation & Setup

Python 3.9+

# Using pip
pip install PyStreamPDF

# Using uv (recommended)
uv add PyStreamPDF

# From source
git clone https://github.com/Mullassery/PyStreamPDF.git
cd PyStreamPDF
pip install -e .

Optional: OCR Support

For scanned PDFs, install OCR:

# Option 1: Tesseract (system dependency)
brew install tesseract  # macOS
apt-get install tesseract-ocr  # Linux

# Option 2: PaddleOCR (pure Python)
pip install paddleocr

---

Why PyStreamPDF

Core Insight

Most questions need <1% of a PDF. Stop processing the other 99%.

Performance

• Parse 10x faster than traditional methods
• Retrieve in <50ms
• Convert selected pages in <1s

Cost Reduction

• 10-50x fewer tokens needed
• Eliminate unnecessary processing
• Orders of magnitude savings

AI-Native

• Built for how AI agents actually work
• Query-aware budget selection
• Rich metadata for better decisions

Production-Ready

• 94 tests passing
• Security-aware (encryption, audit)
• Handles 1000+ page documents
• MIT Licensed

---

Current Status: v2.1.0

✅ Foundation — PDF parsing, indexing, retrieval
✅ Intelligence — Entity extraction, knowledge graphs, fact verification
✅ Context Assembly — 4 adaptive assembly strategies
✅ Filtering — Multi-criteria filtering (strict/balanced/lenient)
✅ Search — Rich metadata, multiple filters, sorting
✅ Auto-Budget — Query-aware token selection
✅ Security — Encryption, permissions, audit logging
✅ Production — 94 tests, monitoring, error handling

---

The Insight

Traditional systems convert 100% to use 1%.

PyStreamPDF converts 1% to use 100% of that.

10-50x cost reduction. Same accuracy. Better speed.

---

License

MIT License — See LICENSE for details

---

Vision

Transform how the world works with PDF data in AI systems.

From: "A faster PDF converter"
To: "The retrieval engine for PDFs"

Only convert what's needed. Retrieve what matters. Optimize everything else.

---

Getting Help

• 📖 See docs/ for complete documentation
• 💡 Check examples/ for working code
• 🐛 Report issues on GitHub
• 📧 Email: [e-mail]