pystreamdocuments 0.3.0

Smart context provider for LLMs - Minimal, most-relevant knowledge based on query anticipation and semantic understanding

PyStreamDocuments

Stop losing context to noise. Transform your documents into intelligent knowledge with one API.

The Problem

Traditional RAG is broken. You index 1000 documents, your LLM gets 500+ tokens of noise, and still misses the answer that was there all along.

Traditional RAG:

• 📚 Index everything equally
• 🔍 Retrieve 10 documents (hope one is right)
• 🗣️ Feed 500 tokens to LLM (mostly noise)
• 💰 $0.01+ per query just for context

PyStreamDocuments:

• 🧠 Understand document structure upfront
• 📍 Retrieve exactly 1-3 pieces (guaranteed relevant)
• 💬 Feed 50 tokens to LLM (pure signal)
• 💸 <$0.001 per query (10-100x cheaper)

The Solution

PyStreamDocuments is an intelligent document ingestion and retrieval system that:

1. Understands structure — Extracts entities, relationships, hierarchy, and metadata upfront
2. Filters progressively — Eliminates 90%+ of noise across 4 stages before expensive operations
3. Returns minimal context — Only the signal your LLM actually needs
4. Connects anything — S3, PostgreSQL, Notion, Slack, GitHub, web—all with one API

Result: 10-100x fewer tokens, 10-100x lower cost, better answers.

What Makes It Different

Traditional RAG	PyStreamDocuments
❌ Embeds everything	✅ Embeddings optional, on-demand
❌ Vendor lock-in	✅ Your choice of DB/model
❌ Returns noise	✅ Returns pure signal
❌ Flat retrieval	✅ Smart progressive filtering
❌ Expensive	✅ Dramatically cheaper

Quick Start (60 seconds)

1️⃣ Install

pip install pystreamdocuments

2️⃣ Ingest

from pystreamdocuments import PyStreamDocuments

client = PyStreamDocuments()

# One API for all sources
await client.ingest("s3://my-bucket/docs")        # AWS S3
await client.ingest("postgres://host/db")         # Database
await client.ingest("notion://workspace-id")      # Notion
await client.ingest("github://owner/repo")        # GitHub

3️⃣ Search

# Smart retrieval with context
results = await client.search(
    "What's our Q3 revenue forecast?",
    context={"domain": "finance", "agent_role": "analyst"}
)

# Returns exactly what you need
for result in results:
    print(result.summary)  # ~50 tokens of pure signal

That's it. Your LLM now has perfect context every time.

Why People Love It

🚀 It's Fast

• Metadata extraction: >100 docs/sec
• Search: <100ms for complex queries
• No wait for embeddings

💰 It's Cheap

• Traditional RAG: $0.01+ per query
• PyStreamDocuments: <$0.001 per query
• 10-100x cost reduction

🎯 It Works

• Real understanding of document structure
• 4-stage progressive filtering (90%+ noise elimination)
• Entity-aware retrieval with freshness ranking
• Intelligent deduplication across sources

🔧 It's Flexible

• Optional embeddings (your choice)
• Multiple vector DBs (Pinecone, Weaviate, Qdrant, Postgres)
• Works without embeddings entirely
• Custom models supported
• Zero vendor lock-in

📦 It's Complete

• 171 tests passing
• Production-ready Docker deployment
• CI/CD automation with GitHub Actions
• Comprehensive REST API
• Enterprise monitoring built-in

Real-World Example

Problem: Your LLM misses the updated authorization policy because it's buried in Notion, referenced in Slack, and implemented in GitHub.

Traditional RAG:

1. Search all 1000 documents
2. Rank by keyword + embeddings (slow, expensive)
3. Return top 10 (5 are useless)
4. LLM still confused

PyStreamDocuments:

1. Recognize query is about "policies" + "authorization"
2. Filter to policy documents (metadata, $0 cost)
3. Rank by freshness (use document metadata, not index time)
4. Find Notion page (authoritative source, updated today)
5. Retrieve GitHub implementation (for reference)
6. Return 50 tokens of pure signal
7. LLM has exactly what it needs

Cost: 100x cheaper. Time: 2x faster. Accuracy: infinitely better.

Features That Matter

🌐 Universal Ingestion

One API for everything. S3, SharePoint, Google Drive, Notion, Slack, GitHub, PostgreSQL, MySQL, BigQuery, web, local files—handled uniformly.

🧠 Intelligent Understanding

• Extract entities (people, orgs, products, dates)
• Map relationships (who works with who, what depends on what)
• Preserve structure (sections, hierarchy, cross-references)
• Classify knowledge (procedure, policy, definition, data, code)

📊 Progressive Filtering

4-stage pipeline eliminates 90%+ before expensive operations:

1. Source relevance (metadata: $0)
2. Entity/type matching (index: $0)
3. Freshness ranking (document metadata: $0)
4. Semantic reranking (optional embeddings: ~$0.0001)

🔄 Smart Ranking

• Pre-ranked at index time (instant O(1) retrieval)
• Hybrid ranking (semantic + BM25 + RRF)
• Freshness-aware (use document metadata, not index time)
• Topic coherence scoring
• Automatic deduplication

📝 Quality Assurance

• Pre-ingestion validation gates
• Real-time quality metrics
• Health monitoring
• Trend analysis
• Automatic recommendations

🔍 Query Intelligence

• Intent classification (10 types)
• Entity extraction from queries
• Constraint parsing (time, domains, sources)
• Query anticipation (predict next question)

🚀 Production Ready

• Docker deployment (with docker-compose)
• Multi-cloud guides (AWS, GCP, Azure, K8s)
• Health checks and monitoring
• REST API with full documentation
• OpenTelemetry integration

Architecture

The Data Flow

Any Source (Docs, Databases, Web, APIs)
         ↓
   [Ingest & Parse]
         ↓
  [Extract Metadata]
  ├─ Entities & relationships
  ├─ Document structure
  ├─ Classification
  ├─ Freshness markers
  └─ Cross-source links
         ↓
   [Smart Indexing]
  ├─ BM25 keyword index
  ├─ Topic clustering
  ├─ Entity graph
  ├─ Relationship map
  └─ Freshness tracking
         ↓
   [User Query]
         ↓
  [4-Stage Filtering]
  ├─ Stage 1: Source relevance (metadata: $0)
  ├─ Stage 2: Entity/type match (index: $0)
  ├─ Stage 3: Freshness rank (metadata: $0)
  ├─ Stage 4: Semantic rerank (optional: ~$0.0001)
  └─ Deduplication
         ↓
 [Minimal Context]
  ~50 tokens of pure signal

Why It Works

• No noise upfront: Structure extracted once at index time
• Progressive filtering: Eliminates 90%+ before expensive operations
• Freshness awareness: Uses document metadata (created/modified/effective dates)
• Hybrid retrieval: Combines keyword + semantic + topic coherence
• Deduplication: Same fact from multiple sources? Return once

Installation & Setup

Requirements

• Python 3.10+
• PostgreSQL 14+ (optional, for persistence)

Install

# Basic (embeddings optional)
pip install pystreamdocuments

# With server extras
pip install pystreamdocuments[server]

# Development
pip install pystreamdocuments[dev]

Local Development

# Clone repo
git clone https://github.com/Mullassery/pystreamdocuments.git
cd pystreamdocuments

# Start full stack (includes PostgreSQL)
docker-compose up

# Server is now running on http://localhost:8080
# Try health check: curl http://localhost:8080/health

API Reference

Quick Examples

from pystreamdocuments import PyStreamDocuments

client = PyStreamDocuments()

# Ingest from any source
await client.ingest("s3://bucket/docs")

# Search with context
results = await client.search(
    query="authorization policy",
    context={
        "domain": "security",
        "agent_role": "compliance_officer",
        "time_period": ("2024-Q1", "2024-Q4")
    },
    top_k=3  # Return only top 3 (very specific)
)

# Get quality metrics
metrics = await client.get_quality_metrics()
print(f"Pass rate: {metrics.validation_pass_rate:.1%}")
print(f"Freshness: {metrics.freshness_score:.1%}")

# Health check
health = await client.health_check()
assert health["status"] == "healthy"

REST API

# Start server
pystreamdocuments-server

# Health check
curl http://localhost:8080/health

# Get quality metrics
curl http://localhost:8080/quality/metrics

# Ingest document
curl -X POST http://localhost:8080/ingest \
  -H "Content-Type: application/json" \
  -d '{
    "source_type": "pdf",
    "source_location": "s3://bucket/doc.pdf",
    "knowledge_type": "policy",
    "domain": "security",
    "content": "..."
  }'

# Search
curl -X POST http://localhost:8080/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What is our authorization policy?",
    "domain": "security"
  }'

Full Documentation

• API Reference — All endpoints with examples
• Deployment Guide — Cloud deployment (AWS, GCP, Azure, K8s)
• Contributing — How to contribute

Performance

Metric	Result
Ingestion speed	>100 docs/sec (metadata-first)
Search latency	<100ms (including semantic)
Context reduction	10-100x fewer tokens
Cost reduction	10-100x cheaper per query
Test coverage	171 tests passing (100% core)
Production ready	✅ Docker, K8s, multi-cloud

Status & Roadmap

✅ v0.2.0 (Current)

• Universal ingestion (5+ formats)
• Progressive filtering (4 stages)
• Quality validation & monitoring
• Query intelligence (intent + entity extraction)
• Semantic search with hybrid ranking
• REST API with health/metrics
• Docker & multi-cloud deployment
• 171 tests, production-ready

🚀 Coming Soon

• v0.3.0: Semantic caching, LLM-powered reranking, advanced analytics
• v1.0.0: Multi-tenancy, enterprise auth, GraphRAG integration

Why Open Source?

We believe intelligent document retrieval should be:

• Accessible: Not locked behind closed-source services
• Trustworthy: Code you can read and audit
• Flexible: Integrate however you want
• Affordable: Not 100x markup on commodities

That's why PyStreamDocuments is open source and MIT-licensed.

Community

• Issues: Report bugs or request features
• Discussions: Ask questions and share ideas
• Contributing: See CONTRIBUTING.md

License

MIT License — you own what you build.

---

Built With

• Rust + Python — Performance + accessibility
• PyO3 — Zero-copy Python bindings
• Axum — Modern async HTTP server
• tantivy — Fast BM25 search
• PostgreSQL — Reliable persistence
• OpenTelemetry — Production observability

Extensibility

PyStreamDocuments integrates with specialized tools:

• Document Parsers — PDF, Word, Markdown parsing with structure extraction
• Spreadsheet Engines — Schema-aware querying and extraction
• Database Connectors — Schema inspection and data retrieval
• Web Browsers — Automated page navigation and content extraction
• Code Analyzers — Language-aware parsing and API extraction
• MCP Tools — Integration with Model Context Protocol providers (Notion, Slack, GitHub, etc.)

---

Stop losing context. Start using intelligence.

Get Started → | Deployment Guide → | API Reference →

⭐ If you find this useful, please star the repository! It helps us grow.