Socratic RAG

Why Socratic RAG?

Building reliable retrieval systems is complex. Socratic RAG provides production-grade RAG:

• Multiple Vector Stores - ChromaDB, Qdrant, FAISS, Pinecone with extensible provider pattern
• Flexible Embeddings - Sentence Transformers or OpenAI embeddings via Socrates Nexus
• Smart Chunking - Fixed-size, semantic, and recursive strategies for optimal chunk processing
• Document Processing - Handle Text, PDF, Markdown, and code files out of the box
• Async Support - Full async/await interface for non-blocking operations at scale

Production-grade Retrieval-Augmented Generation (RAG) package for Python.

Features

• Multiple Vector Databases: ChromaDB, Qdrant, FAISS, Pinecone (with extensible provider pattern)
• Flexible Embedding Providers: Sentence Transformers, OpenAI (via Socrates Nexus)
• Smart Chunking: Fixed-size, semantic, and recursive chunking strategies
• Document Processing: Text, PDF, Markdown, and code files
• Framework Integrations: Openclaw skills and LangChain components
• Async Support: Full async/await interface for non-blocking operations
• Production Ready: 70%+ test coverage, type hints, comprehensive documentation
• Part of Socrates Ecosystem: Built on Socrates Nexus for LLM integration

Part of the Socrates Ecosystem

Socratic RAG is a core component of the Socrates Ecosystem - a collection of production-grade AI packages that work together.

How It Uses Socrates Nexus

• Embedded document generation uses Socrates Nexus for embeddings
• Answer generation uses Socrates Nexus for LLM calls (multi-provider support)
• Works with any Socrates Nexus provider (Claude, GPT-4, Gemini, Ollama)

Related Packages in the Ecosystem

• Socrates Nexus (Dependency) - Universal LLM client
• Socratic Analyzer - Code analysis (also uses Socrates Nexus)
• Socratic Agents (Coming Q4 2026) - Multi-agent orchestration
• Socratic Knowledge (Coming Q1 2027) - Enterprise knowledge management

👉 Full ecosystem guide: See Socrates Nexus ECOSYSTEM.md

📊 Track development: View the Socrates Ecosystem Roadmap to see progress across all packages

Installation

Basic Installation

pip install socratic-rag

With Optional Dependencies

# All features
pip install socratic-rag[all]

# Specific vector stores
pip install socratic-rag[chromadb,qdrant,faiss]

# Document processing
pip install socratic-rag[pdf,markdown]

# Integrations
pip install socratic-rag[langchain,openclaw,nexus]

# Development
pip install socratic-rag[dev]

Quick Start

Basic Usage

from socratic_rag import RAGClient

# Initialize client
client = RAGClient()

# Add documents
doc_id = client.add_document(
    content="Python is a programming language created by Guido van Rossum.",
    source="python_facts.txt"
)

# Search
results = client.search("What is Python?", top_k=5)
for result in results:
    print(f"Score: {result.score:.2f}")
    print(f"Text: {result.chunk.text}\n")

# Retrieve formatted context for LLM
context = client.retrieve_context("What is Python?")
print(context)

Custom Configuration

from socratic_rag import RAGClient, RAGConfig

config = RAGConfig(
    vector_store="chromadb",
    embedder="sentence-transformers",
    chunking_strategy="fixed",
    chunk_size=512,
    chunk_overlap=50,
    top_k=5,
)

client = RAGClient(config)

Async Usage

import asyncio
from socratic_rag import AsyncRAGClient

async def main():
    client = AsyncRAGClient()

    # Add documents asynchronously
    doc_id = await client.add_document(
        content="Document content",
        source="source.txt"
    )

    # Search asynchronously
    results = await client.search("query")

    # Retrieve context asynchronously
    context = await client.retrieve_context("query")

asyncio.run(main())

Architecture

Provider Pattern

Socratic RAG uses an extensible provider pattern for easy integration of new components:

RAGClient
├── Embedder Provider (sentence-transformers, OpenAI, etc.)
├── Chunker Provider (fixed, semantic, recursive)
└── Vector Store Provider (ChromaDB, Qdrant, FAISS, Pinecone)

Core Models

• Document: Raw document with metadata
• Chunk: Text chunk with position and metadata
• SearchResult: Search result with relevance score
• RAGConfig: Configuration object

Support Development

If you find this package useful, consider supporting development:

• Become a Sponsor - Get early access to new features
• Star on GitHub - Shows your appreciation
• Report Issues - Help improve the package

Your support helps fund development of the entire Socratic ecosystem.

Examples

See the examples/ directory for:

1. 01_basic_rag.py - Basic RAG workflow
2. 02_qdrant_rag.py - Using Qdrant vector store
3. 03_faiss_rag.py - Using FAISS vector store
4. 04_document_processing.py - Document processing
5. 05_openclaw_integration.py - Openclaw skill integration
6. 06_langchain_integration.py - LangChain integration
7. 07_llm_powered_rag.py - RAG with LLM (using Socrates Nexus)

Testing

Run the test suite:

pytest tests/ -v --cov=socratic_rag

Specific test categories:

# Unit tests only
pytest tests/ -m unit

# Integration tests only
pytest tests/ -m integration

# Exclude slow tests
pytest tests/ -m "not slow"

Vector Store Providers

ChromaDB (Default)

from socratic_rag import RAGClient, RAGConfig

config = RAGConfig(vector_store="chromadb")
client = RAGClient(config)

Qdrant

config = RAGConfig(vector_store="qdrant")
client = RAGClient(config)

FAISS

config = RAGConfig(vector_store="faiss")
client = RAGClient(config)

Embedding Providers

Sentence Transformers (Default)

config = RAGConfig(embedder="sentence-transformers")
client = RAGClient(config)

OpenAI (via Socrates Nexus)

config = RAGConfig(embedder="openai")
client = RAGClient(config)

Document Processing

Text Files

from socratic_rag.processors import TextProcessor

processor = TextProcessor()
documents = processor.process("path/to/file.txt")

PDF Files

from socratic_rag.processors import PDFProcessor

processor = PDFProcessor()
documents = processor.process("path/to/file.pdf")

Markdown Files

from socratic_rag.processors import MarkdownProcessor

processor = MarkdownProcessor()
documents = processor.process("path/to/file.md")

Framework Integrations

Openclaw

from socratic_rag.integrations.openclaw import SocraticRAGSkill

skill = SocraticRAGSkill(vector_store="chromadb")
skill.add_document("content", "source.txt")
results = skill.search("query")

LangChain

from socratic_rag.integrations.langchain import SocraticRAGRetriever
from langchain.chat_models import ChatAnthropic
from langchain.chains import RetrievalQA

retriever = SocraticRAGRetriever(client=rag_client, top_k=5)
llm = ChatAnthropic(model="claude-sonnet")
qa = RetrievalQA.from_chain_type(llm=llm, retriever=retriever)

answer = qa.run("What is Python?")

Documentation

• Caching Strategy Guide - Complete guide to performance optimization with embedding caching and search result caching, including configuration, memory management, and best practices

API Reference

RAGClient

Methods

• add_document(content, source, metadata=None) -> str - Add document to knowledge base
• search(query, top_k=None, filters=None) -> List[SearchResult] - Search for relevant documents
• retrieve_context(query, top_k=None) -> str - Retrieve formatted context for LLM
• clear() -> bool - Clear all documents

AsyncRAGClient

Same methods as RAGClient but with async/await.

Configuration Options

RAGConfig(
    vector_store="chromadb",           # Vector store provider
    embedder="sentence-transformers",  # Embedding provider
    chunking_strategy="fixed",         # Chunking strategy
    chunk_size=512,                    # Characters per chunk
    chunk_overlap=50,                  # Overlap between chunks
    top_k=5,                           # Default number of results
    embedding_cache=True,              # Cache embeddings
    cache_ttl=3600,                    # Cache TTL in seconds
    collection_name="socratic_rag",    # Collection name
)

Exceptions

• SocraticRAGError - Base exception
• ConfigurationError - Configuration validation error
• VectorStoreError - Vector store operation error
• EmbeddingError - Embedding operation error
• ChunkingError - Chunking operation error
• ProcessorError - Document processing error
• DocumentNotFoundError - Document not found
• ProviderNotFoundError - Provider not found

Performance Tips

1. Use appropriate chunk size: Smaller chunks (256-512) for dense retrieval, larger (1024+) for sparse
2. Set overlap wisely: 10-15% overlap usually works well
3. Cache embeddings: Enable embedding cache for repeated queries
4. Batch operations: Use embed_batch() for multiple embeddings
5. Choose right embedder: Sentence Transformers for local, OpenAI for production scale

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE file for details

Citation

If you use Socratic RAG in your research, please cite:

@software{socratic_rag_2024,
  title={Socratic RAG: Production-grade Retrieval-Augmented Generation},
  author={Your Name},
  year={2024},
  url={https://github.com/Nireus79/Socratic-rag}
}

Roadmap

v0.1.0 (Current)

• ✅ Core RAG functionality
• ✅ ChromaDB vector store
• ✅ Sentence Transformers embeddings
• ✅ Fixed-size chunking
• ✅ Openclaw integration
• ✅ LangChain integration

v0.2.0

• Qdrant and FAISS vector stores
• Semantic chunking
• Pinecone cloud provider
• Vision model support

v0.3.0

• Hybrid search (vector + keyword)
• Re-ranking with cross-encoders
• Multi-language support

Documentation

Getting Started

• Quickstart Guide - 5-minute introduction
• Contributing Tutorial - Step-by-step guide for first-time contributors

Guides & References

• API Reference - Complete API documentation
• Vector Stores Guide - Compare and select vector stores
• Embeddings Guide - Embedding models and providers
• Integrations Guide - Framework integration guides
• Architecture Overview - System design and diagrams
• Architecture Deep Dive - Technical architecture details

Operations & Deployment

• Deployment Guide - Local, Docker, production, Kubernetes, AWS, GCP
• Kubernetes Setup - Kubernetes manifests and deployment
• Docker Setup - Docker containerization examples
• Performance Guide - Benchmarks, optimization, scaling
• Troubleshooting - Common issues and solutions

Community & Project

• FAQ - 50+ frequently asked questions
• Code of Conduct - Community guidelines
• Security Policy - Vulnerability reporting and best practices
• ROADMAP - Future plans for v0.2.0, v0.3.0, v1.0.0
• CONTRIBUTING - Contribution guidelines

Examples

All examples are in the examples/ directory:

1. 01_basic_rag.py - Basic RAG workflow
2. 02_multi_vector_stores.py - Vector store comparison
3. 03_openclaw_integration.py - Openclaw skill integration
4. 04_langchain_integration.py - LangChain integration
5. 05_llm_powered_rag.py - LLM-powered answers
6. 06_rest_api.py - REST API with FastAPI
7. 07_docker_containerization.py - Docker setup guide
8. 08_deployment_patterns.py - Deployment strategies
9. 09_advanced_rag_patterns.py - Multi-agent RAG, conversation context
10. 10_streaming_rag.py - Streaming responses
11. 11_real_time_updates.py - Real-time knowledge base updates

Support

For issues, questions, or suggestions:

• GitHub Issues: https://github.com/Nireus79/Socratic-rag/issues
• Discussions: https://github.com/Nireus79/Socratic-rag/discussions