Supabase Easy RAG

A modular, lightweight Hybrid RAG engine built natively on PostgreSQL & Supabase.

Why Supabase Easy RAG?

• 🎯 High-Accuracy Hybrid Search: Combines semantic vector search with keyword matching — catches both conceptual questions and exact IDs/terms.
• 📖 Parent-Context Expansion: Searches precise 400-token chunks for high retrieval precision (89%+ Document Top-1, 92%+ Fact Recall), while feeding the full parent section to your LLM.
• 🔒 Native Multi-Tenant Security: Out-of-the-box Row-Level Security (RLS) via Supabase Auth (auth.uid()) — users only see documents they own or are shared with them.
• ⚡ High-Speed Parallel Ingestion: Automatic change detection (SHA-256) verifies 700+ docs/sec incrementally, with multi-threaded embedding generation.
• 🌍 Battle-Tested Multilingual: Evaluated across 11 languages with automatic text search dictionary fallbacks.
• 🔌 Zero Framework Overhead: Clean Python SDK and pure PostgreSQL RPCs. No heavy dependencies.

📊 Comprehensive Multilingual Benchmark (TyDi QA)

Evaluated against the complete Google Research TyDi QA gold-standard validation corpus (4,488 authentic Wikipedia articles, 5,077 real human questions across 11 typologically diverse languages) with 400-character chunking and Reciprocal Rank Fusion (RRF).

1. Global Information Retrieval (IR) Quality Metrics

[!NOTE] Metric Rigor:

• Document Hit Rate @ K strictly requires the retrieved chunk to belong to the target ground-truth document.
• Answer Span Recall @ K verifies that the exact answer span is present in the retrieved chunk content for QA generation.
• Full benchmark is 100% reproducible via python eval/corpora/fetch_tydiqa.py and python eval/benchmark.py.

2. Multilingual Breakdown Across 11 Languages

Quick Start

1. Installation

pip install supabase-easy-rag

2. Apply Database Migrations to Supabase

Export and run the SQL migrations:

easy-rag init-sql --output ./migrations --dimensions 1536

Apply 01_schema.sql and 02_functions.sql to your Supabase project (via Supabase CLI supabase db query --file ... or SQL Editor).

3. Configure Environment Variables

SUPABASE_URL="https://your-project.supabase.co"
SUPABASE_SERVICE_ROLE_KEY="your-service-role-key"

# Embedding Provider (OpenAI or Azure OpenAI)
OPENAI_API_KEY="sk-..."
# or Azure:
AZURE_OPENAI_API_KEY="your-azure-key"
AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com"
AZURE_OPENAI_EMBEDDING_DEPLOYMENT="text-embedding-3-large"

💻 Python SDK Usage

1. Hybrid Search with Parent-Context Expansion

from supabase_easy_rag import EasyRagClient
from supabase_easy_rag.providers.azure import AzureEmbeddingProvider

# Initialize provider & client
provider = AzureEmbeddingProvider(
    api_key="...",
    endpoint="https://your-resource.openai.azure.com",
    model="text-embedding-3-large",
)
client = EasyRagClient(embedding_provider=provider)

# Search by dense chunk, expand to full parent section for LLM context
results = client.search_hybrid(
    query="When was the Ottoman Empire established?",
    match_count=5,
    candidate_count=50,
    rrf_k=60,
    expand_context="section",  # "section" or "document"
)

for item in results:
    print(f"[{item.final_score:.4f}] {item.document_title} > {item.section_title}")
    # Effective text contains the full parent section text
    print(item.effective_text[:200] + "...")

2. High-Throughput Parallel Directory Ingestion

# Sync a directory of Markdown documents with 8 parallel workers
sync_stats = client.sync_directory(
    directory_path="./knowledgebase_docs",
    batch_size=30,
    max_workers=8,
    enable_chunking=True,
    chunk_size=800,
    chunk_overlap=100,
)

print(f"Synced {sync_stats['total_files']} files in parallel.")

3. Asynchronous Client (FastAPI / High-Concurrency)

from supabase_easy_rag import AsyncEasyRagClient

async_client = AsyncEasyRagClient(embedding_provider=provider)

results = await async_client.search_hybrid(
    query="PostgreSQL connection pooling guidelines",
    match_count=3,
    expand_context="section",
)

🛠️ CLI Commands

# Export migration files
easy-rag init-sql --dimensions 1536 --output ./migrations

# Sync directory of documents
easy-rag sync ./docs --workers 8

# Execute test query with diagnostics
easy-rag query "What is distributed erasure coding?" --mode hybrid --count 5

# Manage backend API access tokens
easy-rag create-token "Production Ingestion Worker"
easy-rag list-tokens

⚙️ Architecture & Implementation Details

• Two-Stage Hybrid Fusion: Independent candidate pools are retrieved via indexed scans (HNSW for vector distance, GIN for full-text match) and fused using Reciprocal Rank Fusion: $$RRF_Score = \frac{w_v}{k + rank_v} + \frac{w_t}{k + rank_t}$$ This avoids arbitrary score normalization issues between cosine similarities and BM25/FTS weights.
• Weighted Lexical Search: Uses PostgreSQL native setweight() indexing where document titles receive Weight A ($1.0$), section headings receive Weight B ($0.4$), and chunk text receives Weight D ($0.1$).
• Candidate Pre-filtering: Full-text candidate scans limit initial matches to Top-200 before applying ts_rank to minimize CPU cycles on frequent keywords.
• Multi-Tenant Security: Enforces database-level isolation via Supabase Auth (auth.uid()) using Row-Level Security (RLS) policies on documents and chunks.

📄 License

Distributed under the MIT License. See LICENSE for details.