सूत्र DB (SutraDB)

सूत्र (Sūtra): An aphorism or thread of knowledge designed to hold vast wisdom in the most concise, unbreakable form.

SutraDB is an ultra-fast, zero-dependency hybrid vector search and BM25 lexical engine engineered in pure Python. It combines SIMD-accelerated linear algebra with Robertson-Spärck Jones BM25 ranking and in-flight compound metadata filtering.

Designed specifically for the 95% of AI applications (local RAG, agent memory, enterprise document search, catalog matching) that need sub-millisecond retrieval without the multi-gigabyte dependency trees of Chroma or the network latency of cloud-managed vector databases.

🏗️ Architecture

                              CLIENT REQUEST
             [ Text Query: "P99 latency bug" | Vector: [0.12, ...] ]
             [ Metadata Filter: {"status": "resolved", "priority": {"$lte": 2}} ]
                                     │
                                     ▼
                      ┌──────────────────────────────┐
                      │    SutraDB Execution Core    │
                      └──────────────┬───────────────┘
                                     │
         ┌───────────────────────────┼───────────────────────────┐
         ▼                           ▼                           ▼
┌──────────────────┐       ┌──────────────────┐       ┌──────────────────┐
│  Metadata Engine │       │ Dense Vector Core│       │ Sparse BM25 Core │
│ (AST Predicates) │       │ (SIMD BLAS / SQ8)│       │ (Lexical Tokens) │
└────────┬─────────┘       └────────┬─────────┘       └────────┬─────────┘
         │                          │                          │
         │ Dynamic Bitmask          │ Dense Scores             │ Lexical Scores
         │ (e.g., 0b101100)         │ [0.89, 0.42, ...]        │ [12.4, 0.0, ...]
         └─────────────┬────────────┴─────────────┬────────────┘
                       │                          │
                       ▼                          ▼
               ┌───────────────┐          ┌───────────────┐
               │ Masked Dense  │          │ Masked BM25   │
               │ Top-K Heap    │          │ Top-K Heap    │
               └───────┬───────┘          └───────┬───────┘
                       │                          │
                       └───────────┬──────────────┘
                                   │
                                   ▼
                   ┌───────────────────────────────┐
                   │ Reciprocal Rank Fusion (RRF)  │
                   │ Merges semantic + exact words │
                   └───────────────┬───────────────┘
                                   │
                                   ▼
                   ┌───────────────────────────────┐
                   │    Ranked Final Results       │
                   │    P50: 0.36ms | P99: 5.9ms   │
                   └───────────────────────────────┘

⚡ Key Highlights

• Pure SIMD / BLAS Velocity: Pre-normalizes vectors at insertion time so Cosine Similarity reduces to a single GEMV matrix-vector multiplication executed in L1 cache lines.
• IVF-Flat Approximate Nearest Neighbor (ANN): Sublinear clustering index powered by k-means++ centroid initialization. Dramatically prunes search space for large-scale datasets while preserving high recall (collection.build_index(), use_ann=True).
• Thread-Safe Readers-Writer Concurrency: Built-in ReadWriteLock with writer priority guarantees zero race conditions during parallel batch insertions, deletes, and query executions.
• Reciprocal Rank Fusion (RRF): Dense embeddings understand semantic intent; BM25 matches exact serial numbers, error codes, and technical jargon. SutraDB dynamically fuses both ranking signals via RRF.
• Single-Stage In-Flight Predicate Masking: Zero subset memory allocations. Evaluates complex JSON conditions ($eq, $ne, $gt, $gte, $in, $nin, $contains, $and, $or) into high-speed bitmasks in under $30\mu\text{s}$.
• Zero-Copy Memory-Mapped Persistence: Custom .sutra 64-byte aligned binary format allows near-instant cold starts via mmap, backed by an append-only CRC32 Write-Ahead Log (WAL) for durability.
• Embedded HTTP REST Micro-server: Built-in zero-dependency server exposes /health, /collections, /insert, and /query endpoints for microservice architectures.

📊 Benchmark Comparison

Ran on standard 4-vCPU Linux environment (5,000 documents, 128 dimensions):

🚀 Quickstart

1. Installation

Install via PyPI:

pip install sutradb-core

Or clone for local development & benchmarking:

git clone https://github.com/Sam-CodesAI/SutraDB.git
cd SutraDB
pip install -e .

2. Basic Usage (Python SDK)

from sutradb import SutraDB, Document

# Initialize SutraDB with disk persistence
db = SutraDB(persist_directory="./sutra_data")

# Create or load collection
collection = db.get_or_create_collection(name="kb", dimension=4, metric="cosine")

# Insert documents
collection.insert([
    Document(
        id="doc_1",
        vector=[0.95, 0.05, 0.10, 0.00],
        text="Deploying containerized microservices to Kubernetes",
        metadata={"category": "devops", "tier": "internal"}
    ),
    Document(
        id="doc_2",
        vector=[0.02, 0.98, 0.05, 0.01],
        text="PostgreSQL connection pooling and pgbouncer tuning",
        metadata={"category": "database", "tier": "public"}
    )
])

# Hybrid query combining semantic vector + text keywords + metadata filter
results = collection.query(
    vector=[1.0, 0.0, 0.0, 0.0],
    text="Kubernetes microservices",
    filter={"tier": "internal"},
    top_k=5,
    hybrid=True
)

for r in results:
    print(f"[{r.score:.4f}] {r.id}: {r.text}")

# Optional: Build IVF-Flat ANN index for sublinear search on large datasets
collection.build_index(n_lists=4, n_probe=2)
ann_results = collection.query(
    vector=[1.0, 0.0, 0.0, 0.0],
    top_k=5,
    use_ann=True
)

🌐 Running as an HTTP Microservice

Start the built-in HTTP server:

python3 -m sutradb.server 8765

Query via curl:

# Health check
curl http://localhost:8765/health

# Insert documents
curl -X POST http://localhost:8765/collections/demo/insert \
  -H "Content-Type: application/json" \
  -d '{"documents": [{"id": "d1", "vector": [1,0,0], "text": "Sample", "metadata": {"tag": "ai"}}]}'

# Hybrid search
curl -X POST http://localhost:8765/collections/demo/query \
  -H "Content-Type: application/json" \
  -d '{"vector": [1,0,0], "text": "Sample", "filter": {"tag": "ai"}, "top_k": 5}'

🧪 Test Suite

Run the full verification and benchmark suite:

pytest -v tests

📜 License

MIT License. Engineered by Samarth.