rbsp 0.1.1

Resonance-Based Semantic Protocol - Ultra-fast search and indexing

RBSP

Resonance-Based Semantic Protocol -- a zero-dependency, pure-Python search engine with BM25 ranking, HNSW vector search, resonance-based hybrid fusion, and a built-in HTTP API. 85 modules. 3,233 tests. No pip install required beyond Python 3.9+.

---

What is RBSP?

RBSP is a full-featured search engine written entirely in Python's standard library. It combines classical information retrieval (BM25, inverted index, Porter stemmer) with modern vector search (HNSW, binary/product quantization) and a novel resonance-based hybrid fusion algorithm. It ships as a library, CLI tool, and HTTP server -- all with zero external dependencies.

85 modules	Full IR stack: indexing, ranking, vector search, hybrid fusion, reranking
3,233 tests	Comprehensive suite across 85 test files -- no external services needed
0 dependencies	Pure Python stdlib. pip install rbsp and you are done
Python 3.9 -- 3.13	Tested on every supported CPython release

---

Installation

pip install rbsp

Or install from source:

git clone https://github.com/yethikrishna/rbsp-framework.git
cd rbsp-framework
pip install -e .

---

Quick Start

Python API

from rbsp import init, index, search

# Initialize and index a project
init("/path/to/project")
stats = index("/path/to/project")
print(f"Indexed {stats.files_indexed} files in {stats.duration:.2f}s")

# Search
results = search("authentication logic", max_results=10)
for r in results[:5]:
    print(f"{r.path}:{r.line}  ({r.score:.2f})")

Or use the one-line API -- resolve auto-initializes on first call:

from rbsp import resolve

results = resolve("database connection")
for r in results[:5]:
    print(f"{r.path}:{r.line}  ({r.score:.2f})")

CLI

# Initialize and index
rbsp init .
rbsp index .

# Search
rbsp search "authentication logic"

# Search with options
rbsp search "database" --limit 10 --verbose --json

# Watch for file changes and re-index automatically
rbsp watch .

# View index status
rbsp status

# Manage configuration
rbsp config max_results 50

HTTP API

Start the server:

from rbsp.runtime.server import RBSPServer

server = RBSPServer(host="127.0.0.1", port=7342, auto_init=".")
server.serve_forever()

Query via curl:

# Quick search via query params
curl "http://127.0.0.1:7342/api/v1/search?q=authentication"

# Full query with options
curl -X POST http://127.0.0.1:7342/api/v1/resolve \
  -H "Content-Type: application/json" \
  -d '{"query": "authentication logic", "max_results": 10}'

# Trigger indexing
curl -X POST http://127.0.0.1:7342/api/v1/index \
  -H "Content-Type: application/json" \
  -d '{"path": ".", "force": false}'

# Health check
curl http://127.0.0.1:7342/health

Endpoints

Method	Endpoint	Description
GET	/health	Liveness probe
GET	/api/v1/status	Engine status, metrics, and throttle stats
POST	/api/v1/resolve	Full query (JSON body)
GET	/api/v1/search?q=...	Query via query parameters
POST	/api/v1/index	Trigger indexing
POST	/api/v1/init	Initialize the engine
POST	/api/v1/watcher/start	Start file watcher
POST	/api/v1/watcher/stop	Stop file watcher
GET	/api/v1/watcher	Watcher status

---

Features

Classical Information Retrieval

• Okapi BM25 Ranking -- k1=1.2, b=0.75 with normalized scoring
• BM25F -- Field-weighted BM25 for multi-field documents
• Inverted Index -- Term-to-document posting lists with document frequency tracking
• WAND-style Early Termination -- Skips non-competitive candidates for top-K
• LSH (Locality-Sensitive Hashing) -- MinHash with configurable bands/rows
• Phrase Queries -- Exact and near-phrase matching with configurable slop
• Boolean Queries -- AND / OR / NOT query composition
• Wildcard Queries -- Prefix, suffix, and infix pattern matching
• Proximity Search -- Find terms within a positional window
• Porter Stemmer -- Algorithmic stemming at index and query time
• Spell Checking -- Edit-distance-based query correction
• Query Suggestions -- Typeahead / autocomplete from the index
• Query Expansion -- Automatic term expansion for broader recall
• Synonyms -- Synonym expansion for query enrichment
• Highlighting -- Snippet extraction with match highlighting
• Faceted Search -- Drill-down aggregations by field, directory, file type
• Freshness Scoring -- Time-decay boost for recently modified documents
• Diversity Ranking -- MMR-style result diversification
• Language Model Scoring -- Statistical language model retrieval
• Learning to Rank -- Pluggable LTR scoring with feature extraction

Vector Search

• HNSW Index -- Hierarchical Navigable Small World graph for O(log n) ANN search
• Binary Quantization -- Hamming distance via XOR + popcount on Python ints
• Product Quantization -- Subspace codebook compression for memory efficiency
• Unified VectorStore -- Four search modes: exact, hnsw, binary, hybrid
• FilterExpression DSL -- Composable metadata filters (&, |, ~ operators)
• Filtered HNSW -- 3-tier adaptive fallback (HNSW -> widened ef -> exact)
• Cosine and L2 Distance -- Pluggable distance functions

Hybrid Fusion

• Resonance Model -- Novel wave-interference-inspired fusion of BM25 + vector scores
• 3 Interference Modes -- Constructive, adaptive, and linear
• Coherence Scoring -- Based on rank agreement between signals
• Adaptive Resonance -- Self-tuning alpha via click feedback (EMA)
• Multi-Signal Resonance -- N-signal generalization beyond two rankers
• Ensemble Methods -- Linear, RRF, CombScore, CombMax, Borda, Min fusion

Reranking

• Cross-Encoder Pipeline -- Two-stage retrieve-then-rerank architecture
• Pluggable Backends -- Cohere Rerank, HuggingFace Transformers, or custom encoders
• DummyCrossEncoder -- Zero-dependency BM25-based heuristic for testing
• Score Fusion -- Linear interpolation with alpha blending

Embeddings

• Provider Abstraction -- Unified interface for OpenAI, Cohere, Ollama, and custom backends
• Document Chunking -- Fixed, sentence, and paragraph strategies with configurable overlap
• EmbeddingPipeline -- Chunk + embed in a single call
• DummyProvider -- Deterministic vectors for testing without API keys

Advanced Query

• Query Planning -- Cost-based query plan optimization
• Query Rewriting -- Automatic query normalization and transformation
• Score Explanation -- Per-result scoring breakdown for debugging

HTTP API

• REST/JSON Server -- Built on stdlib http.server with ThreadPoolExecutor
• HTTP/1.1 Keep-Alive -- Configurable socket timeout
• API Key Authentication -- Optional X-API-Key header validation
• CORS Support -- Cross-origin headers for browser access
• Rate Limiting -- Token bucket + backpressure with 429 / 503 and Retry-After
• Slow Query Log -- Queries exceeding a latency threshold are logged
• Metrics -- Counters, histograms, and gauges for monitoring
• Graceful Shutdown -- Drains in-flight requests before stopping

Durability

• Write-Ahead Log -- Binary WAL with CRC32 integrity checks and crash recovery
• 7 Operation Types -- INSERT, DELETE, UPDATE, CHECKPOINT, SEGMENT_CREATE/MERGE/DELETE
• Configurable Sync -- FSYNC, FDATASYNC, or NONE modes
• Checkpointing -- Periodic WAL truncation at safe sequence numbers
• Snapshots -- Point-in-time index snapshots
• Segments -- Segmented index architecture with background merging

Distribution

• Jump Consistent Hash -- Minimal reassignment on shard count changes (Lamping & Veach 2014)
• Shard Partitions -- Each wraps its own VectorStore + InvertedIndex
• Scatter-Gather Queries -- Parallel query across shards with result merging
• Replica Manager -- Read replicas with health tracking
• Consistency Levels -- ONE, QUORUM, ALL for reads and writes
• Online Rebalancing -- Migrate to a new shard count without downtime

Indexing

• Incremental Indexing -- Only re-indexes changed files
• Concurrent Indexing -- Thread-pool-based parallel file processing
• File Watching -- Live re-indexing on file system changes
• MMap Store -- Memory-mapped binary storage for large indexes
• Positional Postings -- Term positions stored for phrase and proximity queries
• Doc Values -- Columnar storage for sort and aggregation fields
• Deletion Tracking -- Soft deletes with periodic compaction
• Index Warmup -- Pre-loads hot data into memory on startup

Data Structures

• Bloom Filter -- Probabilistic set membership with configurable false-positive rate
• Counting Bloom Filter -- Supports deletions via counters
• Roaring Bitmaps -- Compressed integer sets for posting list intersection
• Skip Lists -- O(log n) sorted set for ordered iteration
• Finite State Transducers -- FST for fast prefix/fuzzy lookups
• Levenshtein Automata -- Edit distance computation for fuzzy matching
• Varint Encoding -- Variable-length integer encoding for compact storage
• LZ4/Snappy-style Compression -- Block compression for stored fields

Integrations

• File System -- Recursive directory traversal with gitignore-style filtering
• Git -- Git-aware indexing (respects .gitignore, tracks blame metadata)
• Plugin System -- Extensible resolvers, indexers, and rankers via plugin registry
• Schema System -- Typed field definitions with validation
• Migration System -- Index schema migrations across versions

---

Advanced Usage

Vector Search

from rbsp.indexer.vector_store import VectorStore, FieldCondition

store = VectorStore(dim=128, distance="cosine")
store.insert("doc1", [0.1] * 128, metadata={"category": "tech"})

# HNSW search with metadata filter
results = store.search(
    [0.1] * 128, k=5, mode="hnsw",
    filter_expr=FieldCondition("category", "eq", "tech"),
)

Hybrid Retrieval

from rbsp.query.hybrid import HybridRetriever, ResonanceConfig

retriever = HybridRetriever(ResonanceConfig(interference_mode="adaptive"))
fused = retriever.fuse(bm25_ranking, vector_ranking, top_k=10)

Write-Ahead Log

from rbsp.indexer.wal import WriteAheadLog, WALConfig, WALOperation

with WriteAheadLog(WALConfig(log_dir="/tmp/rbsp-wal")) as wal:
    wal.append(WALOperation.INSERT, b'{"doc_id": "abc"}')
    wal.checkpoint()
    entries = wal.recover()  # Replay after crash

Distributed Sharding

from rbsp.runtime.distributed import ShardedIndex, ShardConfig

idx = ShardedIndex(ShardConfig(num_shards=4, replication_factor=2))
idx.insert("doc1", tokens=["python", "search"], vector=[0.1] * 128)
results = idx.search_vector([0.1] * 128, k=10)

---

Architecture

rbsp/
  core/         Classical IR primitives (BM25, stemmer, LSH, bloom, HNSW,
                quantization, embeddings, chunker, roaring bitmaps, FST, ...)
  indexer/      Storage layer (inverted index, vector store, WAL, mmap,
                segments, concurrent indexing, merge, ...)
  query/        Query pipeline (hybrid fusion, reranker, ensemble, boolean,
                phrase, wildcard, spellcheck, facets, learning-to-rank, ...)
  runtime/      Orchestration (engine, config, plugins, CLI, HTTP server,
                distributed sharding, monitoring, throttle, ...)
  integration/  External integrations (filesystem, git)

How a query works

1. Parse -- Tokenize and stem via Porter stemmer.
2. Retrieve -- O(1) candidate lookup via InvertedIndex + LSH.
3. Score -- BM25 with trigram fuzzy matching (0.8 / 0.2 blend).
4. Prune -- WAND-style early termination skips non-competitive candidates.
5. Rank -- heapq.nlargest for top-K selection.
6. Snippet -- Extract best-matching lines only for final results.

Optionally, vector search (HNSW) retrieves dense-embedding candidates, resonance fusion merges BM25 and vector rankings via wave interference, and a cross-encoder reranker refines the final top candidates.

How indexing works

1. Files are discovered via recursive directory traversal (respecting .gitignore).
2. Each file is tokenized, stemmed, and added to the inverted index and LSH index.
3. Term frequencies, document lengths, and trigrams are stored for BM25 and fuzzy matching.
4. All mutations are recorded in the WAL before being applied.
5. Periodic background merging compacts segments for efficient storage.

---

Design Philosophy

1. Zero dependencies -- stdlib only, no C extensions, no pip install overhead.
2. O(1) candidate retrieval -- InvertedIndex + LSH instead of brute-force scan.
3. WAND early termination -- skips documents that cannot reach top-K.
4. heapq everywhere -- nlargest / nsmallest avoids full sort.
5. Deferred snippet extraction -- only reads files for final top-K results.
6. Binary quantization -- Python int as SIMD: XOR + popcount in 2 ops.
7. Thread-safe by design -- threading.Lock / RLock on all shared structures.
8. Lazy initialization -- heavy resources allocated only on first use.
9. MMapStore binary persistence -- fast index load and save.
10. Jump consistent hash -- minimal reassignment on shard count changes.

---

Benchmarks

RBSP ships a benchmark suite covering core operations, indexing throughput, search latency, and vector search performance:

python benchmarks/benchmark_core.py      # Signature, bloom, LSH, registry
python benchmarks/benchmark_index.py     # Indexing throughput
python benchmarks/benchmark_search.py    # Search latency vs corpus size
python benchmarks/benchmark_vector.py    # HNSW and quantized vector search
python benchmarks/benchmark_e2e.py       # End-to-end pipeline

---

Development

Prerequisites

• Python 3.9+

Setup

git clone https://github.com/yethikrishna/rbsp-framework.git
cd rbsp-framework
pip install -e ".[dev]"

Running Tests

python -m pytest tests/ -v

The full suite of 3,233 tests runs without any external services or API keys.

Python Version Support

Python	Status
3.9	Supported
3.10	Supported
3.11	Supported
3.12	Supported
3.13	Supported

---

Contributing

See CONTRIBUTING.md for guidelines on development setup, testing, code style, and the pull request process.

---

License

MIT -- Copyright (c) 2026 kkvin