lshrs 0.2.0b1

Redis-backed Locality Sensitive Hashing toolkit for fast approximate nearest neighbor search

LSHRS

Redis-backed locality-sensitive hashing toolkit that stores bucket membership in Redis while keeping the heavy vector payloads in your primary datastore.

Table of Contents

• Positioning
• How It Works
• Installation
• Quick Start
• Ingestion
• Querying
• Persistence & Lifecycle
• API Surface
• Benchmarks
• Development & Testing
• License

Positioning

LSHRS adds cosine approximate-nearest-neighbor (ANN) search on top of the datastore you already run. Redis holds only a compact bucket -> set-of-indices index — typically much smaller than the vectors it indexes, with the gap widening as embedding dimensionality grows. The vectors themselves stay external in your system of record (PostgreSQL/pgvector, Parquet, an object store, anything) and are fetched on demand, only when you opt into cosine reranking via vector_fetch_fn.

See docs/positioning.md for the full comparison (LSHRS vs datasketch vs RediSearch HNSW vs pgvector vs FAISS) and when not to use LSHRS.

How It Works

LSHRS runs the full LSH workflow: hash vectors into banded random-projection signatures, store only bucket membership in Redis for low-latency candidate enumeration, then optionally rerank candidates by cosine similarity using vectors fetched from your system of record. It auto-selects bands/rows, pipelines Redis operations, and exposes hooks for streaming ingestion, persistence, and maintenance.

Concern	Component
Hashing	LSHHasher — banded random-projection signatures
Storage	RedisStorage — bucket membership via Redis sets + pipelines
Ingestion	LSHRS.create_signatures() — streams from PostgreSQL or Parquet
Reranking	top_k_cosine() — cosine similarity for candidates
Configuration	get_optimal_config() — bands/rows for a target threshold

Installation

pip install lshrs              # core
pip install 'lshrs[postgres]'  # + PostgreSQL streaming (psycopg)
pip install 'lshrs[parquet]'   # + Parquet ingestion (pyarrow)

From source:

git clone https://github.com/mxngjxa/lshrs.git
cd lshrs
uv sync --dev

[!NOTE] Requires Python >= 3.10 (see pyproject.toml).

Quick Start

import numpy as np
from lshrs import LSHRS

def fetch_vectors(indices: list[int]) -> np.ndarray:
    # Replace with your vector store retrieval (PostgreSQL, disk, object store, etc.)
    embeddings = np.load("vectors.npy")
    return embeddings[indices]

lsh = LSHRS(
    dim=768,
    num_perm=256,
    redis_host="localhost",
    redis_prefix="demo",
    vector_fetch_fn=fetch_vectors,
)

# Stream index construction from PostgreSQL
lsh.create_signatures(
    format="postgres",
    dsn="postgresql://user:pass@localhost/db",
    table="documents",
    index_column="doc_id",
    vector_column="embedding",
)

# Insert an ad-hoc document
lsh.ingest(42, np.random.randn(768).astype(np.float32))

# Retrieve candidates
query = np.random.randn(768).astype(np.float32)
top10 = lsh.get_top_k(query, topk=10)         # fast collision lookup -> List[int]
reranked = lsh.get_above_p(query, p=0.2)      # cosine-reranked -> List[(index, score)]

Ingestion

• LSHRS.create_signatures() streams batches from PostgreSQL (iter_postgres_vectors(), server-side cursors) or Parquet (iter_parquet_vectors()). Tune batch_size, filter with where_clause, or pass a custom connection_factory for pooling/TLS.
• LSHRS.index() ingests in-memory batches; LSHRS.ingest() handles single realtime updates. Both coalesce writes through RedisStorage.batch_add().

[!IMPORTANT] Install pyarrow before using the Parquet loader, or iter_parquet_vectors() raises ImportError.

Querying

LSHRS.query() offers two retrieval modes:

Mode	When	Result
Top-k (top_p=None)	Latency-critical, coarse candidates	List[int] ordered by band collisions
Top-p (top_p=0.0–1.0)	Precision-sensitive, rerank by cosine	List[Tuple[int, float]] of (index, similarity)

Convenience wrappers: get_top_k() and get_above_p().

[!CAUTION] Top-p reranking requires vector_fetch_fn at construction; otherwise it raises RuntimeError.

Persistence & Lifecycle

Operation	Reference
Inspect runtime config / Redis namespace	LSHRS.stats()
Clear all buckets for the prefix (irreversible)	LSHRS.clear()
Hard-delete specific indices	LSHRS.delete()
Save / restore projection matrices	save_to_disk() / load_from_disk()

Because LSHRS stores only bucket -> set-of-indices membership, LSHRS.delete() makes per-id removal cheap and exact — an id is dropped from its bands with no rebuild or compaction, which suits GDPR / right-to-be-forgotten workflows. See docs/positioning.md for details.

[!WARNING] LSHRS.clear() deletes every key with the configured prefix. Back up with save_to_disk() first if you need to rebuild.

API Surface

Area	Primary Entry Point
Bulk streaming ingestion	LSHRS.create_signatures()
Batch / single ingestion	LSHRS.index() / LSHRS.ingest()
Search with optional reranking	LSHRS.query()
Hash persistence	save_to_disk() / load_from_disk()
Redis maintenance	RedisStorage.clear() / RedisStorage.remove_indices()
Probability utilities	compute_collision_probability() / compute_false_rates()

Benchmarks

A reproducible benchmark harness lives in benchmarks/. It generates synthetic random unit vectors and measures index build throughput, top-k / top-p query latency (p50/p95), the Redis footprint ratio (raw N x dim x 4 bytes vs the bucket index — the ratio grows with dim), and delete latency.

# Runs anywhere via in-memory fakeredis (functional timings, estimated footprint)
uv run python benchmarks/run_benchmark.py --n 5000 --dim 256 --num-perm 128

# Use a real Redis server for measured memory (MEMORY USAGE / INFO memory)
uv run python benchmarks/run_benchmark.py --real-redis --redis-host localhost

Results print as a markdown table and are written to benchmarks/RESULTS_TEMPLATE.md. See benchmarks/README.md for full details. All numbers are machine-dependent.

[!TIP] get_optimal_config() picks bands/rows for a target similarity threshold. Large buckets signal low selectivity — raise num_perm or the threshold to trade recall for precision. Inspect distribution with SCAN 0 MATCH lsh:*.

Development & Testing

git clone https://github.com/mxngjxa/lshrs.git
cd lshrs
uv sync --dev
uv run pytest                  # tests
uv run ruff check .            # lint
uv run ruff format --check .   # format check

License

Licensed under the terms of LICENSE.