LSHRS

Redis-backed locality-sensitive hashing toolkit that stores bucket membership in Redis while keeping the heavy vector payloads in your primary datastore.

Table of Contents

• Positioning
• How It Works
• Installation
• Quick Start
• Ingestion
• Querying
• Persistence & Lifecycle
• API Surface
• Benchmarks
• Development & Testing
• License

Positioning

LSHRS adds cosine approximate-nearest-neighbor (ANN) search on top of the datastore you already run. Redis holds only a compact bucket -> set-of-indices index — typically much smaller than the vectors it indexes, with the gap widening as embedding dimensionality grows. The vectors themselves stay external in your system of record (PostgreSQL/pgvector, Parquet, an object store, anything) and are fetched on demand, only when you opt into cosine reranking via vector_fetch_fn.

See docs/positioning.md for the full comparison (LSHRS vs datasketch vs RediSearch HNSW vs pgvector vs FAISS) and when not to use LSHRS.

How It Works

LSHRS runs the full LSH workflow: hash vectors into banded random-projection signatures, store only bucket membership in Redis for low-latency candidate enumeration, then optionally rerank candidates by cosine similarity using vectors fetched from your system of record. It auto-selects bands/rows, pipelines Redis operations, and exposes hooks for streaming ingestion, persistence, and maintenance.

Installation

pip install lshrs              # core
pip install 'lshrs[postgres]'  # + PostgreSQL streaming (psycopg)
pip install 'lshrs[parquet]'   # + Parquet ingestion (pyarrow)

From source:

git clone https://github.com/mxngjxa/lshrs.git
cd lshrs
uv sync --dev

Quick Start

import numpy as np
from lshrs import LSHRS

def fetch_vectors(indices: list[int]) -> np.ndarray:
    # Replace with your vector store retrieval (PostgreSQL, disk, object store, etc.)
    embeddings = np.load("vectors.npy")
    return embeddings[indices]

lsh = LSHRS(
    dim=768,
    num_perm=256,
    redis_host="localhost",
    redis_prefix="demo",
    vector_fetch_fn=fetch_vectors,
)

# Stream index construction from PostgreSQL
lsh.create_signatures(
    format="postgres",
    dsn="postgresql://user:pass@localhost/db",
    table="documents",
    index_column="doc_id",
    vector_column="embedding",
)

# Insert an ad-hoc document
lsh.ingest(42, np.random.randn(768).astype(np.float32))

# Retrieve candidates
query = np.random.randn(768).astype(np.float32)
top10 = lsh.get_top_k(query, topk=10)         # fast collision lookup -> List[int]
reranked = lsh.get_above_p(query, p=0.2)      # cosine-reranked -> List[(index, score)]

Ingestion

• LSHRS.create_signatures() streams batches from PostgreSQL (iter_postgres_vectors(), server-side cursors) or Parquet (iter_parquet_vectors()). Tune batch_size, filter with where_clause, or pass a custom connection_factory for pooling/TLS.
• LSHRS.index() ingests in-memory batches; LSHRS.ingest() handles single realtime updates. Both coalesce writes through RedisStorage.batch_add().

Querying

LSHRS.query() offers two retrieval modes:

Convenience wrappers: get_top_k() and get_above_p().

Persistence & Lifecycle

Because LSHRS stores only bucket -> set-of-indices membership, LSHRS.delete() makes per-id removal cheap and exact — an id is dropped from its bands with no rebuild or compaction, which suits GDPR / right-to-be-forgotten workflows. See docs/positioning.md for details.

API Surface

Benchmarks

A reproducible benchmark harness lives in benchmarks/. It generates synthetic random unit vectors and measures index build throughput, top-k / top-p query latency (p50/p95), the Redis footprint ratio (raw N x dim x 4 bytes vs the bucket index — the ratio grows with dim), and delete latency.

# Runs anywhere via in-memory fakeredis (functional timings, estimated footprint)
uv run python benchmarks/run_benchmark.py --n 5000 --dim 256 --num-perm 128

# Use a real Redis server for measured memory (MEMORY USAGE / INFO memory)
uv run python benchmarks/run_benchmark.py --real-redis --redis-host localhost

Results print as a markdown table and are written to benchmarks/RESULTS_TEMPLATE.md. See benchmarks/README.md for full details. All numbers are machine-dependent.

Development & Testing

git clone https://github.com/mxngjxa/lshrs.git
cd lshrs
uv sync --dev
uv run pytest                  # tests
uv run ruff check .            # lint
uv run ruff format --check .   # format check

License

Licensed under the terms of LICENSE.