iscc-usearch 0.6.0

Scalable approximate nearest neighbor search for variable-length binary bit-vectors using NPHD metric.

iscc-usearch

Larger-than-RAM writable HNSW indexes, and variable-length binary vector search.

iscc-usearch is a Python library that extends USearch - a high-performance HNSW library adopted by ClickHouse, LangChain, and others - with three independent capabilities:

Sharded HNSW indexes (ShardedIndex) keep a single active shard in RAM for writes while completed shards are memory-mapped for reads. Works with any vector type and metric USearch supports, including user-defined distance functions. Insert throughput stays consistent and memory stays bounded as the index grows to billions of vectors.

Normalized Prefix Hamming Distance (NphdIndex, ShardedNphdIndex) compares binary vectors of mixed bit-lengths - a 64-bit query finds nearest neighbors among 256-bit vectors with comparable distances. Purpose-built for ISCC (ISO 24138) content fingerprints, also applicable to Matryoshka embeddings, perceptual hashes, and locality-sensitive hashing.

128-bit UUID keys (ShardedIndex128, ShardedNphdIndex128) extend the key space from 64-bit integers to 128-bit bytes(16) keys. Useful when your identifiers are UUIDs, 128-bit hashes, or structured multi-part keys that don't fit in a uint64.

Key features:

• Bounded memory - only one shard in RAM at a time, the rest memory-mapped
• Billions of vectors - sharded indexes scale well beyond single-machine RAM
• Incremental writes - append vectors without rebuilding the index
• Mixed bit-lengths - 64-bit and 256-bit vectors coexist in the same index
• 128-bit keys - bytes(16) UUID keys when 64-bit integers are not enough
• Any distance metric - user-defined metrics via USearch's plugin system
• Fast - inherits USearch's HNSW engine, benchmarked at 10x the throughput of FAISS

Which index class?

Class	Var-len	Keys	Shards	Use when...
NphdIndex	✓	uint64	—	Binary variable-length, fits in RAM
ShardedIndex	—	uint64	✓	Exceeds RAM, any metric
ShardedIndex128	—	128-bit	✓	Same, with 128-bit keys
ShardedNphdIndex	✓	uint64	✓	Binary variable-length, exceeds RAM
ShardedNphdIndex128	✓	128-bit	✓	Binary variable-length, 128-bit keys

Installation

pip install iscc-usearch

Quick start

Variable-length binary (NphdIndex):

import numpy as np
from iscc_usearch import NphdIndex

index = NphdIndex(max_dim=256)

# Mix 64-bit and 128-bit vectors in the same index
index.add(1, np.array([255, 128, 64, 32, 16, 8, 4, 2], dtype=np.uint8))
index.add(2, np.array([255, 128, 64, 32, 16, 8, 4, 2, 1, 0, 255, 128, 64, 32, 16, 8], dtype=np.uint8))

# Search with a 64-bit query - NPHD compares the common prefix
query = np.array([255, 128, 64, 32, 16, 8, 4, 2], dtype=np.uint8)
matches = index.search(query, count=2)

print(matches.keys)  # Nearest neighbor keys
print(matches.distances)  # NPHD distances in [0.0, 1.0]

Sharded HNSW (ShardedIndex):

import numpy as np
from iscc_usearch import ShardedIndex

# Shards are stored in a directory on disk
index = ShardedIndex(ndim=64, path="my_index", dtype="f32")

# Add vectors - shards rotate automatically when size limit is reached
keys = list(range(1000))
vectors = np.random.rand(1000, 64).astype(np.float32)
index.add(keys, vectors)

# Search across all shards
matches = index.search(vectors[0], count=10)

print(matches.keys)  # Nearest neighbor keys
print(matches.distances)  # Cosine distances

Documentation

Full documentation: https://usearch.iscc.codes/

• Tutorials - Step-by-step getting started guides
• How-to Guides - Persistence, sharding, upsert, bloom filters
• Explanation - NPHD metric, architecture, performance
• API Reference - Auto-generated from source
• Development - Dev setup, testing, and contribution guidelines

License

Apache-2.0