GigaVector

A high-performance vector database library for Python. GigaVector provides efficient similarity search with support for multiple index types, metadata filtering, and persistent storage.

Features

Core Database:

• Index types: KD-tree, HNSW, Flat, IVF-PQ, IVF-Flat, IVF-SQ8, IVF-TurboQuant, PQ, LSH, Sparse
• Distance metrics: Euclidean and Cosine similarity
• Rich metadata support with key-value pairs
• Metadata filtering in search queries
• Persistent storage with snapshot and WAL (Write-Ahead Log)
• Batch operations for vector insertion and search
• Thread-safe operations

Advanced Features:

• GPU acceleration with CUDA support
• HTTP REST API server
• BM25 full-text search
• Hybrid search (vector + text fusion)
• Backup and restore with compression
• TTL (Time-to-Live) for automatic data expiration

Enterprise Features:

• Multi-tenancy with namespaces
• Sharding for horizontal scaling
• Replication for high availability
• Cluster management
• API key and JWT authentication

Installation

pip install gigavector

Pre-built wheels for Linux (x86_64), macOS (x86_64 + arm64), and Windows (AMD64) are published to PyPI. The wheel bundles the native library and all runtime DLLs — no compiler or MinGW required.

Building from source requires a C toolchain. On Windows this means MSYS2 with MinGW-w64 (mingw-w64-x86_64-gcc, mingw-w64-x86_64-cmake, mingw-w64-x86_64-make) and C:\msys64\mingw64\bin on PATH. MinGW is a build-time dependency only and is not needed at runtime.

Quick Start

from gigavector import Database, DistanceType, IndexType

# Create an in-memory database
with Database.open(None, dimension=128, index=IndexType.HNSW) as db:
    # Add vectors with metadata
    db.add_vector([0.1] * 128, metadata={"id": "vec1", "category": "A"})
    db.add_vector([0.2] * 128, metadata={"id": "vec2", "category": "B"})
    
    # Search for similar vectors
    hits = db.search([0.1] * 128, k=5, distance=DistanceType.EUCLIDEAN)
    for hit in hits:
        print(f"Distance: {hit.distance}, Metadata: {hit.vector.metadata}")

API Reference

Database

The main class for vector database operations.

Database.open(path, dimension, index=IndexType.KDTREE, ...)

Create or open a database instance.

Parameters:

• path (str | None): File path for persistent storage. Use None for in-memory database.
• dimension (int): Vector dimension (must be consistent for all vectors).
• index (IndexType): Index type to use. Defaults to IndexType.KDTREE.
• hnsw_config (HNSWConfig | None): Used when index=IndexType.HNSW.
• ivfpq_config (IVFPQConfig | None): Used when index=IndexType.IVFPQ.
• ivfflat_config (IVFFlatConfig | None): Used when index=IndexType.IVFFLAT.
• ivfsq8_config (IVFSQ8Config | None): Used when index=IndexType.IVFSQ8.
• ivfturboquant_config (IVFTurboQuantConfig | None): Used when index=IndexType.IVFTURBOQUANT.
• pq_config (PQConfig | None): Used when index=IndexType.PQ.
• lsh_config (LSHConfig | None): Used when index=IndexType.LSH.

Returns: Database instance

Example:

# In-memory database
db = Database.open(None, dimension=128, index=IndexType.HNSW)

# Persistent database
db = Database.open("vectors.db", dimension=128, index=IndexType.KDTREE)

add_vector(vector, metadata=None)

Add a single vector to the database.

Parameters:

• vector (Sequence[float]): Vector data as a sequence of floats. Length must match database dimension.
• metadata (dict[str, str] | None): Optional dictionary of key-value metadata pairs.

Raises:

• ValueError: If vector dimension doesn't match database dimension.
• RuntimeError: If insertion fails.

Example:

# Vector without metadata
db.add_vector([1.0, 2.0, 3.0])

# Vector with single metadata entry
db.add_vector([1.0, 2.0, 3.0], metadata={"id": "123"})

# Vector with multiple metadata entries
db.add_vector([1.0, 2.0, 3.0], metadata={
    "id": "123",
    "category": "electronics",
    "price": "99.99"
})

add_vectors(vectors)

Add multiple vectors to the database in batch. Vectors added via this method cannot include metadata.

Parameters:

• vectors (Iterable[Sequence[float]]): Iterable of vectors. All vectors must have the same dimension.

Raises:

• ValueError: If vectors have inconsistent dimensions.
• RuntimeError: If batch insertion fails.

Example:

vectors = [
    [1.0, 2.0, 3.0],
    [4.0, 5.0, 6.0],
    [7.0, 8.0, 9.0]
]
db.add_vectors(vectors)

search(query, k, distance=DistanceType.EUCLIDEAN, filter_metadata=None)

Search for k nearest neighbors to a query vector.

Parameters:

• query (Sequence[float]): Query vector. Length must match database dimension.
• k (int): Number of nearest neighbors to return.
• distance (DistanceType): Distance metric to use. Defaults to DistanceType.EUCLIDEAN.
• filter_metadata (tuple[str, str] | None): Optional metadata filter as (key, value) tuple. Only vectors matching the filter are considered.

Returns: list[SearchHit] - List of search results, ordered by distance (ascending).

Raises:

• ValueError: If query dimension doesn't match database dimension.
• RuntimeError: If search fails.

Example:

# Basic search
hits = db.search([1.0, 2.0, 3.0], k=5, distance=DistanceType.EUCLIDEAN)

# Search with metadata filter
hits = db.search(
    [1.0, 2.0, 3.0],
    k=5,
    distance=DistanceType.EUCLIDEAN,
    filter_metadata=("category", "electronics")
)

search_batch(queries, k, distance=DistanceType.EUCLIDEAN)

Search for k nearest neighbors for multiple query vectors in batch.

Parameters:

• queries (Iterable[Sequence[float]]): Iterable of query vectors.
• k (int): Number of nearest neighbors to return per query.
• distance (DistanceType): Distance metric to use. Defaults to DistanceType.EUCLIDEAN.

Returns: list[list[SearchHit]] - List of search result lists, one per query.

Raises:

• ValueError: If any query dimension doesn't match database dimension.
• RuntimeError: If batch search fails.

Example:

queries = [
    [1.0, 2.0, 3.0],
    [4.0, 5.0, 6.0]
]
results = db.search_batch(queries, k=5)
for i, hits in enumerate(results):
    print(f"Query {i}: {len(hits)} results")

save(path=None)

Persist the database to a binary snapshot file. If a file path was provided when opening the database, writes to that path. Otherwise, use the provided path.

Parameters:

• path (str | None): Optional file path. If None and database was opened with a path, uses that path.

Raises:

• RuntimeError: If save operation fails.

Example:

# Save to the path used when opening
db.save()

# Save to a different path
db.save("backup.db")

train_ivfpq(data)

Train the IVF-PQ index. Required before insert when using IndexType.IVFPQ.

Parameters:

• data (Sequence[Sequence[float]]): Training vectors. All vectors must match the database dimension.

Raises:

• ValueError: If training data is empty or dimensions don't match.
• RuntimeError: If training fails.

Example:

train_data = [[(i % 10) / 10.0 for _ in range(128)] for i in range(256)]
db.train_ivfpq(train_data)

train_ivfflat(data)

Train the IVF-Flat index. Required before insert when using IndexType.IVFFLAT.

train_ivfsq8(data)

Train the IVF-SQ8 index (coarse centroids + scalar quantizer). Required before insert when using IndexType.IVFSQ8.

train_ivfturboquant(data)

Train the IVF-TurboQuant index (IVF centroids only; TurboQuant needs no codebook training). Required before insert when using IndexType.IVFTURBOQUANT. Vector dimension must be even.

train_pq(data)

Train the PQ index. Required before insert when using IndexType.PQ.

close()

Close the database and release resources. Automatically called when using the context manager.

Example:

db = Database.open(None, dimension=128)
# ... use database ...
db.close()

IndexType

• IndexType.KDTREE — exact search, low/medium dimension
• IndexType.HNSW — approximate graph index
• IndexType.IVFPQ — IVF + product quantization; requires training
• IndexType.SPARSE — sparse vectors
• IndexType.FLAT — brute-force exact search
• IndexType.IVFFLAT — IVF with full float vectors in lists; requires training
• IndexType.PQ — product quantization; requires training
• IndexType.LSH — locality-sensitive hashing
• IndexType.IVFSQ8 — IVF with 8-bit scalar-quantized vectors in lists; requires training
• IndexType.IVFTURBOQUANT — IVF with TurboQuant (PolarQuant + optional QJL) in lists; requires training; even dimension

Index config dataclasses

Pass the matching config to Database.open() for the chosen index type.

• HNSWConfig — M, ef_construction, ef_search, ...
• IVFPQConfig — nlist, m, nbits, nprobe, default_rerank, ...
• IVFFlatConfig — nlist, nprobe, train_iters, use_cosine
• IVFSQ8Config — nlist, nprobe, train_iters, use_cosine, per_dimension, default_rerank
• IVFTurboQuantConfig — nlist, nprobe, train_iters, use_cosine, default_rerank, turbo (TurboQuantConfig)
• TurboQuantConfig — bits, projections, seed, use_qjl, rotation (TurboQuantRotation: AUTO, FHWT, QR)
• PQConfig — m, nbits, train_iters
• LSHConfig — num_tables, num_hash_bits, seed, bucket_width

Per-query nprobe for IVF-Flat, IVF-SQ8, and IVF-TurboQuant: db.search_with_params(query, k, params=SearchParams(nprobe=16)).

DistanceType

Enumeration of distance metrics.

• DistanceType.EUCLIDEAN: Euclidean (L2) distance.
• DistanceType.COSINE: Cosine similarity distance.

Vector

Data class representing a vector with metadata.

Attributes:

• data (list[float]): Vector data.
• metadata (dict[str, str]): Dictionary of metadata key-value pairs.

SearchHit

Data class representing a search result.

Attributes:

• distance (float): Distance from the query vector.
• vector (Vector): The matched vector with its metadata.

Usage Examples

Persistent Storage with WAL

from gigavector import Database, IndexType, DistanceType

# Create a persistent database
with Database.open("vectors.db", dimension=128, index=IndexType.KDTREE) as db:
    db.add_vector([0.1] * 128, metadata={"id": "1", "tag": "A"})
    db.add_vector([0.2] * 128, metadata={"id": "2", "tag": "B"})
    db.save()  # Create snapshot

# Reopen - WAL automatically replays any uncommitted changes
with Database.open("vectors.db", dimension=128, index=IndexType.KDTREE) as db:
    hits = db.search([0.1] * 128, k=5)
    # All vectors are restored, including metadata

IVFPQ Index with Training

from gigavector import Database, IndexType, DistanceType
import random

# Create IVFPQ database
db = Database.open(None, dimension=64, index=IndexType.IVFPQ)

# Generate training data (at least 256 vectors recommended)
train_data = [
    [random.random() for _ in range(64)]
    for _ in range(256)
]
db.train_ivfpq(train_data)

# Add vectors
with db:
    for i in range(1000):
        vec = [random.random() for _ in range(64)]
        db.add_vector(vec, metadata={"id": str(i)})
    
    # Search
    query = [random.random() for _ in range(64)]
    hits = db.search(query, k=10, distance=DistanceType.EUCLIDEAN)

IVF-Flat with Training

from gigavector import Database, IndexType, IVFFlatConfig, DistanceType

cfg = IVFFlatConfig(nlist=64, nprobe=4)
db = Database.open(None, dimension=64, index=IndexType.IVFFLAT, ivfflat_config=cfg)
db.train_ivfflat(train_data)  # at least nlist vectors
# ... add vectors, search ...

IVF-SQ8 with Training

from gigavector import Database, IndexType, IVFSQ8Config, DistanceType

cfg = IVFSQ8Config(nlist=64, nprobe=4, default_rerank=200)
db = Database.open(None, dimension=64, index=IndexType.IVFSQ8, ivfsq8_config=cfg)
db.train_ivfsq8(train_data)  # at least nlist vectors
# ... add vectors, search ...

IVF-TurboQuant with Training

from gigavector import Database, IndexType, IVFTurboQuantConfig, TurboQuantConfig, DistanceType

cfg = IVFTurboQuantConfig(
    nlist=64,
    nprobe=4,
    default_rerank=200,
    turbo=TurboQuantConfig(bits=8, projections=16, use_qjl=True),
)
db = Database.open(None, dimension=64, index=IndexType.IVFTURBOQUANT, ivfturboquant_config=cfg)
db.train_ivfturboquant(train_data)  # at least nlist vectors; dimension must be even
# ... add vectors, search ...

Metadata Filtering

from gigavector import Database, IndexType, DistanceType

with Database.open(None, dimension=128, index=IndexType.HNSW) as db:
    # Add vectors with different categories
    db.add_vector([0.1] * 128, metadata={"category": "A", "price": "10"})
    db.add_vector([0.2] * 128, metadata={"category": "B", "price": "20"})
    db.add_vector([0.15] * 128, metadata={"category": "A", "price": "15"})
    
    # Search only in category A
    hits = db.search(
        [0.1] * 128,
        k=10,
        distance=DistanceType.EUCLIDEAN,
        filter_metadata=("category", "A")
    )
    # Returns only vectors with category="A"

Batch Operations

from gigavector import Database, IndexType, DistanceType

with Database.open(None, dimension=128, index=IndexType.KDTREE) as db:
    # Batch insert vectors (without metadata)
    vectors = [[i * 0.01] * 128 for i in range(1000)]
    db.add_vectors(vectors)
    
    # Batch search
    queries = [[i * 0.01] * 128 for i in range(10)]
    results = db.search_batch(queries, k=5)
    for i, hits in enumerate(results):
        print(f"Query {i}: {len(hits)} results")

Advanced Features

GPU Acceleration

from gigavector import gpu_available, gpu_device_count, gpu_get_device_info, GPUIndex, GPUConfig

# Check GPU availability
if gpu_available():
    print(f"GPU devices: {gpu_device_count()}")
    info = gpu_get_device_info(0)
    print(f"Device 0: {info.name}, {info.total_memory // 1024**2} MB")

    # Create GPU-accelerated index
    config = GPUConfig(device_id=0, use_float16=True)
    gpu_index = GPUIndex(dimension=128, config=config)
    gpu_index.add_vectors(vectors)
    results = gpu_index.search(query, k=10)

HTTP REST Server

from gigavector import Database, Server, ServerConfig, IndexType

# Create database and server
db = Database.open(None, dimension=128, index=IndexType.HNSW)
config = ServerConfig(port=8080, enable_cors=True)

with Server(db, config) as server:
    server.start()
    print("Server running on http://localhost:8080")
    # Server handles REST API requests:
    # GET  /health - Health check
    # POST /vectors - Add vector
    # POST /search - Search vectors
    # GET  /stats - Server statistics

BM25 Full-Text Search

from gigavector import BM25Index, BM25Config

# Create BM25 index for text search
config = BM25Config(k1=1.2, b=0.75)
bm25 = BM25Index(config)

# Add documents
bm25.add_document(0, "Machine learning for vector databases")
bm25.add_document(1, "Neural networks and deep learning")
bm25.add_document(2, "Vector similarity search algorithms")

# Search
results = bm25.search("vector search", k=10)
for r in results:
    print(f"Doc {r.doc_id}: score={r.score:.4f}")

bm25.close()

Hybrid Search (Vector + Text)

from gigavector import Database, BM25Index, HybridSearcher, HybridConfig, IndexType

db = Database.open(None, dimension=128, index=IndexType.HNSW)
bm25 = BM25Index()

# Add vectors and corresponding documents
for i, (vec, text) in enumerate(zip(vectors, documents)):
    db.add_vector(vec, metadata={"id": str(i)})
    bm25.add_document(i, text)

# Create hybrid searcher
config = HybridConfig(vector_weight=0.7, text_weight=0.3)
hybrid = HybridSearcher(db, bm25, config)

# Search with both vector and text
results = hybrid.search(query_vector, "search query", k=10)
for r in results:
    print(f"Index {r.vector_index}: combined={r.combined_score:.4f}")

hybrid.close()

Namespaces (Multi-Tenancy)

from gigavector import NamespaceManager, NamespaceConfig

# Create namespace manager
ns_mgr = NamespaceManager("/path/to/data")

# Create isolated namespaces for different tenants
config = NamespaceConfig(name="tenant_a", dimension=128)
tenant_a = ns_mgr.create(config)

config = NamespaceConfig(name="tenant_b", dimension=128)
tenant_b = ns_mgr.create(config)

# Each namespace is isolated
tenant_a.add_vector([0.1] * 128)
tenant_b.add_vector([0.2] * 128)

print(f"Tenant A vectors: {tenant_a.count}")
print(f"Tenant B vectors: {tenant_b.count}")

ns_mgr.close()

TTL (Time-to-Live)

from gigavector import TTLManager, TTLConfig

# Create TTL manager for automatic expiration
config = TTLConfig(
    default_ttl_seconds=3600,  # 1 hour default
    cleanup_interval_seconds=60
)
ttl = TTLManager(config)

# Set TTL for vectors
ttl.set_ttl(vector_index=0, ttl_seconds=1800)  # 30 minutes

# Get stats
stats = ttl.get_stats()
print(f"Vectors with TTL: {stats.total_vectors_with_ttl}")
print(f"Expired: {stats.total_expired}")

ttl.close()

Authentication

from gigavector import AuthManager, AuthConfig, AuthType

# Create auth manager with API key authentication
config = AuthConfig(auth_type=AuthType.API_KEY)
auth = AuthManager(config)

# Generate API key
key, key_id = auth.generate_api_key("My Application")
print(f"API Key: {key}")
print(f"Key ID: {key_id}")

# Authenticate requests
result, identity = auth.authenticate(key)
if result == AuthResult.SUCCESS:
    print(f"Authenticated: {identity.key_id}")

auth.close()

Backup and Restore

from gigavector import (
    Database, backup_create, backup_restore, backup_verify,
    BackupOptions, RestoreOptions, BackupCompression
)

# Create backup
options = BackupOptions(
    compression=BackupCompression.ZSTD,
    include_metadata=True
)
result = backup_create(db, "backup.gvb", options)
print(f"Backup created: {result.vectors_backed_up} vectors")

# Verify backup
if backup_verify("backup.gvb"):
    print("Backup is valid")

# Restore to new database
restore_opts = RestoreOptions(verify_checksums=True)
restored_db = backup_restore("backup.gvb", "restored.db", restore_opts)

Requirements

• Python 3.9 or higher
• cffi >= 1.16
• CUDA toolkit (optional, for GPU acceleration)

On Windows, MinGW-w64 is only required when building from source. The PyPI wheel bundles the MinGW runtime DLLs so no extra software is needed at runtime.

License

Licensed under the DBaJ-NC-CFL License. See LICENCE.md for details.

Links

• GitHub Repository: https://github.com/jaywyawhare/GigaVector