hyperstreamdb 0.5.4

HyperStreamDB - Serverless Index-Streaming Database with Overlay Indexing

HyperStreamDB

Serverless Index-Streaming Database with Overlay Indexing

A production-ready indexed data lake format that combines the transactional guarantees of Apache Iceberg with persistent indexes (scalar bitmaps + HNSW vector search) for blazing-fast queries on object storage.

🎯 What Makes HyperStreamDB Different?

HyperStreamDB = Iceberg + Persistent Indexes

Feature	Iceberg/Delta	HyperStreamDB
Transactional Updates	✅ Yes	✅ Yes
Time Travel	✅ Yes	✅ Yes
Scalar Indexes	❌ No	✅ RoaringBitmap
Boolean Indexes	❌ No	✅ Native Boolean
TurboQuant	❌ No	✅ TQ8 & TQ4 (8-bit/4-bit)
Fluent Indexing API	❌ No	✅ Method Chaining
Hybrid Queries	❌ No	✅ Scalar + Vector
Native SQL	❌ No	✅ DataFusion
Index-Optimized Joins	❌ No	✅ Index Nested Loop
Query Engines	Spark/Trino	Spark/Trino/Python

⚡ Iceberg V2/V3 Compliance

HyperStreamDB implements Apache Iceberg table format with full V2 and V3 feature support:

Feature	V1	V2	V3	HyperStreamDB
Sort Orders	❌	✅	✅	✅ Implemented
Partition Evolution	❌	✅	✅	✅ Implemented
Statistics (NDV)	❌	✅	✅	✅ HyperLogLog
Row Lineage	❌	❌	✅	✅ UUID + Sequence
Default Values	❌	❌	✅	✅ Schema Fields
Delete Files	❌	✅	✅	✅ Position + Equality

New APIs

import hyperstreamdb as hdb

# Create table with sort order (V2)
table = hdb.Table("s3://bucket/table")
table.replace_sort_order(["timestamp", "user_id"], ascending=[False, True])

# V3 tables automatically include row lineage
# _row_id (UUID) and _last_updated_sequence_number are added when format_version >= 3

Migration Guide: V2 → V3

Upgrading to V3 enables row-level operations and enhanced tracking:

1. Automatic: V3 metadata columns added transparently when format_version >= 3
2. No Data Rewrite: Existing data remains compatible
3. New Columns: _row_id (UUID v4), _last_updated_sequence_number (i64)

🚀 Quick Start

Installation

Standard Install (CPU + WGPU/Vulkan): The default package includes automatic high-performance hardware detection for NVIDIA CUDA, Apple Metal, Intel Graphics/XPU, and AMD ROCm.

pip install hyperstreamdb

Windows Users: HyperStreamDB is optimized for Linux/POSIX. Windows users should use WSL2.

GPU Acceleration (Optional)

For GPU-accelerated vector operations, install the appropriate backend:

NVIDIA CUDA:

# Ubuntu/Debian
sudo apt-get install cuda-toolkit-12-3
# Verify: nvidia-smi

AMD ROCm: ROCm support is now native on Linux via WGPU/Vulkan.

# Verify Vulkan support (standard in modern ROCm drivers)
vulkaninfo | grep vendor
# Verify: rocm-smi

Apple Metal:

• Included with macOS 12.3+ on Apple Silicon (M1, M2, M3, M4, M5)
• No additional installation required

Intel XPU / Graphics: Intel Arc and Data Center GPUs are supported natively on Linux.

# Verify intel-media-va-driver or similar is present
clinfo | grep Intel

See Python Vector API Documentation for detailed GPU setup instructions.

pgvector SQL Compatibility

HyperStreamDB provides full pgvector-compatible SQL syntax for vector operations:

-- Use familiar pgvector operators
SELECT id, content, 
       embedding <-> '[0.1, 0.2, 0.3]'::vector AS l2_distance,
       embedding <=> '[0.1, 0.2, 0.3]'::vector AS cosine_distance
FROM documents
WHERE category = 'science'
ORDER BY l2_distance
LIMIT 10;

-- All six distance operators supported
-- <->  L2 (Euclidean)
-- <=>  Cosine  
-- <#>  Inner Product
-- <+>  L1 (Manhattan)
-- <~>  Hamming
-- <%>  Jaccard

See pgvector SQL Guide for complete documentation.

Basic Usage

import hyperstreamdb as hdb

# Create table
table = hdb.Table("s3://bucket/my-table")

# Write data (Pandas/PyArrow)
import pandas as pd
df = pd.DataFrame({
    "id": [1, 2, 3],
    "embedding": [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]]
})
table.write_pandas(df)
table.commit()

# Create high-performance vector index (TQ8 - 4x compression)
table.add_index("embedding", "hnsw_tq8")

# Query with filters (uses indexes!)
results = table.to_pandas(filter="id > 1")

# Vector search
query_vec = [0.15, 0.25]
results = table.to_pandas(
    vector_filter={"column": "embedding", "query": query_vec, "k": 10}
)

# Hybrid query (scalar + vector)
results = table.to_pandas(
    filter="category = 'science'",
    vector_filter={"column": "embedding", "query": query_vec, "k": 10}
)

🔄 Fluent Query API

HyperStreamDB features a fluent query API in Rust with method chaining. Python uses the to_pandas() API with filter and vector_filter arguments.

Rust Fluent API

The same fluent interface is available in native Rust:

use hyperstreamdb::{Table, VectorValue};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let table = Table::new("s3://bucket/my-table")?;
    
    // Method chaining
    let results = table
        .query()
        .filter("age > 25")
        .vector_search("embedding", VectorValue::Float32(query_vec), 10)
        .select(vec!["name".to_string(), "score".to_string()])
        .to_batches()
        .await?;
    
    println!("Found {} result batches", results.len());
    Ok(())
}

Benefits

• Method Chaining: Intuitive, readable query construction
• Type Safe: Compile-time validation in Rust, runtime validation in Python
• Performance: Same underlying optimized execution as traditional APIs
• Interoperable: Mix with SQL queries and traditional to_pandas() calls
• GPU Acceleration: Automatic GPU context propagation for vector operations
• TurboQuant Optimized: Seamless integration with 8-bit/4-bit quantization

TurboQuant Quantization (TQ8 / TQ4)

HyperStreamDB features TurboQuant, an optimized quantization engine that reduces vector storage costs while maintaining high search accuracy:

• TQ8 (8-bit): 4x compression vs. float32. Near-lossless accuracy (typically >99% recall retention). Ideal for general-purpose RAG.
• TQ4 (4-bit): 8x compression vs. float32. Maximum efficiency for massive datasets where storage cost is the primary bottleneck.

# Use enterprise defaults (HNSW-TQ8)
table.add_index("embedding", "hnsw_tq8")

# High-compression mode
table.add_index("embedding", "hnsw_tq4")

# Custom HNSW-PQ configuration
table.add_index("embedding", {
    "type": "hnsw_pq",
    "complexity": 32,
    "quality": 300,
    "compression": 32 # PQ subspaces
})

Python Vector Distance API with GPU Acceleration

HyperStreamDB provides a comprehensive Python API for vector distance computations with GPU acceleration:

import hyperstreamdb as hdb
import numpy as np

# GPU-accelerated batch distance computation
ctx = hdb.GPUContext.auto_detect()  # Auto-detect CUDA/ROCm/Metal/XPU
print(f"Using GPU backend: {ctx.backend}")

# Create query and database vectors
query = np.random.randn(768).astype(np.float32)
database = np.random.randn(100000, 768).astype(np.float32)

# Compute distances on GPU (10x+ faster for large databases)
distances = hdb.l2_distance_batch(query, database, context=ctx)

# Find top-k nearest neighbors
k = 10
top_k_indices = np.argsort(distances)[:k]

# Single-pair distance computation
vec1 = np.array([1.0, 2.0, 3.0])
vec2 = np.array([4.0, 5.0, 6.0])
distance = hdb.cosine_distance(vec1, vec2)

# Sparse vector support for high-dimensional sparse data
sparse1 = hdb.SparseVector(
    indices=np.array([0, 5, 100], dtype=np.int32),
    values=np.array([1.0, 2.5, 0.8], dtype=np.float32),
    dim=1000
)
sparse2 = hdb.SparseVector(
    indices=np.array([5, 50, 100], dtype=np.int32),
    values=np.array([2.0, 1.5, 0.9], dtype=np.float32),
    dim=1000
)
distance = hdb.l2_distance_sparse(sparse1, sparse2)

# Binary vector operations (bit-packed for efficiency)
binary1 = np.packbits(np.random.randint(0, 2, 128))
binary2 = np.packbits(np.random.randint(0, 2, 128))
distance = hdb.hamming_distance_packed(binary1, binary2)

Supported GPU Backends:

• CUDA - NVIDIA GPUs (Linux, Windows via WSL2)
• ROCm - AMD GPUs (Native Linux via WGPU)
• Intel XPU - Intel Graphics (Native Linux via WGPU)
• Metal (MPS) - Apple Silicon (macOS)
• Torch Alignment - Automatically aliases cuda to rocm on AMD hardware if torch.version.hip is detected.
• CPU - Fallback for all platforms

Supported Distance Metrics:

• L2 (Euclidean), Cosine, Inner Product, L1 (Manhattan), Hamming, Jaccard

See Python Vector API Documentation for complete API reference and GPU installation instructions

SQL queries (full DataFusion support with pgvector syntax)

import hyperstreamdb as hdb session = hdb.Session() session.register("users", table)

Optional: Enable GPU acceleration for SQL queries

device = hdb.Device.auto_detect() device.activate()

Simple SQL (via table — registers as table 't')

results = table.execute_sql("SELECT * FROM t WHERE id > 100")

Vector similarity search with pgvector operators (GPU-accelerated)

results = session.sql(""" SELECT id, content, embedding <-> '[0.1, 0.2, 0.3]'::vector AS distance FROM documents WHERE category = 'science' ORDER BY distance LIMIT 10 """)

Joins (uses Index Nested Loop Join optimization)

results = session.sql(""" SELECT u.name, o.amount FROM users u JOIN orders o ON u.id = o.user_id WHERE u.category = 'premium' """)

Maintenance

table.compact()

## 📊 Real-World Testing Plan

### Phase 1: Core Stability (Current)

**Test Datasets:**
- ✅ NYC Taxi (1.5B rows, ~200GB) - Scalar filtering
- ✅ Synthetic Embeddings (10M vectors, 768-dim) - Vector search
- 🔄 Wikipedia + Embeddings (100M docs) - Hybrid queries

**Download Test Data:**
```bash
# NYC Taxi dataset
./tests/data/download_nyc_taxi.sh

# Generate synthetic embeddings
python tests/data/generate_embeddings.py

Run Benchmarks:

# Rust benchmarks
cargo bench

# Integration tests
python tests/integration/test_nyc_taxi.py

Performance Targets:

• Scalar Ingest: >10K rows/sec ✅
• Vector Ingest (768D): >22,000 rows/sec ✅ (v0.5.0)
• Query (indexed): <100ms p99 ⏱️
• Vector search: <50ms for k=10 on 10M vectors ⏱️
• Compaction: <5min for 10GB ⏱️

Benchmarking Environment: Lenovo T480

• System: Lenovo T480
• CPU: Intel(R) Core(TM) i5-8350U CPU @ 1.70GHz
• RAM: 64GB
• OS: Linux

Benchmarking Environment: Apple M4 Max

• System: MacBook Pro (M4 Max, 16-core CPU, 40-core GPU)
• Memory: 128GB Unified Memory
• OS: macOS (Arm64)
• Optimizations: target-cpu=native (NEON SIMD)
• Results (100K vectors, 768D) [OUT OF DATE - Pre-v0.5.0]:
  • Vector Ingest: 16,707 rows/sec (CPU) ✅
  • Vector Search (k=10): 819ms (CPU / NEON) ✅
  • Vector Search (k=10): 860ms (MPS GPU) ⏱️

Phase 2: Nessie Integration (Next)

Catalog Strategy:

• ✅ Use Nessie REST v2 (don't build custom catalog)
• Implement Rust client for Iceberg REST Catalog API
• Support Git-like branching for tables

Why Nessie?

• Iceberg-standard protocol
• Multi-table transactions
• Battle-tested (Netflix, Apple, Dremio)

Phase 3: Production Hardening

• Schema evolution support
• Partition evolution
• Pluggable Distributed Locking (Redis/etcd)
• CLI tools (hyperstream compact, vacuum)
• Prometheus metrics
• Error handling & retries

🏗️ Architecture

Overlay Indexing

HyperStreamDB stores indexes as sidecar files alongside Parquet data:

s3://bucket/table/
├── data/
│   ├── segment_001.parquet                   # Main Data (Parquet)
│   ├── segment_001.id.inv.parquet           # Scalar index (Inverted Parquet)
│   ├── segment_001.emb.centroids.parquet    # Vector index centroids
│   └── segment_001.emb.cluster_0.hnsw.graph # Vector index graph (HNSW)
├── _manifest/
│   ├── v1.avro                              # Manifest (Iceberg/Avro)
│   └── v2.avro
└── _metadata/
    └── v1.metadata.json

Manifest Format

Apache Iceberg V2/V3 compliant (Avro encoding):

{
  "version": 2,
  "timestamp_ms": 1705512000000,
  "entries": [
    {
      "file_path": "segment_001.parquet",
      "file_size_bytes": 104857600,
      "record_count": 1000000,
      "index_files": [
        {
          "file_path": "segment_001.id.inv.parquet",
          "index_type": "scalar",
          "column_name": "id"
        },
        {
          "file_path": "segment_001.embedding.cluster_0.hnsw.graph",
          "index_type": "vector",
          "column_name": "embedding"
        }
      ]
    }
  ],
  "prev_version": 1
}

🔌 Connectors

Spark

// Read
val df = spark.read
  .format("hyperstream")
  .option("path", "s3://bucket/table")
  .load()

// Write
df.write
  .format("hyperstream")
  .option("path", "s3://bucket/table")
  .save()

Trino

SELECT * FROM hyperstream.default.my_table
WHERE id > 100;  -- Uses scalar index

Python (Direct)

# No Spark needed for local/notebook work
import hyperstreamdb as hdb
df = hdb.Table("s3://bucket/table").query().execute()
# Or using traditional API: df = hdb.Table("s3://bucket/table").to_pandas()

🔨 Building Connectors

The Spark and Trino connectors require building shaded "fat" JARs that bundle the native Rust core.

Matrix Build

We provide a script to build a full matrix of connectors (Java 17/21, Spark 3.5/4.0):

./build-connectors.sh

Hardware Acceleration

• Standard: Build with CPU + Intel Graphics/XPU support (default).
• CUDA: Build for NVIDIA GPUs:

  ./build-connectors.sh --cuda
  

Portable Toolchain

The build script automatically downloads a project-local Maven and JDK 21 if they are missing from your system, ensuring a consistent build environment.

Artifacts

Final JARs and ZIPs are collected in the connector-artifacts/ directory.

🧪 Development

Build & Test

# Build Rust library
cargo build --release

# Run tests
cargo test

# Run benchmarks
cargo bench

# Build Python bindings
maturin develop

# Python tests
pytest tests/

Project Structure

hyperstreamdb/
├── src/
│   ├── lib.rs                  # Main library & PyO3 module registration
│   ├── core/
│   │   ├── table/              # Table API (read, write, schema, fluent query)
│   │   ├── reader/             # Index-aware Parquet reader
│   │   ├── manifest/           # Manifest management (Iceberg/Avro)
│   │   ├── index/              # HNSW, inverted, bitmap indexes
│   │   ├── catalog/            # REST, Nessie, Glue, Hive, Unity catalogs
│   │   ├── sql/                # DataFusion integration & pgvector operators
│   │   ├── planner/            # Query planner & optimizer
│   │   ├── iceberg/            # Iceberg V2/V3 metadata & schema
│   │   ├── compaction.rs       # Compaction engine
│   │   ├── maintenance.rs      # Vacuum/GC
│   │   ├── storage.rs          # Multi-cloud storage (S3, GCS, Azure, local)
│   │   ├── wal.rs              # Write-Ahead Log
│   │   ├── ffi.rs              # JNI bindings (Spark/Trino)
│   │   └── error.rs            # Structured error types
│   ├── python_binding.rs       # PyO3 bindings
│   ├── python_distance.rs      # Vector distance API
│   └── python_gpu_context.rs   # GPU device management
├── hyperstreamdb-enterprise/    # Enterprise extensions (TurboQuant, SIMD)
├── spark-hyperstream/          # Spark connector (Java)
├── trino-hyperstream/          # Trino connector (Java)
├── tests/
│   ├── integration/            # Infrastructure integration tests
│   ├── benchmarks/             # Performance benchmarks
│   └── python/                 # Python binding tests
└── benches/                    # Criterion benchmarks

📈 Roadmap

✅ Completed

• Hybrid segment format (Parquet + indexes)
• Manifest management (Iceberg-like)
• Compaction engine
• Maintenance (expire_snapshots, remove_orphan_files)
• Python bindings (Pandas-compatible)
• Native SQL support (DataFusion integration)
• pgvector-compatible SQL operators and syntax
• Index Nested Loop Join optimization
• Boolean column indexing
• Multi-table JOIN support
• Real-world testing (NYC Taxi, Wikipedia, embeddings)
• Nessie catalog integration
• Iceberg V2 compliance (Sort Orders, Partition Evolution, Statistics)
• Iceberg V3 features (Row Lineage, Default Values, HyperLogLog NDV)
• Standard Iceberg API (update_spec, replace_sort_order, rewrite_data_files, rollback_to_snapshot)
• Python Vector Distance API with GPU acceleration
• Multi-backend GPU support (CUDA, ROCm, Metal, XPU)
• Sparse and binary vector operations

📝 In Progress

• Spark/Trino connectors
• Elasticsearch-like REST Search API (hyperstreamdb-search add-on)

📋 Planned

• Pluggable Distributed Locking (Redis/etcd)
• CLI tools (hyperstream admin)
• Prometheus metrics & Grafana dashboards
• Apache Polaris catalog support (OAuth2)

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

📄 License

The Python wrapper is licensed under the MIT License. The underlying Rust engine and core database logic is licensed under the Apache License 2.0.

This project contains modified source code from various upstream open-source projects (including hnsw_rs for pre-filtering support), which were originally licensed under Apache 2.0. HyperStreamDB maintains compliance by retaining all original copyright notices and providing prominent notice of modifications in the relevant source files.

🙏 Acknowledgments

• Apache Iceberg - Inspiration for manifest design
• Apache Arrow - Columnar format
• hnsw_rs - Vector indexing
• RoaringBitmap - Scalar indexing

---

Built with ❤️ in Rust