vecmindb 0.2.0

High-performance vector database with auto-tuning capabilities

VecminDB

A high-performance vector database with multiple index algorithms, optimizers, and auto-tuning capabilities.

Features

• Multiple Index Algorithms: HNSW, IVF, PQ, LSH, VPTree, ANNOY, NGT, and more
• Multi-Objective Optimization: NSGA-II, MOEAD, MOPSO for index tuning
• Auto-Tuning: Automatic parameter optimization for best performance
• Parallel Processing: Built-in support for parallel vector operations
• Flexible Storage: RocksDB-based persistent storage with transaction support
• High Concurrency: Sharding architecture with fine-grained locking (no global write lock)
• High Availability: Raft consensus with persistent WAL (Write-Ahead Log) stored in RocksDB. Supports Streaming Snapshot Downloads from the leader for zero-memory-pressure node catch-up.
• Strong Consistency: Sequential consistency for all write operations via "Propose-and-Accept" distributed workflow.
• Caching System: Multi-tier caching with Redis support
• High Performance Persistence: Auto-snapshotting and fast restoration (ms-level startup) for production reliability. Supports Coordinated Snapshot Barriers to ensure absolute index-storage consistency.
• Resource Management: Intelligent memory and CPU allocation with Admission Control (Backpressure) to prevent OOM loops during heavy ingestion or Raft replay.
• Shadow Memory Consolidation: Unified heap memory tracking for HNSW and other vector indices, eliminating "Phantom Memory" blindness.
• Memory Safety: Enforced Non-Moving GC strategies to prevent dangling pointers and Segmentation Faults during high-concurrency search operations.
• Monitoring: Real-time performance metrics, query statistics, and full-stack memory footprint reporting.
• Memory Optimization: Scalar Quantization (SQ8) support to reduce memory usage by up to 75%
• Multimodal Support: Integrated feature extraction for Image, Audio, Video, and Text
• Resilience & Chaos Engineering: Built-in fault injection for network partitioning, SIGKILL simulations, and single-node cluster orchestration.
• Security & Sandboxing: Persistent Data Encryption (DEK/KEK), Keyword-based Algorithm Sandbox, and code-level system/network access interception.
• Diagnostics & Stability: Integrated real-time stack trace capture, Streaming XOR Checksumming for multi-terabyte data integrity, and Atomic-based thread leak prevention.
• Multi-Language Support: First-class support for Rust and Python, plus HTTP API

🚀 Key Innovations (Patent Pending)

VecminDB is not just another vector database; it introduces several architectural breakthroughs:

1. Hybrid GPU Pipeline (Mixed-Precision Indexing)

Traditional GPU indexes suffer from the "Memory Wall" — dataset size is limited by VRAM. VecminDB introduces a Three-Stage Pipeline:

• L1 (CPU RAM): Fast recall using HNSW graph on compressed vectors.
• L2 (GPU VRAM): Streaming exact distance reranking using mixed precision (FP16/FP32).
• L3 (SSD): Zero-copy integration with RocksDB for billion-scale storage. Result: 10x larger dataset support on single GPU compared to Faiss/CuVS.

4. Deterministic XOR Checksumming (1B-Scale) ⭐ Industrial Reliability

For massive datasets (1B+ vectors), validating data integrity via full scans is impossible. VecminDB introduces a Streaming XOR-Bucket Checksumming mechanism:

• Partitioned Hashing: Vectors are deterministically mapped to 100+ buckets.
• XOR Combining: Checksums are combined using XOR, allowing incremental and parallel validation.
• Safety: Enables sub-second consistency checks between nodes even with petabyte-scale datasets.

2. Semantic-Aware Adaptive Sharding

Instead of random hashing, VecminDB uses a lightweight Router Index to distribute vectors based on feature clusters.

• Query Routing: Search only relevant shards (e.g., 2 out of 16), reducing I/O amplification.

• Dynamic Balancing: Hot shards are automatically split and migrated to high-performance nodes.

3. Game-Theoretic Auto-Tuning

Beyond static parameter tuning, VecminDB employs a Real-time Micro-Benchmarking approach combined with Nash Equilibrium game theory to balance Query Latency vs. Recall dynamically. The system runs isolated performance tests on live data samples to find the optimal configuration that maximizes the collective utility of all performance objectives, ensuring SLA compliance without manual DBA intervention.

Quick Start

As a Library

use vecmindb::{VectorDB, IndexType, SimilarityMetric};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a new vector database
    let db = VectorDB::new("./data")?;
    
    // Create a collection with 128-dimensional vectors using HNSW index and Cosine similarity
    let collection = db.create_collection("my_vectors", 128, IndexType::HNSW, SimilarityMetric::Cosine).await?;
    
    // Add vectors (collection is Arc<RwLock<VectorCollection>>)
    // Note: Due to Sharding, we only need a READ lock for concurrent inserts!
    // The internal shards manage their own write locks.
    {
        let coll = collection.read().await;
        coll.add_vector("vec1", &vec![0.1; 128], None).await?;
        coll.add_vector("vec2", &vec![0.2; 128], None).await?;
    }
    
    // Search for similar vectors
    let results = {
        let coll = collection.read().await;
        coll.search(
            &vec![0.15; 128], 
            10, 
            SimilarityMetric::Cosine
        ).await?
    };
    
    for result in results {
        println!("ID: {}, Score: {}", result.id, result.score);
    }
    
    Ok(())
}

// Configuration (db_config.json)
// {
//   "snapshot_interval_secs": 300
// }

As an HTTP Server (Standalone Mode)

VecminDB features a Dual-Mode Architecture. You can run it as a standalone server with a Web Dashboard and REST API.

To easily run vecmindb-server from anywhere, install it globally via cargo:

cargo install --path . --features http-server

# Start the server (includes Dashboard + API)
vecmindb-server --config ./config.yml

📊 Dashboard

Once the server is running, visit http://localhost:8080/dashboard to access the visual management console.

• Visual Management: Create collections, monitor memory/CPU, and view cluster topology.
• Zero Config: It just works out of the box.

🌐 Distributed Cluster (Optional)

VecminDB supports embedded Raft consensus with Dynamic Configuration. You can add or remove nodes on-the-fly without downtime using the Dashboard or the /api/v1/cluster/nodes API. No external Zookeeper/Etcd required.

RBAC Security

VecminDB provides enterprise-grade security:

• JWT Authentication: Secure token-based access via /api/v1/cluster/login.
• RBAC: Fine-grained permissions (Global, Collection-specific) for Users and Roles.
• TLS/HTTPS: Encrypted transport for both Public API (HTTPS) and Internal Cluster traffic (mTLS).
• Security Configuration: Sensitive credentials (API Keys, JWT Secrets, Admin Passwords) are strictly managed via environment variables.

Environment Variables

For production deployments, the following environment variables are required:

Variable	Description
VECMIN_ADMIN_KEY	Admin API Key (Read/Write access). Fail-fast if missing.
VECMIN_CLUSTER_TOKEN	Internal token for Raft cluster communication.
VECMIN_ADMIN_PASSWORD	Initial password for the 'admin' user.
VECMIN_JWT_SECRET	Secret key for signing JWT tokens.
VECMIN_MASTER_KEY	(Optional) Master key for database encryption.
VECMIN_VIEWER_KEY	(Optional) Read-only API key.

YAML Configuration: You can also use a YAML configuration file to define all advanced parameters without clogging up your command lines. Run the server with --config ./config.yml to override default settings seamlessly.

# Create a collection
curl -X POST http://localhost:8080/api/v1/collections \
  -H "x-api-key: $VECMIN_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "my_vectors", "dimension": 128, "index_type": "HNSW"}'

# Add a vector
curl -X POST http://localhost:8080/api/v1/collections/my_vectors/vectors \
  -H "x-api-key: $VECMIN_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "vec1", "values": [0.1, 0.2, ...], "metadata": {}}'

# Search vectors
curl -X POST http://localhost:8080/api/v1/collections/my_vectors/search \
  -H "x-api-key: $VECMIN_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": [0.15, 0.25, ...], "top_k": 10, "metric": "cosine"}'

🐳 Docker & Cluster Deployment

Run high-availability VecminDB clusters in seconds with Docker or built-in operation scripts:

# Using Docker Compose (Recommended for HA 3-Node Cluster)
docker-compose up -d

# Using Docker CLI (Standalone Mode)
docker build -t vecmindb .
docker run -p 8080:8080 -v ./data:/home/vecminDB/data vecmindb

# Using the built-in Bash Control Script for Bare-metal Servers
./scripts/cluster_control.sh start   # Boot 3 nodes in background
./scripts/cluster_control.sh status  # Perform cluster health check
./scripts/cluster_control.sh stop    # Clean shutdown

Installation

Add this to your Cargo.toml:

[dependencies]
vecmindb = "0.2.0"

# Optional features
vecmindb = { version = "0.2.0", features = ["http-server", "distributed"] }

Supported Index Types

• HNSW: Hierarchical Navigable Small World graphs
• IVF: Inverted File Index
• PQ: Product Quantization
• LSH: Locality-Sensitive Hashing
• VPTree: Vantage-Point Tree
• ANNOY: Approximate Nearest Neighbors Oh Yeah
• NGT: Neighborhood Graph and Tree
• KMeans: K-Means clustering index (New in v0.1.12)
• Tiered: Automated Tiered Storage Index (New in v0.1.12)
• GPU: CUDA-accelerated search pipeline
• Flat: Brute-force exact search

Performance

VecminDB is designed for high performance:

• Parallel vector operations using Rayon
• SIMD optimizations (optional)
• Efficient memory management
• Smart caching strategies
• Auto-tuning for optimal parameters

Documentation

For detailed documentation, see:

• API Documentation
• User Guide
• Examples

🐍 Python Client (Official Release)

VecminDB Python Client is distributed as a highly optimized, pre-compiled Native Binary Wheel.

pip install vecmindb

👉 Quick Start Guide - Get started in 5 minutes. 📚 Python Installation Guide - Detailed environment setups.

License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.