⬜️ HEKTOR

Vector Database Engine
High-Fidelity Semantic Search & AI Platform

A High-Performance C++23 vector database with SIMD-optimized similarity search, Dolby-compatible perceptual quantization, and comprehensive hybrid search capabilities. Designed for sub-3ms queries at billion-scale.

---

---

FEATURE GRID

Core Database

Feature	Description
HNSW Index	Production-grade HNSW for O(log n) approximate nearest neighbor with tunable M/ef parameters
Memory-Mapped Storage	Zero-copy vector access via mmap for efficient I/O and fast cold starts
Cosine Similarity	AVX-512 SIMD-optimized cosine similarity for high-throughput scoring
Euclidean Distance	AVX-512 L2 distance metric for direct spatial comparisons
Inner Product	AVX-512 dot-product similarity for maximum inner product search (MIPS)
Filtered Search	Rich metadata predicate support: boolean, range, and tag filters
Batch Operations	Multi-query and bulk ingest processing with parallel execution

Hybrid Search

Feature	Description
BM25 Full-Text	Production-ready BM25 with Porter stemming, stopword handling, and token normalization
RRF Fusion	Reciprocal Rank Fusion for robust ensemble ranking
Weighted Sum	Configurable weighted fusion (α parameter) for lexical/vector combination
CombSUM	Additive score fusion for combined relevance scoring
CombMNZ	CombMNZ fusion emphasizing precision by multiplicative weighting
Borda Count	Rank voting fusion for combining ordinal signals

Perceptual Quantization

Feature	Standards
SMPTE ST 2084 (PQ)	HDR10, Dolby Vision compatible perceptual quantization curve
HLG (Rec.2100)	Hybrid Log-Gamma support for broadcast HDR (BBC/NHK)
Gamma 2.2	sRGB gamma curve support for standard displays
Gamma 2.4	BT.1886 studio gamma profile for broadcast-grade encoding
HDR1000 Profile	HDR profile tuned for 1000 nit displays and high-dynamic-range content

RAG Pipeline

Feature	Description
Fixed-Size Chunking	Uniform chunk sizes for predictable retrieval and batching
Sentence Chunking	Natural sentence boundaries to preserve semantic units
Paragraph Chunking	Document-structure-aware chunking for contextual coherence
Semantic Chunking	Embedding-based segmentation for meaning-preserving splits
Recursive Chunking	Hierarchical splitting for multi-scale context windows
LangChain Adapter	Native adapter for LangChain integration and pipelines
LlamaIndex Adapter	Native adapter for LlamaIndex (GPT Index) integrations

Distributed System

Feature	Description
Async Replication	Low-latency async replication (sub-100ms typical) for availability
Sync Replication	Strong-consistency synchronous replication option
Hash Sharding	Consistent-hashing sharding for even data distribution
Range Sharding	Range-based partitioning for ordered key spaces
gRPC Networking	HTTP/2 gRPC with TLS/mTLS for secure RPC communication
Service Discovery	DNS/Consul based discovery and health-check integration

Observability

Feature	Description
Prometheus Metrics	50+ production metrics exposed for monitoring and alerting
OpenTelemetry	Distributed tracing and context propagation support
eBPF Profiling	Low-overhead kernel-level profiling for performance hotspots
Structured Logging	JSON structured logs with log levels and correlation IDs

Studio Native Addon

Component	Description
BM25Engine	Full-text search engine with ranking, tokenization, and scoring
KeywordExtractor	TF‑IDF based keyword extraction with configurable stoplists
HybridSearchEngine	Tight integration of vector + lexical retrieval and fusion
QueryRewriter	Query expansion and normalization for improved recall
Quantization	PQ, SQ and HDR-aware perceptual quantization implementations

Platform Support

Platform	Compiler / Notes
Windows 10/11	MSVC 19.33+ (MSVC toolchain, Windows SDK)
Ubuntu 22.04+	GCC 13+ / Clang 16+ (glibc, libstdc++ compatibility)
macOS 13+	Apple Clang 15+ (universal macOS builds)
Docker	Multi-arch images (amd64, arm64), runtime-ready containers

---

Performance

Scale	Latency (p50)	Latency (p99)	Recall@10	Throughput
100K	1.2 ms	2.8 ms	98.5%	10,000 QPS
1M	2.1 ms	4.8 ms	98.1%	10,000 QPS
10M	4.3 ms	9.2 ms	97.5%	8,000 QPS
100M	6.8 ms	15 ms	96.8%	5,000 QPS
1B	8.5 ms	22 ms	96.8%	85,000 QPS (distributed)

---

Table of Contents

• Features
• Quick Start
• Architecture
• Installation
• Usage
• Embedding Models
• Performance
• Configuration
• Documentation
• Development
• License

---

Features

Feature	Description
SIMD-Optimized Distance	AVX2/AVX-512 accelerated cosine similarity, Euclidean distance
HNSW Index	Hierarchical Navigable Small World graph for O(log n) approximate nearest neighbor
Hybrid Search	BM25 full-text search with 5 fusion algorithms (RRF, Weighted, CombSUM, CombMNZ, Borda)
Distributed System	Replication (async/sync/semi-sync), sharding (hash/range/consistent), gRPC networking
ML Framework Integration	TensorFlow C++ API and PyTorch (LibTorch) with GPU acceleration
Local Embeddings	ONNX Runtime inference for text and images without API calls
Cross-Modal Search	Unified 512-dim space for text and image embeddings
Memory-Mapped Storage	Zero-copy vector access via mmap for efficient I/O
Universal Data Ingestion	Support for XML, JSON, CSV, Excel, PDF, Parquet, SQLite, and pgvector with read & write
RAG Engine	Complete RAG pipeline with 5 chunking strategies and framework adapters
Perceptual Quantization	HDR-aware quantization (SMPTE ST 2084 PQ curve, HLG) for image/video embeddings
Comprehensive Logging	Thread-safe logging with anomaly detection and Prometheus metrics
Gold Standard Integration	Native ingestion of journals, charts, and analysis reports
Python Bindings	pybind11-based Python API for seamless integration
Thread-Safe Operations	Concurrent reads with exclusive writes
AI Training Export	Export vector pairs and triplets for model fine-tuning
Rich Metadata	JSONL storage with full attribute filtering
Database Connectors	Direct integration with SQLite and PostgreSQL with pgvector extension
Billion-Scale Support	Tested and optimized for 1B+ vectors in distributed mode

---

Quick Start

Automated Build (Recommended)

The easiest way to get started:

# Clone the repository
git clone https://github.com/amuzetnoM/hektor.git
cd hektor

# Run the automated build script (handles everything!)
./build-hektor.sh

# Or with options:
./build-hektor.sh --dev --test    # Include dev tools and run tests
./build-hektor.sh --repair        # Repair broken installation
./build-hektor.sh --clean         # Clean build from scratch

What the script does automatically:

• ✅ Detects your system (Linux, macOS, Windows)
• ✅ Installs all dependencies (Python, CMake, compilers)
• ✅ Sets up virtual environment
• ✅ Builds the project with optimizations
• ✅ Verifies installation
• ✅ Can repair regressions

Verify installation:

python verify-installation.py

Installation via pip (Easiest)

# Install from source (requires CMake and C++ compiler)
pip install hektor-vdb

# Or install with ML dependencies
pip install hektor-vdb[ml]

Requirements:

• Python 3.10+
• CMake 3.20+
• C++23 compatible compiler (GCC 13+, Clang 16+, MSVC 2022 17.3+)

Automated Setup from Source

Windows PowerShell:

git clone https://github.com/amuzetnoM/hektor.git
cd hektor
.\scripts\setup.ps1
.\scripts\build.ps1 -Release

Unix/macOS/Linux:

git clone https://github.com/amuzetnoM/hektor.git
cd hektor
chmod +x scripts/setup.sh
./scripts/setup.sh
mkdir build && cd build
cmake .. -G Ninja -DCMAKE_BUILD_TYPE=Release
ninja

Detailed Installation: See docs/02_INSTALLATION.md for complete setup instructions, system requirements, and troubleshooting. Docker/Kubernetes: See docs/03_QUICKSTART.md for quick deployment options.

First Database

import pyvdb

# Create database optimized for Gold Standard
db = pyvdb.create_gold_standard_db("./my_vectors")

# Add a document
db.add_text("Gold broke above $4,200 resistance with strong volume", {
    "type": "Journal",
    "date": "2025-12-01",
    "bias": "BULLISH"
})

# Semantic search
results = db.search("gold breakout bullish momentum", k=5)
for r in results:
    print(f"{r.score:.4f}: {r.metadata.date} - {r.metadata.type}")

---

Architecture

+-----------------------------------------------------------------------------+
|                              VECTOR STUDIO                                   |
+-----------------------------------------------------------------------------+
|                                                                             |
|  +---------------+   +---------------+   +--------------------------+       |
|  |  Text Encoder |   | Image Encoder |   |   Gold Standard Ingest   |       |
|  |  MiniLM-L6-v2 |   |  CLIP ViT-B32 |   | Journals | Charts | Rpts |       |
|  +-------+-------+   +-------+-------+   +------------+--------------+       |
|          |                   |                        |                     |
|          +--------+----------+-----------+------------+                     |
|                   |                                                         |
|          +--------v--------+                                                |
|          |   Projection    |   384-dim -> 512-dim unified space             |
|          +--------+--------+                                                |
|                   |                                                         |
|  +----------------v----------------+   +--------------------------+         |
|  |          HNSW Index             |   |     Metadata Store       |         |
|  |  M=16, ef_construction=200      |   |   JSONL, rich filters    |         |
|  +----------------+----------------+   +------------+-------------+         |
|                   |                                 |                       |
|  +----------------v---------------------------------v-------------+         |
|  |                   Memory-Mapped Storage                        |         |
|  |          vectors.bin | index.hnsw | metadata.jsonl             |         |
|  +----------------------------------------------------------------+         |
|                                                                             |
+-----------------------------------------------------------------------------+

Component Overview

Component	Technology	Purpose
Core Engine	C++23	Vector operations, HNSW index, storage
Distance Functions	AVX2/AVX-512 SIMD	Optimized similarity computation
Text Embeddings	ONNX (MiniLM-L6-v2)	Sentence embeddings (384-dim)
Image Embeddings	ONNX (CLIP ViT-B/32)	Visual embeddings (512-dim)
Python Bindings	pybind11	High-level Python API
CLI	C++	Command-line database operations

---

Installation

For detailed installation instructions, see docs/02_INSTALLATION.md.

System Requirements

Component	Minimum	Recommended
OS	Windows 10 (1903+) / Linux	Windows 11 / Ubuntu 24.04
CPU	x64 with SSE4.1	Intel 11th gen+ / AMD Zen3+ (AVX-512)
RAM	8 GB	16+ GB
Storage	5 GB	SSD with 20+ GB
Python	3.10	3.12+
CMake	3.20	3.28+
Compiler	GCC 13+ / Clang 16+ / MSVC 19.33+	GCC 14+ / Clang 18+ / MSVC Latest

Dependencies

The setup scripts automatically install required dependencies. See docs/02_INSTALLATION.md for manual installation instructions.

Manual Installation

See docs/02_INSTALLATION.md for detailed manual installation steps for each operating system.

---

Usage

Python API

import pyvdb

# Create or open database
db = pyvdb.create_database("./vectors")
# or: db = pyvdb.open_database("./vectors")

# Add text document with metadata
metadata = pyvdb.Metadata()
metadata.type = pyvdb.DocumentType.Journal
metadata.date = "2025-12-01"
metadata.bias = "BULLISH"
metadata.gold_price = 4220.50

doc_id = db.add_text("Gold analysis content here", metadata)

# Add chart image
chart_id = db.add_image("charts/GOLD.png", {"type": "Chart", "asset": "GOLD"})

# Basic search
results = db.search("gold bullish momentum", k=10)

# Filtered search
options = pyvdb.QueryOptions()
options.k = 10
options.type_filter = pyvdb.DocumentType.Journal
options.date_from = "2025-11-01"
options.date_to = "2025-12-31"
options.min_score = 0.7

results = db.query_text("resistance breakout", options)

# Cross-modal: search images with text
results = db.search("bullish flag pattern chart", k=5)

# Database management
stats = db.stats()
db.sync()
db.optimize()
db.close()

CLI Usage

# Initialize database
vdb init ./my_database

# Add documents
vdb add ./my_database --text document.txt --type Journal --date 2025-12-01
vdb add ./my_database --image chart.png --type Chart --asset GOLD

# Ingest Gold Standard outputs
vdb ingest ./my_database ../gold_standard/output

# Search
vdb search ./my_database "gold breakout" --k 10 --min-score 0.7

# Statistics
vdb stats ./my_database

# Export for training
vdb export ./my_database ./training_data.jsonl

Gold Standard Integration

from pyvdb import create_gold_standard_db, GoldStandardIngest, IngestConfig

# Create database configured for Gold Standard
db = create_gold_standard_db("./gold_vectors")

# Ingest all outputs
ingest = GoldStandardIngest(db)
config = IngestConfig()
config.gold_standard_output = "../gold_standard/output"
config.incremental = True  # Only add new documents

stats = ingest.ingest(config)
print(f"Added: {stats.journals} journals, {stats.charts} charts")

---

Embedding Models

Vector Studio uses local ONNX models for embedding generation:

Model	Type	Dimensions	Size	Latency (CPU)
all-MiniLM-L6-v2	Text	384	23 MB	~5 ms
CLIP ViT-B/32	Image	512	340 MB	~50 ms

Text embeddings are projected from 384 to 512 dimensions to enable unified cross-modal search.

Download Models

# Automatic download to ~/.cache/vector_studio/models/
python scripts/download_models.py

---

Document Types

Vector Studio recognizes these Gold Standard document types:

Type	Description	Source
Journal	Daily analysis with bias	output/Journal_*.md
Chart	Asset price charts	output/charts/*.png
CatalystWatchlist	11-category catalyst matrix	output/reports/catalysts_*.md
InstitutionalMatrix	Bank forecasts and scenarios	output/reports/inst_matrix_*.md
EconomicCalendar	Fed/NFP/CPI events	output/reports/economic_calendar_*.md
WeeklyRundown	Weekly technical summary	output/reports/weekly_rundown_*.md
ThreeMonthReport	Tactical 1-3 month outlook	output/reports/3m_*.md
OneYearReport	Strategic 12-24 month view	output/reports/1y_*.md

---

Metadata Fields

Each vector stores rich metadata extracted from Gold Standard:

Field	Type	Description
id	int	Unique vector ID
type	enum	DocumentType classification
date	string	YYYY-MM-DD format
source_file	string	Original file path
asset	string	GOLD, SILVER, DXY, etc.
bias	string	BULLISH, BEARISH, NEUTRAL
gold_price	float	Price at time of document
silver_price	float	Silver spot price
gsr	float	Gold/Silver ratio
dxy	float	Dollar index value
vix	float	Volatility index
yield_10y	float	10Y Treasury yield

---

Performance

Benchmarks

Intel Core i7-12700H, 32GB RAM, NVMe SSD

Operation	Dataset Size	Time	Throughput
Add text	1 document	8 ms	125/sec
Add image	1 image	55 ms	18/sec
Search (k=10)	100,000 vectors	2.1 ms	476 qps
Search (k=10)	1,000,000 vectors	2.9 ms	345 qps
Search (k=10)	10,000,000 vectors	4.3 ms	233 qps
Search (k=10)	100,000,000 vectors	6.8 ms	147 qps
Batch ingest	1,000 documents	12 s	83/sec

HNSW Index Quality

ef_search	Recall@10	Latency (1M vectors)
50	95.2%	2.9 ms
100	98.1%	4.8 ms
200	99.4%	8.5 ms
500	99.9%	19.3 ms

Quantization Performance (See System Snapshot v4.0.0)

Method	Memory	Recall@10	Query Time	Compression
Uncompressed	2048 MB	100.0%	2.9 ms	1x
Scalar Quant	512 MB	96.5%	2.8 ms	4x
SQ + PQ Curve	512 MB	97.8%	3.0 ms	4x
Display-Aware	512 MB	98.1%	3.1 ms	4x
Product Quant	64 MB	88.2%	1.8 ms	32x

Key: Perceptual quantization adds +1-3% recall for image/video content with minimal overhead.

Billion-Scale Performance

Metric	Value	Configuration
Total Vectors	1 billion	10-node cluster
Query Latency (p99)	8.5 ms	With sharding
Recall@10	96.8%	Maintained at scale
Throughput	85,000 QPS	Distributed

Memory Usage

Component	Size per Vector
Vector storage (512-dim float32)	2,048 bytes
HNSW index	~200 bytes
Metadata (avg)	~100 bytes
Total	~2.4 KB

---

Configuration

Database Configuration

config = pyvdb.DatabaseConfig()
config.dimension = 512                      # Vector dimensionality
config.metric = pyvdb.DistanceMetric.Cosine # Distance function
config.hnsw_m = 16                          # HNSW connections per node
config.hnsw_ef_construction = 200           # Build quality
config.hnsw_ef_search = 50                  # Search quality (default)
config.max_elements = 1_000_000             # Maximum capacity
config.provider = pyvdb.ExecutionProvider.Auto  # CPU/CUDA/DirectML

Environment Variables

Variable	Description	Default
VDB_MODELS_DIR	ONNX model directory	~/.cache/vector_studio/models
VDB_LOG_LEVEL	Logging verbosity	INFO
VDB_NUM_THREADS	Thread pool size (0=auto)	0
VDB_SIMD	SIMD level (avx512/avx2/sse4/none)	auto

---

Documentation

Comprehensive documentation is available in the docs/ directory:

#	Document	Description
00	Documentation Index	Comprehensive navigation hub for all documentation
01	Introduction	System overview, key features, and quick start guide
02	Installation	System requirements, installation steps, and configuration
03	Quick Start	Create your first database and perform basic operations
04	User Guide	Complete user guide covering all features
05	Architecture	System design, data flow, component diagrams
06	Data Formats	Supported data types and format specifications
07	Data Ingestion	Data adapters, batch processing, ingestion patterns
08	Embeddings & Models	Text and image encoders, model specifications
09	Vector Operations	HNSW algorithm, distance metrics, mathematical foundations
10	Hybrid Search	BM25 full-text search, fusion methods, RAG toolkit
11	Distributed System	Replication, sharding, gRPC networking
12	ML Framework Integration	TensorFlow and PyTorch C++ API integration
13	LLM Engine	Local text generation with llama.cpp
14	Quantization	Vector compression and quantization techniques
15	Logging & Monitoring	Logging system, Prometheus metrics, observability
16	Deployment Guide	Docker, Kubernetes, production deployment
17	Performance Tuning	Benchmarks, optimization techniques, best practices
18	Security	Security best practices and guidelines
19	Real-World Applications	Production use cases and success stories
20	API Reference	Complete C++ API documentation
21	Python Bindings	Python API reference and examples
22	Custom Development	Developing custom adapters and extensions
23	Contributing Guide	How to contribute to Vector Studio

---

Development

Building from Source

# Debug build with tests
.\scripts\build.ps1 -Debug

# Release build
.\scripts\build.ps1 -Release

# Check dependencies only
.\scripts\build.ps1 -CheckOnly

# Auto-install missing dependencies
.\scripts\build.ps1 -AutoInstall

# With GPU support
.\scripts\build.ps1 -Release -GPU

Running Tests

# C++ tests
cd build
ctest --output-on-failure

# Python tests
pytest tests/ -v

Code Quality

# Format C++ code
clang-format -i src/**/*.cpp include/**/*.hpp

# Format Python code
black scripts/ bindings/python/
isort scripts/ bindings/python/

---

Project Structure

vector_database/
+-- CMakeLists.txt          # Build configuration
+-- README.md               # This file
+-- LICENSE                 # MIT License
+-- CONTRIBUTING.md         # Contribution guidelines
+-- CHANGELOG.md            # Version history
+-- requirements.txt        # Python runtime dependencies
+-- requirements-dev.txt    # Python development dependencies
+-- .gitignore              # Git ignore rules
+-- .gitattributes          # Git attributes
+-- include/
|   +-- vdb/                # Public C++ headers
+-- src/                    # C++ implementation
|   +-- core/               # Distance, threading, vector ops
|   +-- index/              # HNSW implementation
|   +-- storage/            # Memory-mapped persistence
|   +-- embeddings/         # ONNX encoder wrappers
|   +-- cli/                # Command-line tool
+-- bindings/
|   +-- python/             # pybind11 Python bindings
+-- scripts/
|   +-- setup.ps1           # Windows setup script
|   +-- setup.sh            # Unix setup script
|   +-- build.ps1           # Windows build script
|   +-- download_models.py  # ONNX model downloader
+-- tests/                  # Unit tests
+-- docs/                   # Documentation

---

License

This project is licensed under the MIT License - see the LICENSE file for details.

---

Acknowledgments

• HNSW Paper - Malkov and Yashunin
• Sentence-Transformers - MiniLM models
• OpenAI CLIP - Vision-language models
• ONNX Runtime - Cross-platform inference

---

_{Part of the Gold Standard precious metals intelligence system.}