AkshamDB — Fast, Persistent, Embeddable Vector Database
v0.1.0 — A self-hosted vector database built from scratch in Python and C++. Designed for semantic search and RAG applications where you want to own the entire stack.
What Is This?
AkshamDB combines three production systems under one engine:
- LSM Store — Write-Ahead Log (with fsync) → Memtable → SSTables with Bloom filters and background compaction. Crash-safe ACID transactions.
- HNSW Index — C++ approximate nearest-neighbor graph implementing the full Malkov & Yashunin 2018 algorithm. GIL-free search, persistent thread pool, AVX2 batch scoring.
- Query Engine — 5-step pipeline: metadata filter pushdown (O(1)) → ANN retrieval → candidate merge → LSM fetch → hybrid scoring (70% cosine + 30% Okapi BM25).
Optional scale layers: Product Quantization (4–32× memory compression) and IVF coarse index (Voronoi-cell partitioning for 10M+ vector corpora).
Unlike hosted solutions (Pinecone, Weaviate), you own the whole stack — storage format, index, scoring, and server.
Dependencies: Core requires only numpy. The optional REST API server (pip install akshamdb[server]) additionally needs Flask. No sklearn/scipy in the core path — the BM25 keyword index is implemented from scratch.
Architecture
┌──────────────────────────────────────────┐
│ REST API (Flask) │
│ akshamdb-server / python -m akshamdb │
└─────────────────┬────────────────────────┘
│
┌─────────────────▼────────────────────────┐
│ AkshamDB (public API) │
│ akshamdb.open() → add / search / delete │
└─────────────────┬────────────────────────┘
│
┌─────────────────▼────────────────────────┐
│ VectorDBEngine │
│ ┌───────────┐ ┌────────┐ ┌─────────┐ │
│ │ LSMStore │ │ HNSW │ │ Query │ │
│ │ (Python) │ │ (C++) │ │ Engine │ │
│ └───────────┘ └────────┘ └─────────┘ │
│ ┌───────────┐ ┌────────┐ │
│ │ PQ │ │ IVF │ (optional) │
│ │(compress) │ │(coarse)│ │
│ └───────────┘ └────────┘ │
└──────────────────────────────────────────┘
│ │ │
[WAL + SSTables] [index.bin] [manifest.json]
[Bloom filters] [pq_*.pkl] [ivf_index.pkl]
What Has Been Built
Phase 1 — Foundation
| Component | Details |
|---|---|
| LSM Store | WAL append (with fsync for crash durability) → in-memory OrderedDict (memtable) → SSTable files on flush. Atomic WAL truncation after each flush. Background compaction thread merges SSTables with atomic rename. Thread-safe SSTable ID allocation. Bloom filter per SSTable skips disk on definite misses. Full ACID: begin / commit / rollback. |
| HNSW (C++) | Multi-layer navigable small-world graph implementing Malkov & Yashunin 2018. Flat contiguous vector storage. Format v3 serialization (4-byte HNSW magic header + version + full edge lists). Supports reading legacy v1/v2 files. Exposed to Python via pybind11 with GIL released during search(). |
| Query Engine | Five-step pipeline: (1) filter pushdown on in-memory metadata index, (2) ANN retrieval via HNSW, (3) intersect filter ∩ ANN candidates, (4) fetch docs from LSM, (5) hybrid cosine + Okapi BM25 scoring (alpha=0.7 vector, 0.3 BM25). |
| Embedder | all-mpnet-base-v2 via Sentence-Transformers. 768-dim float32 vectors. |
| Reranker | cross-encoder/ms-marco-MiniLM-L-6-v2 CrossEncoder for post-search reranking. |
| Chunker | ingestion/chunker.py — sentence, paragraph, and word-based strategies with configurable chunk size (default 512 chars) and overlap (default 128 chars). |
| REST API | Optional Flask server (pip install akshamdb[server]). Start with akshamdb-server or python -m akshamdb.server. Insert, search, batch search, get, delete, transactions, health, stats endpoints. |
Phase 2 — Parallelism & Throughput
| Feature | Details |
|---|---|
| GIL-free Search | C++ HNSW::search() releases the GIL — concurrent Python threads run in native code without waiting on each other. |
| Persistent Thread Pool | C++ thread pool started at engine init. Reused across calls, no per-query spawn overhead. Controlled by Config.PARALLEL_SEARCH_WORKERS. Explicit destructor prevents thread leak on shutdown. |
| Batch Search (search_batch_reuse) | Phase 1: all query traversals run in the C++ thread pool in parallel. Phase 2: build a union of candidates and score them with an AVX2 blocked kernel — each candidate vector loaded exactly once for all queries. Python-layer scoring is vectorised via numpy batch matmul. Typical 3–10× throughput over a Python loop of single searches, depending on corpus size and batch size. |
| Python-level Parallel Fallback | search_batch_parallel uses ThreadPoolExecutor as a fallback when search_batch_reuse is unavailable. GIL-free C++ search lets threads genuinely overlap. |
Phase 3 — Scale & Reliability
| Feature | Details |
|---|---|
| Three-tier Startup | Tier 1 (Fast): saved HNSW node count matches LSM doc count → load index + remap IDs only. Tier 2 (Incremental): saved HNSW has fewer nodes → load index + add only delta docs from WAL. Tier 3 (Full rebuild): no saved index. Startup cost is O(delta), not O(N). |
| Product Quantization (PQ) | Subspace k-means codebooks (no sklearn dependency). train → encode → adc_distances pipeline. Asymmetric Distance Computation replaces exact cosine when active. Codebooks and codes persist to disk (pq_codebooks.pkl, pq_codes.npy). Enable when memory > ~50 GB. |
| IVF Coarse Index | Partitions corpus into K Voronoi cells via k-means. Search probes only nprobe nearest cells. Supports IVF+PQ mode (ADC scoring inside lists). Enable when HNSW latency is too high at 10M+ vectors. |
| Profile-Driven Optimization | tools/profile_hotspots.py — cProfile analysis of the full insert + search pipeline. Reports time breakdown by subsystem (HNSW graph build, WAL/JSON, LSM/SSTable, threading, Python overhead), hottest functions, WAL microbenchmarks. Includes a _build_recommendations() engine that rates bottlenecks HIGH / MEDIUM / INFO and prints concrete action items. |
| Runtime Index Tuning | index.set_ef_search(n) tunes HNSW search quality at runtime without rebuilding the graph. Takes effect on the next search call. |
| pip-installable Package | setup.py wires CMake into pip's build process. pip install -e . configures and compiles the C++ extension automatically. akshamdb-server CLI entry point registered via pyproject.toml. |
| Bloom Filters | Per-SSTable Bloom filter saved alongside each .bin file. Skips disk I/O for definite key misses during reads. |
| Ghost Node Handling | On rollback, id_map entries are purged so rolled-back docs are invisible to all future searches. HNSW graph nodes can't be removed — they stay as ghost nodes. On close(), the index file is not overwritten if ghosts exist, so the last clean save is preserved for fast-path startup. |
Phase 5 — Correctness & Performance Fixes (v0.3.0)
| Fix | Details |
|---|---|
| Real Okapi BM25 | The keyword scorer was previously sklearn TF-IDF cosine similarity. It is now a pure-Python Okapi BM25 implementation (k1=1.5, b=0.75) backed by an inverted index. Only documents containing at least one query term are scored, making retrieval sub-linear in corpus size. No sklearn dependency in the core path. |
| search_batch O(N²) fix | Previously, each query in a batch called cosine_similarity(query_vec, full_matrix) individually — O(M×N×V) with poor BLAS utilisation. Real BM25 with an inverted index eliminates the dense matrix multiply entirely. Python-layer vector scoring is now a single numpy batch matmul instead of per-doc cosine loops. |
| Vectorised candidate scoring | _hybrid_score_and_rank now stacks all candidate vectors into a matrix and computes all cosine similarities in one doc_mat @ query call, replacing N individual np.dot calls. |
| print() → logging | All print() calls in db_engine.py, lsm_store.py, and ivf_index.py replaced with logging.getLogger(__name__). Control verbosity via Python's standard logging config. |
| sklearn removed from core deps | scikit-learn is no longer a required dependency. Core akshamdb only needs numpy. |
Phase 4 — Correctness & Algorithm Fixes (v0.2.0)
| Fix | Details |
|---|---|
| Real HNSW Beam Search | search_layer now uses two priority queues (Malkov & Yashunin 2018, Algorithm 2): C (min-heap of candidates to expand) and W (max-heap of current best ef neighbors). Early termination fires when the closest unexplored candidate is farther than the worst node in W, giving O(ef · log(ef) · M) complexity vs the previous O(N) BFS. |
| SELECT-NEIGHBORS-HEURISTIC | Neighbor selection during add_point now uses Algorithm 4 from the paper. The heuristic rejects a candidate c if an already-selected neighbor r satisfies dist(c, r) < dist(c, query) — meaning r already "covers" the direction toward c. This prevents graph connectivity from collapsing into one region of space and keeps recall high at all ef values. Falls back to filling from remaining candidates if the heuristic is too aggressive. |
| Neighbor Degree Cap | mmax0 = 2M at layer 0, mmax = M at upper layers, matching the paper exactly. Over-degree neighbors are pruned using the same heuristic, not a simple truncation. |
| Thread-safe RNG | get_random_level() uses thread_local std::mt19937 seeded from std::random_device{}() — each thread gets its own generator, eliminating lock contention on the level-assignment path. |
| Format v3 Serialization | Index files now begin with a 4-byte magic HNSW header followed by an integer version field. This makes format detection unambiguous — the old v2 sentinel (FORMAT_VERSION=2 as the first int) collided with dim=2 in v1 files. v1 and v2 files are still readable on load. |
| WAL Fsync Durability | _write_wal() calls f.flush() + os.fsync(f.fileno()) after every write. WAL truncation after flush uses an atomic rename (wal.log.trunc → wal.log) to prevent partial truncation on crash. |
| LSM Thread-safe SST IDs | _alloc_sst_id() now acquires _sst_lock internally so concurrent compaction threads can't race to allocate the same SSTable ID. |
| HybridSearch In-place Upsert | add_documents() checks _doc_id_to_idx before appending. If the same doc_id is added twice, the text is updated in-place at the existing slot instead of creating a ghost entry. The O(1) swap-remove in remove_document() and O(1) update in update_document() are correct in all cases. |
| Delete Syncs BM25 | delete() now calls _bm25.remove_document() and syncs query_engine.hybrid_search so deleted documents don't appear in keyword results after removal. |
| Parameter Clamping | HNSW constructor clamps M, ef_construction, and ef_search to ≥ 1 so an invalid configuration can never cause an empty-heap crash in search_layer. |
| Dead Code Removed | Duplicate Metrics class removed from utils/schema.py (the real one lives in utils/metrics.py). Dead distance(vector, vector) private method removed from C++ HNSW (was never called). |
Installation
Prerequisites: Python 3.8+, CMake 3.15+, GCC/Clang with C++14 support
Option A — install from wheel (fastest)
python3 -m venv venv
source venv/bin/activate
pip install akshamdb-0.1.0-cp310-cp310-linux_x86_64.whl
Option B — pip install from source (recommended for development)
setup.py wires the CMake build into pip, so a single command configures, compiles, and installs the C++ extension:
python3 -m venv venv
source venv/bin/activate
# Core only (no embedding model, no server)
pip install -e .
# With REST API server
pip install -e ".[server]"
# Development (pytest, benchmarks)
pip install -e ".[dev]"
Option C — manual build
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cd cpp && mkdir build && cd build
cmake .. -Dpybind11_DIR=$(python -m pybind11 --cmakedir) -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
cp hnsw_cpp*.so ../../akshamdb/
cd ../../
Rebuild wheel from source
rm -f akshamdb/hnsw_cpp*.so
python3 -m build --wheel
# wheel lands at dist/akshamdb-0.1.0-cp310-cp310-linux_x86_64.whl
Use python -m build (PEP 517), not python setup.py bdist_wheel directly — the
latter bypasses pyproject.toml's [project] metadata and silently produces a
broken akshamdb-0.0.0 wheel instead.
Verify
python3 -c "import akshamdb; print(akshamdb.__version__)"
Quick Start (Python API)
import akshamdb
# Open (or create) a database
db = akshamdb.open("./my_db", dim=768)
# Insert a single document
db.add("doc_1", vector, text="Hello world", metadata={"source": "wiki"})
# Insert many at once (faster than a loop)
db.add_many(ids, vectors, texts=texts, metadata=meta_list)
# Search — returns [{"id", "score", "text", "metadata"}, ...]
results = db.search(query_vec, top_k=5)
# Filtered search — metadata keys are ANDed
results = db.search(query_vec, top_k=5, filter={"source": "wiki"})
# Hybrid search — blends vector cosine (70%) with BM25 (30%)
results = db.search(query_vec, top_k=5, text="what is deep learning")
# Batch search — parallel ANN, much faster than N individual searches
batch = db.search_batch([vec1, vec2, vec3], top_k=5)
# Fetch and delete
doc = db.get("doc_1")
db.delete("doc_1")
# Check membership
if "doc_1" in db:
print("exists")
# Stats
print(db.info()) # documents, index_nodes, data_path, performance
db.close()
Context Manager
with akshamdb.open("./my_db", dim=768) as db:
db.add("doc_1", vector)
results = db.search(query_vec, top_k=5)
# db.close() called automatically
Transactions
with db.transaction():
db.add_many(ids, vectors, texts=texts)
# commits on clean exit, rolls back on any exception
Ingest throughput note: Outside a transaction,
add_many()saves the HNSW index to disk after every call (crash-safe default). This limits throughput to roughly 100–200 docs/s for 768-dim vectors on a typical SSD. Wrapping bulk ingestion inwith db.transaction()batches all writes into one WAL flush and one index save, achieving 800–1200+ docs/s. For multi-million-document corpora, always use transactions.
Logging
AkshamDB uses standard Python logging — no output is printed unless you configure a handler:
import logging
logging.basicConfig(level=logging.INFO) # show startup and rebuild messages
# or silence everything:
logging.getLogger("akshamdb").setLevel(logging.WARNING)
With Embedding
from embedding.embedder import Embedder
import akshamdb
embedder = Embedder() # loads all-mpnet-base-v2
with akshamdb.open("./my_db", dim=768) as db:
texts = ["Machine learning is a subset of AI.", "Neural networks learn from data."]
vecs = embedder.encode(texts)
db.add_many(["doc_0", "doc_1"], list(vecs), texts=texts, metadata=[{"src": "wiki"}] * 2)
q_vec = embedder.encode(["What is deep learning?"])[0]
for r in db.search(q_vec, top_k=3, text="deep learning"):
print(f"{r['score']:.3f} {r['text']}")
With Chunking + Ingestion
from ingestion.loader import prepare_documents
from embedding.embedder import Embedder
import akshamdb
raw_docs = ["Long document text here ...", "Another document ..."]
chunks, ids, metadata = prepare_documents(raw_docs)
embedder = Embedder()
vecs = embedder.encode(chunks)
with akshamdb.open("./my_db", dim=768) as db:
db.add_many(ids, list(vecs), texts=chunks, metadata=metadata)
REST API
Start the Server
# Default: port 8000, data at ./data
akshamdb-server
# Custom settings via env vars
AKSHAMDB_PATH=/my/db AKSHAMDB_DIM=768 AKSHAMDB_PORT=9000 AKSHAMDB_TOP_K=10 akshamdb-server
# Or: python -m akshamdb.server
# Production (gunicorn, 4 workers)
gunicorn -w 4 -b 0.0.0.0:8000 "akshamdb.server:create_app()"
Endpoints
| Method | Endpoint | Description |
|---|---|---|
POST |
/documents |
Insert one or many documents |
GET |
/documents/<id> |
Fetch document (?include_vector=true for raw vector) |
DELETE |
/documents/<id> |
Delete a document |
POST |
/search |
Single search — hybrid vector + BM25 |
POST |
/search/batch |
Batch search — parallel ANN |
GET |
/health |
Live count: documents + index nodes |
GET |
/stats |
Full performance metrics |
Examples
# Insert (text auto-embedded if sentence-transformers is installed)
curl -X POST http://localhost:8000/documents \
-H "Content-Type: application/json" \
-d '{"documents": [
{"text": "Machine learning is cool", "metadata": {"source": "wiki"}},
{"id": "my-id", "vector": [0.1, 0.2, ...], "text": "raw vector insert"}
]}'
# Search by text query
curl -X POST http://localhost:8000/search \
-H "Content-Type: application/json" \
-d '{"query": "deep learning", "top_k": 5}'
# Search with metadata filter
curl -X POST http://localhost:8000/search \
-H "Content-Type: application/json" \
-d '{"query": "AI", "top_k": 5, "filters": {"source": "wiki"}}'
# Batch search
curl -X POST http://localhost:8000/search/batch \
-H "Content-Type: application/json" \
-d '{"queries": [{"query": "deep learning"}, {"query": "neural nets"}], "top_k": 3}'
# Health check
curl http://localhost:8000/health
# Stats
curl http://localhost:8000/stats
Configuration
All flags live in utils/schema.py → Config:
class Config:
# Storage
DATA_PATH = "data"
MEMTABLE_LIMIT = 1000 # flush to SSTable after N documents in memory
COMPACT_THRESHOLD = 4 # merge SSTables when this many accumulate
BATCH_SIZE = 1000
# HNSW Index
VECTOR_DIM = 768 # all-mpnet-base-v2
HNSW_M = 16 # graph connections per node (16 for production recall)
HNSW_EF_CONSTRUCTION = 200 # build quality (higher = better recall, slower inserts)
HNSW_EF_SEARCH = 150 # candidates explored per query (150 for high recall)
# Query
DEFAULT_TOP_K = 5
ENABLE_RERANKING = True
FILTER_PUSHDOWN = True
# Parallelism
ENABLE_METRICS = True
ENABLE_PARALLEL_SEARCH = True # GIL-free per-query parallel scoring
PARALLEL_SEARCH_WORKERS = None # None = os.cpu_count()
ENABLE_BATCH_REUSE = True # AVX2 union-of-candidates batch path
# Product Quantization — enable when memory > ~50 GB
ENABLE_PQ = False
PQ_M = 96 # subspaces (must divide VECTOR_DIM evenly)
PQ_K = 256 # centroids per subspace (uint8 → max 256)
# IVF Coarse Index — enable when corpus > ~10M vectors
ENABLE_IVF = False
IVF_N_CLUSTERS = 256 # Voronoi cells — rule of thumb: sqrt(corpus_size)
IVF_NPROBE = 8 # cells probed at search time
Tuning cheat sheet:
| Situation | Setting |
|---|---|
| Need better recall | Raise HNSW_EF_SEARCH (150+) or HNSW_M (16–32) |
| Slow inserts | Lower HNSW_EF_CONSTRUCTION (100) — small recall cost |
| Memory > 50 GB | ENABLE_PQ = True — 4–32× compression, ~5% recall loss |
| Corpus > 10M vectors | ENABLE_IVF = True, IVF_N_CLUSTERS = int(sqrt(N)) |
| Recall too low with IVF | Raise IVF_NPROBE or disable IVF and use HNSW only |
| Runtime ef tuning | db._engine.index.set_ef_search(200) — no rebuild needed |
Benchmarking
All tools are in tools/. Data directories are generated at runtime and are not committed.
| Script | What it does |
|---|---|
tools/e2e_benchmark.py |
Core benchmark function: insert N docs, run M queries, report insert throughput and search latency (avg / p50 / p99). Optional Recall@K vs brute-force. |
tools/run_bench_suite.py |
Runs four preset scenarios: large 20K (vector-only), hybrid 5K (text+vector), HNSW m8/ef20, HNSW m16/ef50. |
tools/compare.py |
Head-to-head: AkshamDB vs FAISS vs ChromaDB — insert throughput, search p50/p99, memory delta, Recall@K, recovery time. |
tools/profile_hotspots.py |
cProfile hotspot analysis: time breakdown by subsystem (HNSW, WAL, LSM, threading), top hot functions, WAL microbenchmarks, and a recommendation engine that rates bottlenecks HIGH / MEDIUM / INFO with specific action items. |
tools/tune_parallel_search.py |
Sweep over thread-pool sizes and ef values. Writes a JSON report of QPS vs latency trade-offs. |
tools/cleanup_baseline_benchmarks.py |
Deletes old baseline_benchmark_YYYYMMDD_HHMMSS/ dirs, keeping the newest N. |
run_baseline_benchmark.py |
Quick baseline: python run_baseline_benchmark.py <n_docs> <n_iters> |
Run the full suite
cd tools
python run_bench_suite.py
# results saved to benchmark_results/
Profile hotspots
python tools/profile_hotspots.py
Compare against FAISS / ChromaDB
pip install faiss-cpu chromadb # optional — skipped gracefully if missing
python tools/compare.py # 10K vectors, dim=128
python tools/compare.py --n 50000 --dim 256 --queries 500
python tools/compare.py --n 10000 --skip-chroma --out results/bench.json
Tune parallel search
python tools/tune_parallel_search.py # quick sweep (n=5000)
python tools/tune_parallel_search.py --n 20000 # larger sweep
Testing
python3 test_core.py # quick smoke test
python3 -m pytest tests/test_engine.py -v # unit + integration suite
python3 -m pytest tests/test_reliability.py -v # crash recovery and WAL tests
python3 -m pytest tests/test_benchmark_integration.py -v # benchmark integration
python3 run_baseline_benchmark.py 1000 10 # 1000 docs, 10 iterations
Data Layout
After the database is created at <data_path>/:
<data_path>/
wal.log — Write-Ahead Log (append-only, JSON lines, fsynced after each write)
manifest.json — ordered list of active SSTable files
sst_1.bin — SSTable (pickled dict of documents)
sst_1.bloom — Bloom filter for sst_1.bin
...
index.bin — HNSW graph (C++ binary format, v3 serialization with magic header)
pq_codebooks.pkl — PQ subspace codebooks (if ENABLE_PQ)
pq_codes.npy — encoded corpus codes, shape (N, M) uint8 (if ENABLE_PQ)
pq_hnsw_ids.npy — hnsw_id per row in pq_codes.npy (if ENABLE_PQ)
ivf_index.pkl — IVF inverted lists and centroids (if ENABLE_IVF)
Index format compatibility:
- v3 (current): 4-byte magic
HNSW+ int version + data. Unambiguous detection. - v2 (readable):
FORMAT_VERSION=2sentinel as first int. - v1 (readable): raw dim as first int. Edge lists not saved — requires full rebuild on load.
Troubleshooting
C++ extension not found
cd cpp && rm -rf build && mkdir build && cd build
cmake .. -Dpybind11_DIR=$(python -m pybind11 --cmakedir) -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) && cp hnsw_cpp*.so ../../akshamdb/
Data not persisting — always call db.close() or use the context manager:
with akshamdb.open("./data", dim=768) as db:
db.add(...)
# close() is called automatically
Slow search — raise ef_search at runtime without rebuilding:
db._engine.index.set_ef_search(200)
# or permanently in Config:
Config.HNSW_EF_SEARCH = 200
Config.HNSW_M = 16
High memory usage — enable PQ compression:
Config.ENABLE_PQ = True
# Codebooks are trained automatically on the next full rebuild
Search recall too low — raise IVF nprobe or disable IVF:
Config.IVF_NPROBE = 32 # probe more cells
# or
Config.ENABLE_IVF = False # fall back to full HNSW scan
Old index file fails to load — v3 reader handles v1/v2/v3 automatically. If you see a load error, delete index.bin and let it rebuild from LSM on next startup.
Changelog
v0.3.0 (2026-06-24)
- [Fix] Real Okapi BM25 — keyword scorer replaced: TF-IDF cosine similarity (sklearn) → Okapi BM25 with an inverted index (k1=1.5, b=0.75, stdlib only). Scoring is now sub-linear in corpus size.
- [Fix] search_batch O(N²) — batch BM25 no longer runs dense matrix multiplies per query. Real BM25 inverted index eliminates the bottleneck. Python scoring path vectorised with numpy batch matmul.
- [Fix] Vectorised candidate scoring —
_hybrid_score_and_rankuses a singledoc_mat @ querycall instead of per-docnp.dotloops. - [Fix] print() → logging — all lifecycle/rebuild messages in
db_engine.py,lsm_store.py,ivf_index.pyuselogging.getLogger(__name__). No output unless you configure a handler. - [Fix] sklearn removed from core —
scikit-learnis no longer a required dependency. Core install needs onlynumpy. - [Docs] Ingest throughput clarified — transaction requirement for high throughput documented explicitly.
- [Docs] REST API (Flask server) and optional features (PQ, IVF) documented.
v0.2.0 (2026-06-19)
- [Fix] Real HNSW beam search — dual priority-queue implementation (Algorithm 2, Malkov & Yashunin 2018). O(ef·log(ef)·M) vs previous O(N) BFS.
- [Fix] SELECT-NEIGHBORS-HEURISTIC — neighbor selection during insert now uses Algorithm 4. Diverse neighbor sets keep the graph navigable across all regions of space.
- [Fix] Neighbor degree cap —
mmax0 = 2Mat layer 0,mmax = Mat upper layers. Over-degree lists pruned with the heuristic, not a truncation. - [Fix] Thread-safe RNG —
get_random_level()usesthread_local mt19937seeded per thread fromrandom_device. No mutex needed on the level-assignment path. - [Fix] Index format v3 —
HNSWmagic prefix + version field eliminates format-detection ambiguity. v1/v2 files still load correctly. - [Fix] WAL fsync —
_write_wal()callsfsyncafter every entry. Truncation uses atomic rename. - [Fix] LSM SST ID thread safety —
_alloc_sst_id()holds_sst_lockinternally. - [Fix] HybridSearch duplicate doc_id —
add_documents()updates in-place on duplicate; no zombie BM25 entries. - [Fix] Delete syncs BM25 — deleted docs removed from keyword index and query engine state.
- [Fix] HNSW destructor — explicit
~HNSW()releases the thread pool, no resource leak on shutdown. - [Fix] Parameter clamping —
M,ef_construction,ef_searchclamped to ≥ 1. - [Remove] Dead code — duplicate
Metricsclass inschema.py; deaddistance(vector, vector)in C++ HNSW. - [Config] HNSW_M default raised from 8 → 16 for production-quality graph connectivity.
- [Config] HNSW_EF_SEARCH default raised from 50 → 150 for high recall on real corpora.
v0.1.0
- Initial release: LSM store, HNSW index, query engine, hybrid BM25+vector scoring, PQ, IVF, batch search, REST API.
Built With
- C++ + pybind11 — HNSW graph, AVX2 distance kernel, persistent thread pool, GIL-free search
- Python — LSM store, query engine, hybrid search, ingestion pipeline, REST API
- Sentence-Transformers —
all-mpnet-base-v2embedding model (768-dim) - CrossEncoder —
ms-marco-MiniLM-L-6-v2reranker - Flask — REST API server
- NumPy / scikit-learn — vector ops, TF-IDF BM25 scoring, PQ k-means
Metadata
Release files for akshamdb 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| akshamdb-0.1.0.tar.gz | 266.4 kB | Without provenance | Details for akshamdb-0.1.0.tar.gz |
Release files / akshamdb-0.1.0.tar.gz
| Download URL | akshamdb-0.1.0.tar.gz |
|---|---|
| Size | 266.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
741758e8b2f19279a6fe54c1334110a54354f47f055cc920c927442d955623ed
|
|
BLAKE2b-256 checksum How to use checksums |
e48cf1ba1eb6e1b3cd9ec93d0dc499c848954b48e8b5812cfbc32e1523b12718
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.10.12
|
Security for akshamdb 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.