LSM-Vec -- Memory Friendly, High Performance VDB

Website · Quick Start · Docs · Benchmark

LSM-Vec is a persistent vector database for approximate nearest-neighbor (ANN) search. It combines an HNSW graph index with Aster, a native graph-oriented LSM-tree storage engine.

Why LSM-Vec?

Minimal Memory Overhead

Unlike many vector databases that keep large index state in memory, LSM-Vec is fully disk-oriented. Its memory footprint remains small and predictable even at large data scale.

Graph-Oriented LSM-Tree Storage

LSM-Vec stores the majority of the HNSW index within Aster, which extends RocksDB with a graph data model (RocksGraph). This graph-oriented LSM-tree structure enables LSM-Vec to achieve search and update performance comparable to in-memory vector databases.

Embeddable and Easy to Use

LSM-Vec is offered as a lightweight C++ library with Python bindings. LSM-Vec can be built with just a few lines of commands. Then users can simply link the library or import the module to get started.

Features

• HNSW graph index with fully configurable hyperparameters (M, Mmax, Ml, efConstruction)
• Layer-0 edges persisted in Aster RocksGraph (LSM-tree backed)
• Upper-layer edges kept in-memory for fast navigation
• Two vector storage backends:
  • BasicVectorStorage -- contiguous flat file, offset by logical ID
  • PagedVectorStorage -- 4 KB page-managed layout with a user-space page cache (FIFO eviction)
• Batch vector read -- groups neighbor reads by page to reduce I/O during search
• Persistent metadata -- database can be closed and reopened without re-indexing
• L2 (Euclidean) and Cosine distance metrics with SIMD acceleration (AVX2/SSE2)
• Python SDK via pybind11 (pip install .)

Repository layout

LAINN/
  include/           # C++ headers (public API + internals)
  src/               # C++ source files + Python example
  python/            # pybind11 binding source
  test/              # C++ test binary entry point + Python quick-start
  data/              # dataset preparation scripts
  lib/
    aster/           # Aster submodule (RocksDB fork)
  CMakeLists.txt
  Makefile
  pyproject.toml     # Python packaging (scikit-build-core)

Prerequisites

Compiler & tools

• C++17 compiler (GCC 8+ or Clang 10+)
• CMake >= 3.10
• GNU Make
• Boost (headers only)

System libraries

Ubuntu / Debian

sudo apt-get update
sudo apt-get install -y \
  build-essential cmake libboost-dev \
  libzstd-dev libsnappy-dev liblz4-dev libbz2-dev zlib1g-dev

macOS (Homebrew)

brew install cmake boost zstd snappy lz4 bzip2

Building (C++)

Step 1 -- Build Aster

Aster is included as a Git submodule. Initialize it and build the static library:

git submodule update --init --recursive
make aster

This produces lib/aster/librocksdb.a. The build typically takes a few minutes and uses all available cores automatically.

Step 2 -- Build the LSM-Vec libraries

make lib

Outputs:

Step 3 -- Build the test binary

make bin

The executable lsm_vec is placed in build/bin/. It reads a vector dataset, builds the HNSW index, runs k-NN queries, and compares results against a ground truth file.

One-shot build

make          # equivalent to: make lib bin

Cleaning

make clean    # removes the build/ directory

Running the Test Binary

Prepare a dataset

A helper script downloads the SIFT1M corpus and creates a 100 k-vector subset with pre-computed ground truth:

cd data
python prepare_sift_100k.py
cd ..

This produces three files under data/:

Run with --data-dir (recommended)

If all three files share a common directory and follow the naming convention input.fvecs / query.fvecs / groundtruth.ivecs, point --data-dir at that directory:

./build/bin/lsm_vec \
  --db ./run/db \
  --data-dir ./data/sift_100k_ \
  --out ./run/output.txt

If the files carry a shared prefix, pass --name:

# expects: data/sift_100k_input.fvecs, data/sift_100k_query.fvecs, ...
./build/bin/lsm_vec \
  --db ./run/db \
  --data-dir ./data/ \
  --name sift_100k \
  --out ./run/output.txt

Run with explicit file paths

./build/bin/lsm_vec \
  --db ./run/db \
  --base  ./data/sift_100k_input.fvecs \
  --query ./data/sift_100k_query.fvecs \
  --truth ./data/sift_100k_groundtruth.ivecs \
  --out   ./run/output.txt

CLI Reference

Run ./build/bin/lsm_vec --help for the full list.

Required

HNSW Parameters

Storage

Runtime / Feature Switches

Output

Example with parameters

./build/bin/lsm_vec \
  --db ./run/db \
  --data-dir ./data/sift_100k_ \
  --M 8 --Mmax 16 --efc 32 --k 10 --efs 64 \
  --vec-storage 1 --paged-cache-pages 4096 \
  --batch-read \
  --stats \
  --out ./run/output.txt

Vector Storage Backends

BasicVectorStorage (--vec-storage 0)

• Vectors stored contiguously: offset = id * dim * sizeof(float)
• No user-space caching; relies entirely on the OS page cache
• Simple and fast for sequential ID access patterns

PagedVectorStorage (--vec-storage 1, default)

• Vectors organized into 4 KB pages; no vector crosses a page boundary
• Vectors sharing the same HNSW Level-1 entry point are co-located on the same page, improving spatial locality during graph traversal
• User-space FIFO page cache (--paged-cache-pages controls the capacity)
• Batch read (--batch-read): during k-NN search, all unvisited neighbor vectors in a layer-0 search step are grouped by page and read together, reducing redundant page loads

Embedding LSM-Vec in Your Application

Include headers from include/ and link against liblsmvec.a (static) or liblsmvec.so / liblsmvec.dylib (shared).

Transitive link dependencies: rocksdb (Aster), zstd, snappy, lz4, bz2, z, pthread, dl. On macOS, jemalloc is also required.

#include "lsm_vec_db.h"

lsm_vec::LSMVecDBOptions opts;
opts.dim = 128;
opts.vector_file_path = "./db/vectors.bin";

std::unique_ptr<lsm_vec::LSMVecDB> db;
auto s = lsm_vec::LSMVecDB::Open("./db", opts, &db);

// Insert
std::vector<float> vec(128, 0.1f);
db->Insert(0, vec);

// Search (uses k and ef_search from opts)
std::vector<lsm_vec::SearchResult> results;
db->SearchKnn(vec, &results);

// Close
db->Close();

Python SDK

LSM-Vec provides a Python module (lsm_vec) via pybind11. It supports Python lists and NumPy arrays as vector inputs.

Installation

Prerequisite: Aster must be built first (see Step 1). And please install ninja-build at first.

git submodule update --init --recursive   # if not done already
make aster                                 # builds lib/aster/librocksdb.a
python -m pip install .                    # builds and installs the lsm_vec module

python -m pip install . handles the entire compilation internally via scikit-build-core. You do not need to run make lib beforehand.

To verify:

python -c "import lsm_vec; print('OK')"

Quick Start

A ready-to-run example is provided at test/python_example.py:

python test/python_example.py

import lsm_vec
import os

opts = lsm_vec.LSMVecDBOptions()
opts.dim = 128
db_dir = "./run/db/"
opts.vector_file_path = os.path.join(db_dir, "vectors.bin")
opts.reinit = True

db = lsm_vec.LSMVecDB.open(db_dir, opts)

db.insert(1, [0.1] * 128)

# Search (uses k and ef_search from opts)
results = db.search_knn([0.1] * 128)
print(results[0].id, results[0].distance)

Python API Reference

lsm_vec.LSMVecDBOptions

lsm_vec.SearchOptions

lsm_vec.SearchResult

lsm_vec.LSMVecDB

NumPy Example

import numpy as np
import lsm_vec

opts = lsm_vec.LSMVecDBOptions()
opts.dim = 128
opts.k = 5
opts.ef_search = 100
opts.vector_file_path = "./run/db/vectors.bin"
opts.reinit = True

os.makedirs("./run/db/", exist_ok=True)
db = lsm_vec.LSMVecDB.open("./run/db/", opts)

vec = np.random.rand(128).astype(np.float32)
db.insert(42, vec)

query = np.random.rand(128).astype(np.float32)
results = db.search_knn(query)
for r in results:
    print(f"id={r.id}  distance={r.distance:.4f}")

db.close()

Troubleshooting

Aster RocksDB library or headers not found

Aster hasn't been built yet:

git submodule update --init --recursive
make aster

undefined reference to rocksdb::RocksGraph::AddEdge (or similar)

You are linking against system RocksDB instead of Aster. Make sure CMake resolves the library from lib/aster/, not /usr/lib/. Check the CMake output for the line Using RocksDB library at ....

libzstd not found / libsnappy not found / liblz4 not found / libbz2 not found

Install the missing library:

• Ubuntu: sudo apt-get install libzstd-dev libsnappy-dev liblz4-dev libbz2-dev
• macOS: brew install zstd snappy lz4 bzip2

FetchContent fails to download pybind11 during pip install .

In regions with restricted access to GitHub, CMake's FetchContent may fail to clone the pybind11 repository. As a workaround, install pybind11 locally first and switch CMakeLists.txt to use find_package instead:

1. Install pybind11 via conda or pip:

conda install -c conda-forge pybind11   # or: pip install pybind11

2. In CMakeLists.txt, replace the FetchContent block (inside the if(LSMVEC_BUILD_PYTHON) section) with find_package:

    # include(FetchContent)
    # FetchContent_Declare(
    #     pybind11
    #     GIT_REPOSITORY https://github.com/pybind/pybind11.git
    #     GIT_TAG v2.13.6
    # )
    # FetchContent_MakeAvailable(pybind11)
    find_package(pybind11 REQUIRED)

Then re-run python -m pip install ..

externally-managed-environment when running pip install .

Your system Python is managed by the OS package manager. Use a virtual environment or conda:

conda create -n lainn python=3.12 -y
conda activate lainn
python -m pip install .

cannot allocate memory in static TLS block (Linux, jemalloc)

This occurs when the Python module is loaded via dlopen and jemalloc's thread-local storage cannot be allocated. Preload jemalloc at process startup:

LD_PRELOAD=/lib/x86_64-linux-gnu/libjemalloc.so.2 python test/python_example.py

To make this automatic in a conda environment:

conda activate lainn
mkdir -p $CONDA_PREFIX/etc/conda/activate.d $CONDA_PREFIX/etc/conda/deactivate.d
echo 'export LD_PRELOAD=/lib/x86_64-linux-gnu/libjemalloc.so.2' \
  > $CONDA_PREFIX/etc/conda/activate.d/jemalloc.sh
echo 'unset LD_PRELOAD' \
  > $CONDA_PREFIX/etc/conda/deactivate.d/jemalloc.sh

GLIBCXX_3.4.30 not found (Linux, conda)

Conda ships an older libstdc++. Update it:

conda install -c conda-forge libstdcxx-ng>=12

ImportError: liblsmvec.so: cannot open shared object file: No such file or directory

Update library paths:

export LD_LIBRARY_PATH=/path/to/LSM_Vec/build/lib:$LD_LIBRARY_PATH