luxdb 0.0.10

Store multidimensional vectors and quickly search nearest neighbors.

LuxDB

This is a simple database for multidimensional vectors. It basically provides persistance and connectivity with asyncio to Hnswlib. The project contains the server and also a simple client.

Still under development, there will be breaking changes and you will loose data if you only store it in this database. So don't use it for anything that you want to keep.

TODO

• Language agnostic transport layer =>
• Performance?
• Rollbacks, transactions, ...

Features

In LuxDB you can store your vectors and query them for their nearest neighbors. Vectors are stored in indexes that have a name, a dimension and a metric. After you created the index you need to initialize it with a maximum size. (See the code below for some usage examples.) And luxdb/client.py or luxdb/sync_client.py for a list of all available operations.

Changes are stored on the disk and can be used after the database is shutdown. This is achieved with ZODB, each index is stored seperatly in a OOBTree. The store can be created with a path, in that case a FileStorage will be created there. You can also provide a Storage in the constructor of the storage. For testing you can omit path and storage, in that case the data will be stored in memory only.

Authentication and encryption happens with Fernet, server and client need a shared secret in order to communicate. From this secret a key is derived, and this key is used to encrypt commands that are sent to the server. This guarantees, that only clients with the secret can execute commands on the server. The client needs the secret in the constructor, the server can either take it from the command line (--secret) or from an enviroment variable (LUXDB_SECRET).

Usage

Start the server, either with docker:

docker run -p 8484:8484 registry.gitlab.com/sacovo/luxdb
docker run -p 8484:8484 registry.gitlab.com/sacovo/luxdb --loglevel=info --port 8484 --host 0.0.0.0

Or directly (after installing the dependencies in requirements.txt)

./luxdb-server --port 8484 --loglevel debug path/to/storage.db

The docker container stores the database in /data/ so you can mount something there in order to store data.

Look into the snippets to see some example configurations and code snippets that show how to use the client.

You can then use the client to connect to the server and add or retrieve data.

from luxdb.client import connect
# Connect to the server
async with connect(host, port, secret) as client:
	name = 'my-index'
	# Create an index for 12 dimensional vectors
	await client.create_index(name, 'l2', 12)
	# Initialize the index
	await client.init_index(name, max_elements)
	# Add some data
	data = np.float32(np.random.random((1000, 12)))
	labels = np.arange(1000)
	await client.add_items(name, data, labels)
	# Search the nearest neighbors of data[0]
	found, distances = await client.query_index(name, data[0], k=5)
	# Or the nearest neighbors of all elements
	found, distances = await client.query_index(name, data, k=2)
	# You can resize the index after it was create
	await client.resize_index(name, new_size)
	# Get all ids that are stored
	ids = client.get_ids(name)
	# Get the element with specific ids
	elements = client.get_elements(name, [1, 3, 5])

For more usage examples you can check the tests in tests/test_client.py

Environment variables

Name	Descripton	Default
LUXDB_SECRET	Secret used to encrypt and authenticate communication with clients.	''
LUXBD_SALT	Salt used in key derivation from secret. Needs to be the same on client and server.	'wYfJIy4Nx1hPcxiljwg'
KDF_ITERATIONS	Iterations to use in key derivation	1 << 18
FERNET_TTL	Time in seconds that messages are valid after they are encrypted.	60

Project structure

The project consists of a wrapper around a collection of hnswlib.Index objects, a server that performs modifications and lookups and a client. Communication between the server and the client happen through Command objects.

Development

Tests

The iteration count of the key derivation function can be changed through the environment variable KDF_ITERATIONS. To speed up tests you can set it to a low value. Don't set it to a low value in any other context!