flaxkv 0.1.6

A high-performance dictionary database.

🗲 FlaxKV

A high-performance dictionary database.

English | 简体中文

The flaxkv module provides a dictionary-like interface for interacting with high-performance key-value databases (LMDB, LevelDB). It abstracts the complexities of direct database interaction, allowing users to perform CRUD operations in a simple and intuitive manner. You can use it just like a Python dictionary without worrying about it blocking your main process at any stage.

Use Cases

• Key-Value Structure: flaxkv is suitable for storing simple key-value structured datasets.

• High-Frequency Writing: flaxkv is very suitable for scenarios that require high-frequency insertion/updating of data.

• Machine Learning: flaxkv is perfect for storing various embeddings, images, texts, and other large datasets with key-value structures in machine learning.

---

Key Features

• Always Up-to-date, Never Blocking: It was designed from the ground up to ensure that no write operations block the user process, while users can always read the most recently written data.

• Ease of Use: Interacting with the database feels just like using a Python dictionary! You don't even have to worry about resource release.

• Buffered Writing: Data is buffered and scheduled for write to the database, reducing the overhead of frequent database writes.

• High-Performance Database Backend: Uses the high-performance key-value database LMDB as its default backend.

• Atomic Operations: Ensures that write operations are atomic, safeguarding data integrity.

• Thread-Safety: Employs only necessary locks to ensure safe concurrent access while balancing performance.

TODO

• Client-Server Architecture
• Benchmark

---

Quick Start

Installation

pip install flaxkv

Usage

from flaxkv import dictdb
import numpy as np

db = dictdb('./test_db')
# or run server `flaxkv run --port 8000`, then:
# db = dictdb('http://localhost:8000', remote=True)

db[1] = 1
db[1.1] = 1 / 3
db['key'] = 'value'
db['a dict'] = {'a': 1, 'b': [1, 2, 3]}
db['a list'] = [1, 2, 3, {'a': 1}]
db[(1, 2, 3)] = [1, 2, 3]
db['numpy array'] = np.random.randn(100, 100)

db.setdefault('key', 'value_2')
assert db['key'] == 'value'

db.update({"key1": "value1", "key2": "value2"})

assert 'key2' in db

db.pop("key1")
assert 'key1' not in db

for key, value in db.items():
    print(key, value)

print(len(db))

You might have noticed that even when the program ends, we didn't use db.close() to release resources! Everything will be handled automatically. More importantly, as a persistent database, it offers performance close to dictionary (in-memory) access! (There should be a benchmark here.)

P.S.: Of course, you can also manually call db.close() to release resources immediately.

Citation

If FlaxKV has been helpful to your research, please cite:

@misc{flaxkv,
    title={FlaxKV: An Easy-to-use and High Performance Key-Value Database Solution},
    author={K.Y},
    howpublished = {\url{https://github.com/KenyonY/flaxkv}},
    year={2023}
}

Contributions

Feel free to make contributions to this module by submitting pull requests or raising issues in the repository.

License

FlaxKV is licensed under the Apache-2.0 License.