Let you forget you're using a database. Simple and high-performance persistent database solutions. 一个伪装成字典的数据库。 简单且高性能的持久化数据库解决方案。
Project description
English | 简体中文
FlaxKV
Let you forget you're using a database. Simple and high-performance persistent database solutions.
FlaxKV
A persistent database masquerading as a dictionary.
The flaxkv
module provides a dictionary-like interface for interacting with high-performance key-value databases (LMDB, LevelDB). It abstracts the complexities of direct database interaction, allowing users to perform CRUD (Create, Read, Update, Delete) operations in a simple and intuitive manner. You can use it just like a Python dictionary without worrying about it blocking your main process at any stage.
Use Cases
-
Key-Value Structure:
flaxkv
is suitable for storing simple key-value structured datasets. -
High-Frequency Writing:
flaxkv
is very suitable for scenarios that require high-frequency insertion/updating of data. -
Machine Learning:
flaxkv
is perfect for storing various embeddings, images, texts, and other large datasets with key-value structures in machine learning.
Key Features
-
Always up-to-date, never blocking.: It was designed from the ground up to ensure that no write operations block the user process, while users can always read the most recently written data.
-
Ease of Use: Interacting with the database feels just like using a Python dictionary! You don't even have to worry about resource release.
-
Buffered Writing: Data is buffered and scheduled for write to the database, reducing the overhead of frequent database writes.
-
High-Performance Database Backend: Uses the high-performance key-value database LMDB as its default backend.
-
Atomic Operations: Ensures that write operations are atomic, safeguarding data integrity.
-
Thread-Safety: Employs only necessary locks to ensure safe concurrent access while balancing performance.
TODO
- Client-Server Architecture
- Benchmark
Quick Start
Installation
pip install flaxkv
Usage
from flaxkv import dbdict
import numpy as np
d = dbdict('./test_db')
d[1] = 1
d[1.1] = 1 / 3
d['key'] = 'value'
d['a dict'] = {'a': 1, 'b': [1, 2, 3]}
d['a list'] = [1, 2, 3, {'a': 1}]
d[(1, 2, 3)] = [1, 2, 3]
d['numpy array'] = np.random.randn(100, 100)
d.setdefault('key', 'value_2')
assert d['key'] == 'value'
d.update({"key1": "value1", "key2": "value2"})
assert 'key2' in d
d.pop("key1")
assert 'key1' not in d
for key, value in d.items():
print(key, value)
print(len(d))
Hey! Did you... forget to call d.close()
to release resources here?
No, you don't need to manage it manually! It's all just like using a dictionary!
(Of course, you can also manually call d.close()
to release resources immediately.)
Citation
If FlaxKV
has been helpful to your research, please cite:
@misc{flaxkv,
title={FlaxKV: An Easy-to-use and High Performance Key-Value Database},
author={K.Y},
howpublished = {\url{https://github.com/KenyonY/flaxkv}},
year={2023}
}
Contributions
Feel free to make contributions to this module by submitting pull requests or raising issues in the repository.
License
FlaxKV
is licensed under the Apache-2.0 License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.