Rocksdb Python Binding
Project description
RocksDict
Key-value storage supporting any python object
Abstract
This package enables users to store, query, and delete a large number of key-value pairs on disk.
This is especially useful when the data cannot fit into RAM. If you have hundreds of GBs or many TBs of key-value data to store and query from, this is the package for you.
Installation
This package is built for macOS (x86/arm), Windows 64/32, and Linux x86.
It can be installed from pypi with pip install rocksdict
.
Plans
- set, get, del
- multi get
- support string, float, int, bytes
- support other python objects through pickle
- support BigInt
- compare BigInt by value size
- keys, values, items iterator
- options, read options, write options, all options
- SstFileWriter and bulk ingest
- column families
- write batch
- delete range
- open as secondary, with-ttl, read-only
- Snapshot
- support merge
Supported key-value types:
- key:
int, float, bool, str, bytes
- value:
int, float, bool, str, bytes
and anything that supportspickle
.
Introduction
Below is a minimal example that shows how to do the following:
- Create Rdict
- Store something on disk
- Close Rdict
- Open Rdict again
- Check Rdict elements
- Iterate from Rdict
- Batch get
- Delete storage
from rocksdict import Rdict
import numpy as np
import pandas as pd
path = str("./test_dict")
# create a Rdict with default options at `path`
db = Rdict(path)
db[1.0] = 1
db[1] = 1.0
db["huge integer"] = 2343546543243564534233536434567543
db["good"] = True
db["bad"] = False
db["bytes"] = b"bytes"
db["this is a list"] = [1, 2, 3]
db["store a dict"] = {0: 1}
db[b"numpy"] = np.array([1, 2, 3])
db["a table"] = pd.DataFrame({"a": [1, 2], "b": [2, 1]})
# close Rdict
db.close()
# reopen Rdict from disk
db = Rdict(path)
assert db[1.0] == 1
assert db[1] == 1.0
assert db["huge integer"] == 2343546543243564534233536434567543
assert db["good"] == True
assert db["bad"] == False
assert db["bytes"] == b"bytes"
assert db["this is a list"] == [1, 2, 3]
assert db["store a dict"] == {0: 1}
assert np.all(db[b"numpy"] == np.array([1, 2, 3]))
assert np.all(db["a table"] == pd.DataFrame({"a": [1, 2], "b": [2, 1]}))
# iterate through all elements
for k, v in db.items():
print(f"{k} -> {v}")
# batch get:
print(db[["good", "bad", 1.0]])
# [True, False, 1]
# delete Rdict from dict
db.close()
Rdict.destroy(path)
Supported key value types
- key:
int, float, bool, str, bytes
- value:
int, float, bool, str, bytes
and anything that supportspickle
.
Examples
Go to example folder.
There are examples about:
- column families
- batch write
- RocksDB options
- SstFileWriter
- delete_range
- snapshot
Rocksdb Options
Since the backend is implemented using rocksdb, most of rocksdb options are supported.
Limitations
Currently do not have good support for merge operation.
Full Documentation
Contribution
This project is still in an early stage of development. People are welcome to add tests, benchmarks and new features.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for rocksdict-0.2.13-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed1aee5df912b6baadcdf9203f34d0c37e41dc11cfb7a24303ca620d3ea3c37f |
|
MD5 | cc0011ab0713a475b76746b866b5f5d6 |
|
BLAKE2b-256 | 856f277b20d3642056b3ab16e286207c11997b8331e9d1cb1ad23423b7173e3b |
Hashes for rocksdict-0.2.13-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9010e6b40351f4128dc38912e04d214a07f386af90d32a80446783193792019 |
|
MD5 | 965d2e7e66497f4658bba1021399bb63 |
|
BLAKE2b-256 | decd8d537cd51f8b9204a1b02dc1f13395f15449b9d04a7d2b38cfc54a957807 |
Hashes for rocksdict-0.2.13-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 173739449d3c439b54718e4880870200cdc033d8c8db9761178c24ba7428e36c |
|
MD5 | 2ebc87e3cf5833df2f5f6437cc5227ed |
|
BLAKE2b-256 | 7bb7fabfe0c48d8549cad9606d647a8191ec8f741eda5bac7db914a9ace80fd2 |
Hashes for rocksdict-0.2.13-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b790325b697f4c655d8ecc5ecd01e42aa3fd88fc307fcb81ad971cc1510a0669 |
|
MD5 | 1f219711c2ce3ce80dcc9110849bb47e |
|
BLAKE2b-256 | 46ff29fe504b4d52203affc0242641bb0227a2c2fa9a7196c030ca395628e740 |
Hashes for rocksdict-0.2.13-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b70d422edc8bfc599c87bdc9b2c7b4fbbeee4e6238b94fd3ee9dd571e66b86a5 |
|
MD5 | e3ac260ef802c86c2105316b27b1de05 |
|
BLAKE2b-256 | c2bf718dc0a6d16e1d015f37f44570a8b7878474129d122f65ccdeb698e8bb58 |
Hashes for rocksdict-0.2.13-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1a40ab1d4106b5ca8115f5ce67939118ec19f481793a4e7efb4db2b3d1e4772 |
|
MD5 | 70899ab48b97f44fe0956bf38edcd3ab |
|
BLAKE2b-256 | 9c5437f6f2c20473082f2ee91e4bc05098bb08826636e45a71a9d0e18434bdf5 |
Hashes for rocksdict-0.2.13-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bee2994998c20b39ad220af27e6b63a775826e4a5019f2ce49e7393b7e5a778d |
|
MD5 | 19f4def1179b54c6b6c1bff5b1cfec44 |
|
BLAKE2b-256 | d1ce8fb3b1a6a361629dfc2300318a9b81325316943f25623f5f3f633bec41cd |
Hashes for rocksdict-0.2.13-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0eaeee949687f27a7ca383769b43602b24e1b32305f5b1b942e9240c3a1df3c3 |
|
MD5 | 9a0f940f8e0cc9b3003d307f1f38c4f2 |
|
BLAKE2b-256 | 04230c2f8776fd59dad7337c7ca343381aecf80ffd9752c529e4c265656bda4f |
Hashes for rocksdict-0.2.13-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1aea1d6e1bdf4886afe07fec7ee70ba3222062128dba62637e60b1bd1a0b1768 |
|
MD5 | 43fea05fc453bdb3b2d939fdeb2ea7d7 |
|
BLAKE2b-256 | 93861cfde793fc23fbe0033a93b503b1f3af1fc8302a8a8a6eca5def577721b9 |
Hashes for rocksdict-0.2.13-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 97d89d928eb957bafbbf5dde6cf753af407b921a9f784e50f9226aa79a64e890 |
|
MD5 | fb3ea1a14f98390ee991ad78cb09fb15 |
|
BLAKE2b-256 | 2db8edaa79526e1b6f88161f033c2219df773c3edd2a7429c0137ec1733e22e9 |
Hashes for rocksdict-0.2.13-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7972d7204608db42b01dd3a5f8c94bb5f79fcedf81bb7072f1a05af7cd8c631e |
|
MD5 | d36c1a6a2e24581762213b69e016724b |
|
BLAKE2b-256 | 4d21cdaee7abfb3da333b8763bcfb7584e4a4b22f9f8cab0b749de442ded8165 |
Hashes for rocksdict-0.2.13-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4a2c6d27219cf7b7855d4527e00fa0ec9334dea0f7bfcf7767c73d5696ccd81 |
|
MD5 | 306dc02a2d63cb8dd4668ab932bcb623 |
|
BLAKE2b-256 | 1971e4a500f2d9c7d9c9313b7489013841961d11d330fe87c873cc685d60ebb4 |
Hashes for rocksdict-0.2.13-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36ffcd6c4c286e12b7e860fc4e98b7d8f770bacee340ba2196b9db1365dcfa16 |
|
MD5 | 88412ec8e5286b057abe800a20fe0f82 |
|
BLAKE2b-256 | c2e36a0c1fdd8e029d7e6fd5590da528ae355653652796d5fa887ab9422f3c70 |
Hashes for rocksdict-0.2.13-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a594f790ac532dcece98e4c54a4a5c0cf55c075f7362fea5c4dcc0e918a7422 |
|
MD5 | 9caf3201349b277e14637547050d8955 |
|
BLAKE2b-256 | b77fb27b89974268dd14bc91b81402505e52260c3e957edd2d604e8381e4b9f4 |
Hashes for rocksdict-0.2.13-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4766c6657d516949e44a8619b5a18f7871e56ce46c8db8286fd91d20cad82606 |
|
MD5 | 475d35aeb5e04a15eb2fb8680250b445 |
|
BLAKE2b-256 | 3cbb5c9cdb809a831ebad4f56c3093944f77f1e24ec3c1b0f9b3c65a5fa20421 |
Hashes for rocksdict-0.2.13-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd67110d4f1d3986f3d945f95069b3bee48c8b96ebfaf8bf442223b729a622b6 |
|
MD5 | 5234a0294750c3c48b7736264028ae51 |
|
BLAKE2b-256 | 1ea1120b6b08f8933bdf7058e62bcb6da66fb208b893f6d441f3296ed0783dc9 |
Hashes for rocksdict-0.2.13-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb9bf339b9234de3fc104a568ad274266f22c73d2d5c783b4c0c703bcb27b96f |
|
MD5 | e59f3a274b9593606a1172f54b8ec8b7 |
|
BLAKE2b-256 | 50b6e18b2770512e247e66305456d0594eee414a2f76442f49a582411ddbd4fe |
Hashes for rocksdict-0.2.13-cp36-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb0c48edc318573c5dd03080af68f5b544493360ee44ce45de0cbb06c81395b6 |
|
MD5 | 4e812070e753e4c6eea2b9bb99d5e297 |
|
BLAKE2b-256 | d84d4da81192860e3582bd711cc0245b63599b150a7f88e9f5cc390902300b38 |
Hashes for rocksdict-0.2.13-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06b5263e193d73601b81d97d5228c0947e074e4808411b66f76743cd3393273c |
|
MD5 | f4dc91845f1f19df20a6963f1527453d |
|
BLAKE2b-256 | 94c273e8be824c852c3bc0c2874e072a21c5f5b53157fe517a39899f9182c429 |
Hashes for rocksdict-0.2.13-cp36-cp36m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e32b287bd1974c7ac6e1c774673f584a1d8e5a9304b290022f95ff36bdbfffd |
|
MD5 | 3de9095db979cf5f54daac4d89ac1d30 |
|
BLAKE2b-256 | 0f5562d12de84682025c94168dcd15dcfb9f3951f06322acd9aad09338311be9 |