On-disk Dict With RocksDB (dbm alternative)
Project description
RocksDict
Key-value storage supporting any python object
Abstract
This package enables users to store, query, and delete a large number of key-value pairs on disk.
This is especially useful when the data cannot fit into RAM. If you have hundreds of GBs or many TBs of key-value data to store and query from, this is the package for you.
Installation
This package is built for macOS (x86/arm), Windows 64/32, and Linux x86.
It can be installed from pypi with pip install rocksdict
.
Introduction
Below is a code example that shows how to do the following:
- Create Rdict
- Store something on disk
- Close Rdict
- Open Rdict again
- Check Rdict elements
- Iterate from Rdict
- Batch get
- Delete storage
from rocksdict import Rdict
path = str("./test_dict")
# create a Rdict with default options at `path`
db = Rdict(path)
# storing numbers
db[1.0] = 1
db[1] = 1.0
# very big integer
db["huge integer"] = 2343546543243564534233536434567543
# boolean values
db["good"] = True
db["bad"] = False
# bytes
db["bytes"] = b"bytes"
# store anything
db["this is a list"] = [1, 2, 3]
db["store a dict"] = {0: 1}
# for example numpy array
import numpy as np
import pandas as pd
db[b"numpy"] = np.array([1, 2, 3])
db["a table"] = pd.DataFrame({"a": [1, 2], "b": [2, 1]})
# close Rdict
db.close()
# reopen Rdict from disk
db = Rdict(path)
assert db[1.0] == 1
assert db[1] == 1.0
assert db["huge integer"] == 2343546543243564534233536434567543
assert db["good"] == True
assert db["bad"] == False
assert db["bytes"] == b"bytes"
assert db["this is a list"] == [1, 2, 3]
assert db["store a dict"] == {0: 1}
assert np.all(db[b"numpy"] == np.array([1, 2, 3]))
assert np.all(db["a table"] == pd.DataFrame({"a": [1, 2], "b": [2, 1]}))
# iterate through all elements
for k, v in db.items():
print(f"{k} -> {v}")
# batch get:
print(db[["good", "bad", 1.0]])
# [True, False, 1]
# delete Rdict from dict
db.close()
Rdict.destroy(path)
Supported types:
- key:
int, float, bool, str, bytes
- value:
int, float, bool, str, bytes
and anything that supportspickle
.
Rocksdb Options
Since the backend is implemented using rocksdb, most of rocksdb options are supported:
Example of tuning
from rocksdict import Rdict, Options, SliceTransform, PlainTableFactoryOptions
import os
def db_options():
opt = Options()
# create table
opt.create_if_missing(True)
# config to more jobs
opt.set_max_background_jobs(os.cpu_count())
# configure mem-table to a large value (256 MB)
opt.set_write_buffer_size(0x10000000)
opt.set_level_zero_file_num_compaction_trigger(4)
# configure l0 and l1 size, let them have the same size (1 GB)
opt.set_max_bytes_for_level_base(0x40000000)
# 256 MB file size
opt.set_target_file_size_base(0x10000000)
# use a smaller compaction multiplier
opt.set_max_bytes_for_level_multiplier(4.0)
# use 8-byte prefix (2 ^ 64 is far enough for transaction counts)
opt.set_prefix_extractor(SliceTransform.create_max_len_prefix(8))
# set to plain-table for better performance
opt.set_plain_table_factory(PlainTableFactoryOptions())
return opt
db = Rdict(str("./some_path"), db_options())
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for rocksdict-0.2.8-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b382546d0d57a5fcc995296da0c4b9a0b6943c237919c3753d20730dbe675465 |
|
MD5 | d281b2522ea7fe21199fb2b022b129d3 |
|
BLAKE2b-256 | 1295239f90378487533ed03a251176dcb380cc36697174198c3f8fcaf2a59e72 |
Hashes for rocksdict-0.2.8-cp310-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01bb731a1e343be17dca504b19506bf972d86d822ed9f79d10d74cc7cf9c3438 |
|
MD5 | 5bdb5246f55379437290c65e73343cc3 |
|
BLAKE2b-256 | 4b23410273690cb4b7896ea4f6a99c727cf445a88226b4670f509bed62a65727 |
Hashes for rocksdict-0.2.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01e2b2a4d8012bb061aef07fa3cca9699fc94e61dc1c3b3402e10588bddad432 |
|
MD5 | 722e53a7dbf328c63d605d0848bfbe07 |
|
BLAKE2b-256 | 629ad375cf5b7091f38cc381fad87ab15e80b17e165c500d384b20efce6e88a5 |
Hashes for rocksdict-0.2.8-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7906da964c82d3cfef822ac0aeb7c366a85d4cd285acc62b121b2c586b65961 |
|
MD5 | aec77ca38aa91c44c03d8b7ab4b59b2d |
|
BLAKE2b-256 | 96c264b8912bff04ad271a21e54d6a1bb28a3e641af49df74f4cabf693074830 |
Hashes for rocksdict-0.2.8-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65c5b037f0017ff387a3d088e3fa7310db3116e0e564d3f20a29c6a44e2fc269 |
|
MD5 | 5b5213f456ab32f6663dd5e39d967e54 |
|
BLAKE2b-256 | f671544f28c0160232f4dfd43d1692730018356cbe8025501edc41747e25888a |
Hashes for rocksdict-0.2.8-cp39-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | edad2fb5f63e8c5774366a3d189228d33598c58bebfa3985b47ed320be6d63c6 |
|
MD5 | 676be020996c0c3792514fc7c7667a9f |
|
BLAKE2b-256 | 94b82c816a6c73f7e39c59ec35176937e8630c7936f59e0d864241a2d2ed40a1 |
Hashes for rocksdict-0.2.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87e7407f913b261fb75338d67c339b533652bd0912b10327de77887223754d04 |
|
MD5 | b9fa259f13d797586b0eacdc442b0e18 |
|
BLAKE2b-256 | bff3f906d02a062ce86de029d95430052ca0d39294b6b93023a12c0beccad0f4 |
Hashes for rocksdict-0.2.8-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8403bfa7f497a060632b7eed64faadf4d875f13c2dd3245fe4d59ca0b1429a76 |
|
MD5 | cd50822e90b59bd143cb5238f37cf5e4 |
|
BLAKE2b-256 | 85f33889f8ef4f5ec99f6cac315b5298519563c924347d8b879d099e59625573 |
Hashes for rocksdict-0.2.8-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 63193ea34ef0d54d873ae9a668c21321bbedf3c7146e6e432eddbbfd09d254fb |
|
MD5 | 5ac9a5e54e931e9d117e07d8a5d11caf |
|
BLAKE2b-256 | 7e549aaa10e3a932519d44948b7f599904a2a09aa0f6db83c7039d23146836d3 |
Hashes for rocksdict-0.2.8-cp38-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af66a502fbc6c0da9b943cd8cbd3bd8da445d022138330e06e434ef74786f710 |
|
MD5 | 1fc2c8560644a45368fb4a5147afdeb5 |
|
BLAKE2b-256 | ebd8db0e519a0d88571332f2c1fbcee45b6a8938f3071c5822c6000c48bd08d8 |
Hashes for rocksdict-0.2.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04e17874a4184c2911a0c6b4a9c4289c8a0f9cfc2bd11cc2fc4f4d5203698089 |
|
MD5 | cf8e6b43c4419d38d9ca13d85455587b |
|
BLAKE2b-256 | 67ffd4fb64d9218350862091e4439bed06076ca27b26ecde23909055d4fd8c12 |
Hashes for rocksdict-0.2.8-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e655a80114b73115b6858e2e30f52c95b7d762285675a8957985217db6a6774f |
|
MD5 | 180b98d8a75acc6ab26e90fb2f7869e3 |
|
BLAKE2b-256 | 11aa492beaa1d300c9037a92c5af72fd0fc1e48b6f22aee7abf6603db4ae9a3d |
Hashes for rocksdict-0.2.8-cp37-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6eb1a64d7fbc049cdfd05b063737de8c2282e89aa5be402ea018d7a06a252f2e |
|
MD5 | ed364e93865df2e098f6e86c691edca2 |
|
BLAKE2b-256 | b32e13f8182c368a754681d9a05eb393d4a941c61d8a9653e19582a56dd1247e |
Hashes for rocksdict-0.2.8-cp37-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0056bcf2468919e287920614efb859d8f6eac5864cc4d24e94224f1cb5ac596a |
|
MD5 | 216629d01515b78ec5a7d64bd5b5e9b8 |
|
BLAKE2b-256 | 4fe492a253c48b1f6a7ab46e524c74778d49be83830ae93feabcdc9865b311de |
Hashes for rocksdict-0.2.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eafe210e0558c95708f79a36b592dbb6fa61a4e3c0771864bdf9449c215c30c4 |
|
MD5 | 6546eb6f0a3dca7fa75c6bd10f98d8f6 |
|
BLAKE2b-256 | 1717ff29e684f2117395b5e9a40b39b7de69d5d3ff70415682d3ccf26d3fbbf1 |
Hashes for rocksdict-0.2.8-cp37-cp37m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b8f57fc03c237c2939320a5c9c7efa9db738338ba7d7203150392a04b04bb2d |
|
MD5 | 6402a8f34633609f552dde8d300849cd |
|
BLAKE2b-256 | 6337a7e1a1c5272e9ef76b9fa0063ba03bc4bea21f569e4ebcd7e56c80b76f12 |
Hashes for rocksdict-0.2.8-cp36-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ddeab936dedb613222a05e94d8593ad845cebecaa7174114f1fdf69dc4a2d37 |
|
MD5 | b79dd41b3080a86573589455d264e576 |
|
BLAKE2b-256 | c3bbc04cc598e0a26a01ed80311e32d25bbe8a66e4aeab7a93bb5df41e1a64ba |
Hashes for rocksdict-0.2.8-cp36-none-win32.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5cdc82bab417eda6bc925cec0ad0276cc0a5bbfff37d95406a0bf88ff5ae6fcb |
|
MD5 | 01bba8e7afa647d113ae12d2f072a403 |
|
BLAKE2b-256 | 1efa3ce101a5c30b65346bf9f54ff0c5890ac28d657a739100210ae97d515c72 |
Hashes for rocksdict-0.2.8-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14d6a6aa61b74eac249a9ff069916bb72a3e26e6189e6b52819ee6d25f0d912a |
|
MD5 | a80e4a0f66b50bd8fce0aecee4eb688c |
|
BLAKE2b-256 | ef0e70101f4c344670084b91190a0613f6d2c510c86a3439aa49dceb9655f038 |
Hashes for rocksdict-0.2.8-cp36-cp36m-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4e4d2b724a58286268cca041399c389d00052cdd68a79e6cd92f90648be2ad9 |
|
MD5 | 2ef4b642503e8b83c38416199f70a001 |
|
BLAKE2b-256 | 5a2a0fabb9ded108a36287408d2487334af5c8e46cc595dabacd9e0a131c68c7 |