Skip to main content

Rocksdb Python Binding

Project description

RocksDict

Key-value storage supporting any python object

CI PyPI PyPI Support python versions

Introduction

This library has two purposes.

  1. As an on-disk key-value storage solution for Python.
  2. As a RocksDB interface.

These two purposes operate in different modes:

  • Default mode, which allows storing int, float, bool, str, bytes, and other python objects (with Pickle).

  • Raw mode (options=Options(raw_mode=True)), which allows storing only bytes.

Installation

Wheels available, just pip install rocksdict.

Examples

A minimal example

from rocksdict import Rdict
import numpy as np
import pandas as pd

path = str("./test_dict")

# create a Rdict with default options at `path`
db = Rdict(path)
db[1.0] = 1
db["huge integer"] = 2343546543243564534233536434567543
db["good"] = True
db["bytes"] = b"bytes"
db["this is a list"] = [1, 2, 3]
db["store a dict"] = {0: 1}
db[b"numpy"] = np.array([1, 2, 3])
db["a table"] = pd.DataFrame({"a": [1, 2], "b": [2, 1]})

# reopen Rdict from disk
db.close()
db = Rdict(path)
assert db[1.0] == 1
assert db["huge integer"] == 2343546543243564534233536434567543
assert db["good"] == True
assert db["bytes"] == b"bytes"
assert db["this is a list"] == [1, 2, 3]
assert db["store a dict"] == {0: 1}
assert np.all(db[b"numpy"] == np.array([1, 2, 3]))
assert np.all(db["a table"] == pd.DataFrame({"a": [1, 2], "b": [2, 1]}))

# iterate through all elements
for k, v in db.items():
    print(f"{k} -> {v}")

# batch get:
print(db[["good", "bad", 1.0]])
# [True, False, 1]
 
# delete Rdict from dict
db.close()
Rdict.destroy(path)

An Example of Raw Mode

This mode allows only bytes as keys and values.

from rocksdict import Rdict, Options

PATH_TO_ROCKSDB = str("path")

# open raw_mode, which allows only bytes
db = Rdict(path=PATH_TO_ROCKSDB, options=Options(raw_mode=True))

db[b'a'] = b'a'
db[b'b'] = b'b'
db[b'c'] = b'c'
db[b'd'] = b'd'

for k, v in db.items():
    print(f"{k} -> {v}")

# close and delete
db.close()
Rdict.destroy(PATH_TO_ROCKSDB)

New Feature Since v0.3.3

Loading Options from RocksDict Path.

Load Options and add A New ColumnFamily

from rocksdict import Options, Rdict
path = str("./rocksdict_path")

opts, cols = Options.load_latest(path)
opts.create_missing_column_families(True)
cols["bytes"] = Options()
self.test_dict = Rdict(path, options=opts, column_families=cols)

Reopening RocksDB Reads DB Options Automatically

import shutil

from rocksdict import Rdict, Options, SliceTransform, PlainTableFactoryOptions
import os

def db_options():
    opt = Options()
    # create table
    opt.create_if_missing(True)
    # config to more jobs
    opt.set_max_background_jobs(os.cpu_count())
    # configure mem-table to a large value (256 MB)
    opt.set_write_buffer_size(0x10000000)
    opt.set_level_zero_file_num_compaction_trigger(4)
    # configure l0 and l1 size, let them have the same size (1 GB)
    opt.set_max_bytes_for_level_base(0x40000000)
    # 256 MB file size
    opt.set_target_file_size_base(0x10000000)
    # use a smaller compaction multiplier
    opt.set_max_bytes_for_level_multiplier(4.0)
    # use 8-byte prefix (2 ^ 64 is far enough for transaction counts)
    opt.set_prefix_extractor(SliceTransform.create_max_len_prefix(8))
    # set to plain-table
    opt.set_plain_table_factory(PlainTableFactoryOptions())
    return opt


# create DB
db = Rdict("./some_path", db_options())
db[0] = 1
db.close()

# automatic reloading all options on reopening
db = Rdict("./some_path")
assert db[0] == 1

# destroy
db.close()
Rdict.destroy("./some_path")

More Examples on BatchWrite, SstFileWrite, Snapshot, RocksDB Options, and etc.

Go to example folder.

A Simple Benchmark

Compared to PyVidarDB and semidbm, which are all cross-platform key-value storage solutions.

Small Value: 100 byte value

sequetial insertion

image

sequential read

image

random read

image

Large Value: 100 kb value

sequetial insertion

image

sequential read

image

random read

image

Limitations

Currently, do not support merge operation and custom comparator.

Full Documentation

See rocksdict documentation.

Contribution

This project is still in an early stage of development. People are welcome to add tests, benchmarks and new features.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

rocksdict-0.3.8-cp311-none-win_amd64.whl (2.7 MB view hashes)

Uploaded CPython 3.11 Windows x86-64

rocksdict-0.3.8-cp311-none-win32.whl (2.4 MB view hashes)

Uploaded CPython 3.11 Windows x86

rocksdict-0.3.8-cp311-cp311-manylinux_2_28_x86_64.whl (4.9 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.28+ x86-64

rocksdict-0.3.8-cp311-cp311-manylinux_2_28_aarch64.whl (4.8 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.28+ ARM64

rocksdict-0.3.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.0 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

rocksdict-0.3.8-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl (6.8 MB view hashes)

Uploaded CPython 3.11 macOS 10.14+ universal2 (ARM64, x86-64) macOS 10.14+ x86-64 macOS 11.0+ ARM64

rocksdict-0.3.8-cp310-none-win_amd64.whl (2.7 MB view hashes)

Uploaded CPython 3.10 Windows x86-64

rocksdict-0.3.8-cp310-none-win32.whl (2.4 MB view hashes)

Uploaded CPython 3.10 Windows x86

rocksdict-0.3.8-cp310-cp310-manylinux_2_28_x86_64.whl (4.9 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.28+ x86-64

rocksdict-0.3.8-cp310-cp310-manylinux_2_28_aarch64.whl (4.8 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.28+ ARM64

rocksdict-0.3.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.0 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

rocksdict-0.3.8-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl (6.8 MB view hashes)

Uploaded CPython 3.10 macOS 10.14+ universal2 (ARM64, x86-64) macOS 10.14+ x86-64 macOS 11.0+ ARM64

rocksdict-0.3.8-cp39-none-win_amd64.whl (2.7 MB view hashes)

Uploaded CPython 3.9 Windows x86-64

rocksdict-0.3.8-cp39-none-win32.whl (2.4 MB view hashes)

Uploaded CPython 3.9 Windows x86

rocksdict-0.3.8-cp39-cp39-manylinux_2_28_x86_64.whl (4.9 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.28+ x86-64

rocksdict-0.3.8-cp39-cp39-manylinux_2_28_aarch64.whl (4.8 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.28+ ARM64

rocksdict-0.3.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.0 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

rocksdict-0.3.8-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl (6.8 MB view hashes)

Uploaded CPython 3.9 macOS 10.14+ universal2 (ARM64, x86-64) macOS 10.14+ x86-64 macOS 11.0+ ARM64

rocksdict-0.3.8-cp38-none-win_amd64.whl (2.7 MB view hashes)

Uploaded CPython 3.8 Windows x86-64

rocksdict-0.3.8-cp38-none-win32.whl (2.4 MB view hashes)

Uploaded CPython 3.8 Windows x86

rocksdict-0.3.8-cp38-cp38-manylinux_2_28_x86_64.whl (4.9 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.28+ x86-64

rocksdict-0.3.8-cp38-cp38-manylinux_2_28_aarch64.whl (4.8 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.28+ ARM64

rocksdict-0.3.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.0 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

rocksdict-0.3.8-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl (6.8 MB view hashes)

Uploaded CPython 3.8 macOS 10.14+ universal2 (ARM64, x86-64) macOS 10.14+ x86-64 macOS 11.0+ ARM64

rocksdict-0.3.8-cp37-none-win_amd64.whl (2.7 MB view hashes)

Uploaded CPython 3.7 Windows x86-64

rocksdict-0.3.8-cp37-none-win32.whl (2.4 MB view hashes)

Uploaded CPython 3.7 Windows x86

rocksdict-0.3.8-cp37-cp37m-manylinux_2_28_x86_64.whl (4.9 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.28+ x86-64

rocksdict-0.3.8-cp37-cp37m-manylinux_2_28_aarch64.whl (4.8 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.28+ ARM64

rocksdict-0.3.8-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.0 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

rocksdict-0.3.8-cp37-cp37m-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl (6.8 MB view hashes)

Uploaded CPython 3.7m macOS 10.14+ universal2 (ARM64, x86-64) macOS 10.14+ x86-64 macOS 11.0+ ARM64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page