A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.5.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5c81982a4d49ff9c4b1c0d681aac0aec89e87739bd96f51307ece39be9ab4d8 |
|
MD5 | 5ab493f974ca3e1fbff071021a934203 |
|
BLAKE2b-256 | 9d9f689f357b9c438ba912776d7f018588d9948e2a602bc9c345986c8d84892b |
Hashes for hugedict-2.5.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 41056a68413ab3880ac2dcb2099267db566ac44b241afd12f3a0235f6bf84277 |
|
MD5 | b21b4b23b2ab39ef5ae620b5ce932077 |
|
BLAKE2b-256 | 163fae1e5541b419e0bceb40e229a7c4e3e0b9c782cb50e53a0f4f56ca250092 |
Hashes for hugedict-2.5.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f6e5d27fd601a83997245a9af311dffb192844a73d0047b662ecc43d21fe2645 |
|
MD5 | 2a66db041fc5f32802212fdb0534eb94 |
|
BLAKE2b-256 | 039c7f17c7350404cf3557f8f5ddda213a26f430aee9cf3dd20b031e4e6d75e5 |
Hashes for hugedict-2.5.3-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d7a3e047f745f5a4c8678a4b1022c517b92ac27f437e2bedc6cdf2a644107ead |
|
MD5 | 646edc31443cf2fd4de5c183684d9f0a |
|
BLAKE2b-256 | 8648ea2d9f2810b5401864149644920d1821f5d13c15754d7d6d774081392375 |
Hashes for hugedict-2.5.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c2111d33962d3ea72f0e814bfd488a8d2a612bfbee63044129c67a2d06ba3c7 |
|
MD5 | 9d1fb907c518a81325909dcc7a701514 |
|
BLAKE2b-256 | 1fc1d26ab487368820f5305a56c4e8394c797ecfe2b00fe68a55ce142e43f1d1 |
Hashes for hugedict-2.5.3-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d8ea6d5e1776cc941d311659b3eee7f5aa77b0896db77cc3442591ff5a10a64 |
|
MD5 | 79005bd6eb4e3d050dd5b6f457350333 |
|
BLAKE2b-256 | 7ef826a09310a6f0985d42637218006c51998a289c254e3a872a788a148127ee |
Hashes for hugedict-2.5.3-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07d6fb40f3cdb6d80c4ba7a86f79ba1a672c581855c77e7056620ab358623bbd |
|
MD5 | 3bc93962e7431aafccbbd8bc0d145c76 |
|
BLAKE2b-256 | 1428e518881ac9825b0a29a23abff84cc32bda6023e9afbb38e3dd4e73a39e93 |
Hashes for hugedict-2.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a423d3a785996344158e5bd216ed84cc52648d90fc2fb85ea8ba5b168fbb962b |
|
MD5 | 3d275065d59267178f1a35cbede5d383 |
|
BLAKE2b-256 | 1e429009deb6b8ab905172645a40fa64d4b4cd87afa962a7744193781d637c10 |
Hashes for hugedict-2.5.3-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1f97c55fd5b74d892088ec063c6ea39c5d07c2d2df0f61120ddfff3494ed3a06 |
|
MD5 | 3d8c8e8a9911f041fa351f172ea60bb6 |
|
BLAKE2b-256 | 35be20191015e340cf772dd1dc20db13d4533957a6ebce433c3fc29e6cb1950b |
Hashes for hugedict-2.5.3-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf3951d2cb378f03f39f4aa380b8ec9e1044fbb771cb23e329f0a96647938089 |
|
MD5 | b168a5f1309c0084cbaa7a52aff0af7b |
|
BLAKE2b-256 | 117a7d58020d75f8129b794c4282786f88fb5308c4489a01ce061e868dd2b6b7 |
Hashes for hugedict-2.5.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca5a2532880a497de5f9a1b48b491daa055ba51d885ba58e1ea220b5b2daba46 |
|
MD5 | 1d77641ee9448d9a12c54ad151dae6df |
|
BLAKE2b-256 | a51c3242e002ba69887f1e0e55b1e28935518eeb430a01fb924589fcb3570510 |
Hashes for hugedict-2.5.3-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6cf976ce75d40c548baf661084ac33b0c58fd7f3cdfb65de6c728360cca92006 |
|
MD5 | ab1f4be8aefc3d1ec658ec91bea293ae |
|
BLAKE2b-256 | 9aa8ab922d7d83bb4f0bfe08e6b009e49c859d3ab8aacd7bdd9cf3475a27ef2b |