A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=artial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.3.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 111dd393cbf8d062ba661b25536a513f1a59100e489447209db0590c097abf3c |
|
MD5 | 9650e764c94a1283d46e117afa554f0a |
|
BLAKE2b-256 | e1d7f8b48fc1cc18199efc743b1e525ef8321d1a9a028b94430b4cde3de7d1d7 |
Hashes for hugedict-2.3.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43a9a56d4738d53ce458a46d7c23e460c75c79767504c7aa72380ef9ac50176e |
|
MD5 | 111803ed9d1418ffb14fed15b3c66066 |
|
BLAKE2b-256 | aa0b52e4296acd8fd56ef76e57d9c98ea0acbcf76deb223e3899c95717114b45 |
Hashes for hugedict-2.3.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d04276dd70ace45fe53995408399d51da0e928a4591d46d8471022bcd74a75e2 |
|
MD5 | 7c426f8f7ae881441a145f87a2f0bd72 |
|
BLAKE2b-256 | 929c81c2b3524e7accdba777b957b7c7969b1b953ea50d8d06c0fa0c6e6243ad |
Hashes for hugedict-2.3.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 79e151238918a6facc59759d8630f2d4e05736d792777f44b973c3238578e835 |
|
MD5 | e698d09975b81076d95e73282e67cd45 |
|
BLAKE2b-256 | 56d9eb6ddb39127dbdc8b5e54ddf660615a819285627209c3b8c8553b31335bf |
Hashes for hugedict-2.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4bd5dca744e31f41f52353a55d12a13119b7ce11edbd83b772bdeee4a44771d7 |
|
MD5 | 6da8a91612c705f5134f6c7c274de37c |
|
BLAKE2b-256 | 77d52dc5e8394373eef19717de14708554e00bc980b526bd6828f467c7c02758 |
Hashes for hugedict-2.3.0-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 75bf88af4f57c56e6105838ad46a8a1f3c44aa5ab1cff06abfa34f71862ff746 |
|
MD5 | 7dbe1a094db0203d8e809d59c139ee66 |
|
BLAKE2b-256 | f16bf26cf609c22822e318406a7d83f1afa2b523752a48b444f192fc89d7ad9b |
Hashes for hugedict-2.3.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f50f8588f28d83c7ee3158eeac90bd7e6441821b2c72c94db55eff122c4950d |
|
MD5 | a803d3b8f05abb9103e0bd3b8473643b |
|
BLAKE2b-256 | 42ac329ece41a2326f35c9006ee5b3e1e2f905ac7e45d680434c9b080fc5fa07 |
Hashes for hugedict-2.3.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 261fa2d7d5a44ff77092553f6caed6a864d43a94fa5c1b9690f33b0e2d017554 |
|
MD5 | 2f93da88b937dc65530c15b8fe9271d1 |
|
BLAKE2b-256 | 046ddc11a15bbdb0796fec5eb088f001cc03fe278bf041b4db7de59adf0c45c7 |
Hashes for hugedict-2.3.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 090f9f369b295a2215b5f668765198ca1709b5c273885c156250eff609756ee1 |
|
MD5 | a66943cba0eab936b876522cbb112c84 |
|
BLAKE2b-256 | d9caf90e1b4421f7cc2665097e5ded42f12a34fa87691b0c2b209e1fbebb6ab3 |
Hashes for hugedict-2.3.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | db38f6784df96fddf6b05e20bf4804c4c92c170ac7c2f2011dd6d9432985a2cb |
|
MD5 | 0437a060d51becb6e4e3b8e68ad2e9f6 |
|
BLAKE2b-256 | c291f42c80acdf5a06f9db45c0052c1615fea496f0fbdbc6690a1596aae63930 |
Hashes for hugedict-2.3.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | beffb520e0d6eecf6d607e29418a6f638b0665e7dc30f62b661991a4edc3c28d |
|
MD5 | c447157fa61fd8ff1ebf8d57777fd404 |
|
BLAKE2b-256 | b686f12011a341e747abb842f0953de9480b2ae8fab22d87a28d4ed4c65f857f |
Hashes for hugedict-2.3.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a7122703656b74fd0b23a353b68a13bd6104d2439c73f5d768c32ff33adb6a7 |
|
MD5 | bc265e604a4f685b2005a9c04b663eae |
|
BLAKE2b-256 | d696532e6213548d0f9d6448dbc991d5ccd58825c8f3abf62eba23d04e20730f |
Hashes for hugedict-2.3.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28a5df597426db781a168cca55a39a88cb2564c4a72f090a31b88db86f3d0d77 |
|
MD5 | 684528ea1a3167952cce8d09cf110398 |
|
BLAKE2b-256 | 7ebd12c9573de2ed39fb01698b908230855a265c7694213fe5542ddbb533d15c |
Hashes for hugedict-2.3.0-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e2ff2e01bfce521ab82d1e5afd76ac6bc3221ff8024eb368598c8fbffa0fb77f |
|
MD5 | 79588c307b65fbdc6eba7df45741cfc1 |
|
BLAKE2b-256 | 79b2386e340cd6a49f80a31bae7f2bf5acb2f8833a2cf7e6d284bf6cc516aca0 |
Hashes for hugedict-2.3.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd0f58086c2a88af39b31f0e11e6dc4d3020f61cc6ff491561c5d14d50a5c34d |
|
MD5 | 8739b2d421a9609b7f4e04385847531e |
|
BLAKE2b-256 | a6b66109a59068913029c29c03e5248011911594f33ebfe7646fd770f7f5b438 |