A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=artial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.3.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52114cf1ce6518c1961f8dc35e09040cd9698bb3fd15dd3ddbdefa894a2bf5b9 |
|
MD5 | e0d455ec37efb68daec254e7ba90365b |
|
BLAKE2b-256 | 5806a1788768fb5bf0367763ece8c83d4dce1f3498ac35ee91554c45a100133d |
Hashes for hugedict-2.3.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3527888bc88b50ea6edb32ec9a1a1660d34ecd492cb5cf920e4641c2c92bca7 |
|
MD5 | 030ae672c6b7b36e2dfeedfdcef01750 |
|
BLAKE2b-256 | 111526881b71ac633801704df20522aec81f6b6768e5b886cb9c6c4a00c2b55f |
Hashes for hugedict-2.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c7aadaa503eeba8d2bc88578b3314189c3e7d60437f80a8f4349dc24b159e49 |
|
MD5 | 7582b450f10e63ab799e457130860b7e |
|
BLAKE2b-256 | 3287e2e8937379a6281cc1c010e529e76759497b5375df6e94a29d7617719563 |
Hashes for hugedict-2.3.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93cd6d9491469a09a6a435b0eaa51ee0adbc7f012b03925536101025afd2e7f1 |
|
MD5 | fe168de0d8a88de2d1b2da83535e2404 |
|
BLAKE2b-256 | 30fa972bd4f9ef6df4c0ada12cb535aac706b773ce9bd4f81c89ea7e95fdcf1a |
Hashes for hugedict-2.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f0421f671361b6f6344ae4995c2a56053a67542ed8f1eefd5c17f2fa0b2603c |
|
MD5 | 91a5337e861bebe2195c2d83f0799a60 |
|
BLAKE2b-256 | b56b19a26fd7bc4f1740b3fffd60d9c8b97f6cfbf02954978955d7877e10371d |
Hashes for hugedict-2.3.2-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e1f16beff0fbe48e51026cab843b1957e2f74e4513461e288564b23c3e29163 |
|
MD5 | 2efcf1ea41e00f9c0b0025b27548d199 |
|
BLAKE2b-256 | 4c95dadda57205a23db50322db30f6adb36d62dfdefbd5453786ce711a520ff2 |
Hashes for hugedict-2.3.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20fdff0d32c0c5bb282f5fdb04f2f8d6222f10147cec37785c3b0941344c21e4 |
|
MD5 | 2596b5ed823258b59b2206945f09524e |
|
BLAKE2b-256 | b12b5904530fd059de1bf82bdd58c5e8b30af6d38c82c0ed936efe559935129c |
Hashes for hugedict-2.3.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a59338dcdc6d9453ae6ab61b2b1c81fafa4cd0e99d853b936a5529596bf8ab86 |
|
MD5 | 63f4f5a1a847660ea96c4bac1246dcd2 |
|
BLAKE2b-256 | c88b1fc055cd1fa423d739af1ef21018ce58a4257d8454c6a7a9e5b3af462dd2 |
Hashes for hugedict-2.3.2-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3340f7e060a6c81642f3eb720ec20423875087a5ac738f0650118c2eac43de7d |
|
MD5 | 41a697e0a81bfc12a789e1ebc39b4378 |
|
BLAKE2b-256 | 12133b9c237a0b5c716449990e566b40e58109e2e2230259c1803ebad1da2dbe |
Hashes for hugedict-2.3.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f5e43fc2f14de787ff66661e6dee23716f5b48e0efa2cf950a34c592ae451b7 |
|
MD5 | 82169f565a2c13aff6aefc8c4a6a9a1f |
|
BLAKE2b-256 | 8203ef6d8565fe07cf393692ccc524292857685d211d5ff8c25e189b1f3175d8 |
Hashes for hugedict-2.3.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df79bfda6f3af6db81131054bd600a3c759d0d9cbee36286c6c9309b3f85706e |
|
MD5 | f752ec0d5c97ea7a6869fa5ac48e0ece |
|
BLAKE2b-256 | 3ac081e1ceeabeba4df7b201b53a243f411c163e9bc34c468e821dfed28c5d56 |
Hashes for hugedict-2.3.2-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aae9234a6466ecf3fd78783eb12000d842d942e712a848e0a0cda58f25bc5324 |
|
MD5 | 8252cdf2e27d7f0aabab26c1cd6df92e |
|
BLAKE2b-256 | d72f7e3453da1232b459f3ec28ae3fcfd9862cdb1d5c20036a0c18e8fc98ea13 |