A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=artial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.2.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbf9aa5d5305263baf7b94d528d9d18dd7e77d6dcdfa8e0188ae8f0feed14f4a |
|
MD5 | a5b5db38f71ede84f6f2ac7dbb607023 |
|
BLAKE2b-256 | 740f946b0cd53837224b925acb8ecf270c9b3b4c0afffadf1fd1d36de17b858c |
Hashes for hugedict-2.2.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec2c613f3e395c8fc62f18ad2d3f58eeb29367edd257486bc9a69ea04913b039 |
|
MD5 | 36327c02eb9bad725437f8ed15376a4b |
|
BLAKE2b-256 | 3635f24c46c9436508e24ab94a4706bf929ce27444d61bcc870abf7404decb1f |
Hashes for hugedict-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9edb95b60573a11a7cea7ab9a612c1c61ef4b89055fcd8d8a4cd2fe30f548231 |
|
MD5 | 2b2745c3e8f23120194c43ed9cf3ba86 |
|
BLAKE2b-256 | f2370ccf4e42d51cc5c9c22b9c771d91ef43b2a83bbae8a53766d950faea2cb5 |
Hashes for hugedict-2.2.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04bfbbc4370d25de2bd61e28e798c686bfaa22a06cf5b28f39f6455004f523c5 |
|
MD5 | 7fd3181c4c3650be33fd96538b35e4cc |
|
BLAKE2b-256 | d003eed3d6dc7d5ed96b0dd4f8da75aff3e31daf77bdbd7b1464d6351490e77f |
Hashes for hugedict-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6b4d3941c56392c0e0b7881f51a4a9af9091cef9a65e361d10765fd31a894f9 |
|
MD5 | b883660b6e0a161775016b51eb25bbe0 |
|
BLAKE2b-256 | 8fd904797e1dd71fa940f0d84416ebc881f77a42fad7012881bfe5753553a3c1 |
Hashes for hugedict-2.2.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e3f94157288b25a339e503717fd981b2e526a61b9ae9d43924fcbaec7b6a193 |
|
MD5 | 7c27ce8c9476bc7ed696e1f060ed985b |
|
BLAKE2b-256 | 6c6a68f29afb66f39ba725db0a22ed0132d8a6285798d743d59c403c5ec32542 |
Hashes for hugedict-2.2.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc7a50c3661588047cff8e6fb88bb473d71732d4072ee2b7ab21bb7b1a3fecc7 |
|
MD5 | 787069451f3e7fc420c7be72a0c6a658 |
|
BLAKE2b-256 | b2e839caa144352f167091688f14ff284aaa6a39ac6a067e364e32a18c80558a |
Hashes for hugedict-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f182c5c40bf89e1f837f3f8e0765daecf0f12bef70fbb961d05166bba47c0ed |
|
MD5 | 7f02fde394ad933dc01679ecd93f5d0a |
|
BLAKE2b-256 | e240a3a4cdd2acc27fbfa675f2966f43d8d9b1a4b5c4f2557949d91bab47d6a0 |
Hashes for hugedict-2.2.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06fec8f505f4ac07d24ce80dcfd9aa2d9646bf4b331591238f9919ab9e88450e |
|
MD5 | dcfe0f6ab9abeab8f13fc30531f0c30a |
|
BLAKE2b-256 | 2c33623ff7e64d41cbec39906a00163aaf038a08986f96dcb1e6e1cd2db3c2df |
Hashes for hugedict-2.2.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07638135990f05be0a526f985ca6f4183cc8b399658f51814c432fdfb7c2e454 |
|
MD5 | 2f375e02993e9824c85d17b0ae4e94bc |
|
BLAKE2b-256 | 2cb4105d7c118bed10042b1a1e3bbd1851943f3979bb877bc1b061b8db635058 |
Hashes for hugedict-2.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76571a62e746e1af6514b088ea5be0b413af906f47168caf6a39ea3820ab86ed |
|
MD5 | 33bc3a7ee5950a51f77d0dc106de4f35 |
|
BLAKE2b-256 | 1c92216a7bc6f18111252ebdbb2d341ed2325794724abc5a11bbdf3230c46fd5 |
Hashes for hugedict-2.2.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb1feea0b9dfaad4686c82dc04b91a3b465650480800c56b768abc5360166e8f |
|
MD5 | 25b5732eace6aadf17b712fd3519309c |
|
BLAKE2b-256 | 4ed0e24bd93ffcc78aa341f6f28c41e40c10b23fe477361b8aec7008a01085b1 |