A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.5-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a05e3bdb21081a4b9256bbe9c89f1bad165b5d93a8691c5e13c4b5cfaed65f39 |
|
MD5 | c330f93c806a8a8dc9fdd8fa80d29ebf |
|
BLAKE2b-256 | 187273569f1df68d321b0a99f7963be95d2192e3adcdcf2b5abd30e7e174b1e0 |
Hashes for hugedict-2.12.5-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed42c4bd27a71989f7ff7fd8f20dcf07b61784b662170026a49067e82053398b |
|
MD5 | 523a2bf79f5c225218d4e68c11a38775 |
|
BLAKE2b-256 | 1a6220664a59e8e974432dfa5dd2c523ac699068679f8c84f23627c6917d0df7 |
Hashes for hugedict-2.12.5-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ad286888a6eccedc5e2a4b215e99575d3df52ca27135c641b94ed1242a6fc190 |
|
MD5 | 9e52d177a2c24d75fb5b4c846cbaf305 |
|
BLAKE2b-256 | ae8e7f8c9924e187b1c6b8ace812cd605b24526be6952a11f6f055f2e74da502 |
Hashes for hugedict-2.12.5-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | acb99698d576f80af4bc9d2514f6198d1a89dc3d80abd3c1cc6de2b37da73607 |
|
MD5 | 20604e6ef79284a3e51dd2ed2d6752ff |
|
BLAKE2b-256 | dc366218e1944582a57e55bb2125c4648995e9958b74423203fb4068326be46f |
Hashes for hugedict-2.12.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ca47f08f8d2ada908191f7685c9f77e392d158264708f1ada5a604145451bc4 |
|
MD5 | 078427d0bcd75805e555ab2cec33db11 |
|
BLAKE2b-256 | 9cbd97318e3bf9ed87afa46b147abe27fcb03c90d473bacba288efad39e61f13 |
Hashes for hugedict-2.12.5-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea36b4c725b78be6421adb4bd86e7b67aba53f5342b554fa6583d93e4422a9c4 |
|
MD5 | ef1c33458d1a4cc84623a24256b8e4d0 |
|
BLAKE2b-256 | e2880ce9fb0ce8525b3023f0cff5716e1d30b398fabd55ed306628a07cde8f23 |
Hashes for hugedict-2.12.5-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4fcb29d7c6865f566c3c94cba37cd25a5fed5e3578a9c581e6afddc07d423eb |
|
MD5 | b40cd1d863546e6af7e6ba1af53afbb1 |
|
BLAKE2b-256 | b3b19662613a27d8ba01829b21e2737ade8401dfd6d5b0c6a12aaa91782e97ff |
Hashes for hugedict-2.12.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfc8ab215b2aefe0c09de8db22bf6e132cad90b8607e14f9679e3525e2faf034 |
|
MD5 | 7407ced3e948fd9cd394507c00a05b45 |
|
BLAKE2b-256 | 96627c6e1c6726948f0fb4dc55b295d85eca4d681bff4a4cbe1fe1c5c85ae02c |
Hashes for hugedict-2.12.5-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5bdc64185365625d46e716c4e1f33e52c3174dee66fa200f17677a1c1462a8f7 |
|
MD5 | eb353a6a2a9ee0e76aee7d59cc62803a |
|
BLAKE2b-256 | 6fc43780dbb84e1580625e45e144ac5c35e39b57d3cad2e5f094235d76918f00 |
Hashes for hugedict-2.12.5-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22818c59fb319a28791c819d3e5a597b5bcddcde3c7542be7a92ff0068c9ce6d |
|
MD5 | 752bffadeca601a7a4d26587b3054a40 |
|
BLAKE2b-256 | 613f1c5137249fafdfe7b4c4a6045b03d74308ed4a83ae4d5e8101e41d9a0c06 |
Hashes for hugedict-2.12.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09d9f4b506cd45157eb951128ab115ab32c80b596a8ddfe7d78e3c92fe4b6b71 |
|
MD5 | d650a8e7165889a03252da783fe9fb79 |
|
BLAKE2b-256 | c4de2b3b64593bd610d1653784ab917b58927117a801f3c89c0a852814f359e6 |
Hashes for hugedict-2.12.5-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc83b759c2309ddcd5f20d5db2348e8aaff77ffbba30979e4013d8f03defd653 |
|
MD5 | 4352464636e1727b56668837266f8a78 |
|
BLAKE2b-256 | 3429b2f5c1687241b491a34299e08d8465d530ecb5d63f2966511d3ed776a6a1 |
Hashes for hugedict-2.12.5-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8768954663dc1ccd02feebae8f12c837392ba648078a17f06a725ca2059c0c78 |
|
MD5 | 1d647975f1039c7a082cfb5f0055fe03 |
|
BLAKE2b-256 | 8c6ae265d6e3053f193532ad953b9869a271392084597429653ef3223db4d500 |
Hashes for hugedict-2.12.5-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b8bad5e34a51ea26d62f007bd5a2d16feafafbb880c755e06760c78b253999c7 |
|
MD5 | 8d808573415e89085c07cb5a91a58690 |
|
BLAKE2b-256 | b04c7461c65794520e09e452d348331a7d07559c775e57e8c2911168533aeacd |
Hashes for hugedict-2.12.5-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ae65af9435a5fe00cf01f66f65089966f2c2cb3b6d185ece75d5e325c9fb42f |
|
MD5 | 03178b3b0c9f5ce97dcb06998b0697b7 |
|
BLAKE2b-256 | 334569bd7f558f91dba3d62a5925efae79d54d559ea5d0877f315420d4f6b859 |
Hashes for hugedict-2.12.5-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb16ee6f4ac8db9503e03c8d1fb846f686fd1454f4192f8d900d5e0531ba2ac9 |
|
MD5 | 8491e97fa080ddc583eff19b8fdd1eb6 |
|
BLAKE2b-256 | 65956b760456a801185c19ebee19d6c605ad97e131557dc586f9261c7730f70e |
Hashes for hugedict-2.12.5-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e80413e69bf99f5cc65eb7fd509bb4f39978bfe695f0d407482054c439409737 |
|
MD5 | 90b5f62a31a737a9c9e4220bcca708dc |
|
BLAKE2b-256 | 80bd6094380cdb462da54f99ebbc03c37350d851e0e37eeb769ef939d91a9633 |
Hashes for hugedict-2.12.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8286f322201adfc53360ad9504b90144fd9efa1f039c15d395919fd7f28384e8 |
|
MD5 | cd1b3c0a75f24c55ffa373f9719e3a6a |
|
BLAKE2b-256 | f0c87365f4843d95c3475beeeb626301752a6f55fe85d41374e063ec1a179825 |
Hashes for hugedict-2.12.5-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b95641b99b682a410c94b1fa334b6e19ad3d707908b8f642603651af977a4823 |
|
MD5 | 2c38a5d09ae9fb32f7b35f9c1e5e3e79 |
|
BLAKE2b-256 | 1279f4a31d779e6324e7644eefb4917fea749998b860d39b4d7a449eb9b110af |