A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.8-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f8058c4629989bb155104cc378d1e0fc32bc4e7c6ac805d6d473072fdc4466a |
|
MD5 | 7d7117699d4715746bce0f54cf7112f3 |
|
BLAKE2b-256 | 7ba46b0d25eaac2b321281b92873eaddab7242aa921d5a4c5745203d9111e1a6 |
Hashes for hugedict-2.12.8-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af49c9850de990342b8b416fb04f5ae3fc1d53156ad4732707e34fe350dbc049 |
|
MD5 | f29666390675489eb0bfb53ac06baf49 |
|
BLAKE2b-256 | 7fe32c2a3beed5135033057a28bd5de62aa7a96cbb725fc86adec7af26638586 |
Hashes for hugedict-2.12.8-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3c9a26b37820cd2d879efee9bd08d7413106f6416b5e317936615afe9ef76b0 |
|
MD5 | cdf77dccfcfab9e08a8f8858cfa79971 |
|
BLAKE2b-256 | f5dbb0afb375c3bffdc17264000a22d255ac28c3d4ef58cac2ffe37178deb4d4 |
Hashes for hugedict-2.12.8-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1e2d24f4f21cf649ded759913c0e872d6fab60fb4ecf8b4a21f57d0edc6dc823 |
|
MD5 | dfba146df28aecc70e6ddc35e3e81449 |
|
BLAKE2b-256 | 5d8e30161f8c98225b00c4333af072da664f1dd09cd831d6ee7ecaf7942898e0 |
Hashes for hugedict-2.12.8-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1240549a559c5f5eb7b4a718761bbaa6aa7aab1c0375f84ee31533f01b50e58 |
|
MD5 | 78195516575713e2c27bb3bfc7396dc5 |
|
BLAKE2b-256 | 852b6d3c6bd2e5e2a73f2651f9592784abd4680e8f2a5c404e42496467b2812a |
Hashes for hugedict-2.12.8-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dba3cc21efd8855520cfc39f72ceb56270a609e20e1e137d0b02a3b494c57ba8 |
|
MD5 | 3f2cdafb689543d88427e60d40d00ecc |
|
BLAKE2b-256 | cbb3e80aba2e1828a9f4383419117193840332cf16aec67454e7e6649448c377 |
Hashes for hugedict-2.12.8-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85d722deb968b05b2278285951e937b91c1eecb767f64cfc2a0331db8a50eaa5 |
|
MD5 | 560efb51e48f639471a3adbccebcb84a |
|
BLAKE2b-256 | b293aaa2cf5c6ef2a7d44203a7004b8a3e6ba404de7f2a12a2fa272d2f527713 |
Hashes for hugedict-2.12.8-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | edbb43add2d493a5c65d14eb35bd35ac557562bf0a8c08062b72af15cc471785 |
|
MD5 | af89324fde6cfef76d1f3ac662785312 |
|
BLAKE2b-256 | 96d0f65723ebee4873cf3c1452c46c59a166680f92071cc51d9db84223f4f16d |
Hashes for hugedict-2.12.8-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4fc9d7f620c9c05988f521eefc7eec319c3c910b80582b66e825a061db127e3 |
|
MD5 | fb10bd8b2dbdf269de003b485ce8654f |
|
BLAKE2b-256 | 012f457452a1b1695c44d6b2ed8e36df4b2fd57ed06bf263162f682009e3f1f3 |
Hashes for hugedict-2.12.8-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8d8fd32c093514cb4c57bfbb4c4d5201b8f8501eb6dcd5e5ae445c3861d46a9 |
|
MD5 | 5e85556805ae3a2aedee02bb45dae76b |
|
BLAKE2b-256 | 437d4153c6a652c2b591ef630ce77fc4f73f42a20e6bf9abd00e9e09ed08aca3 |
Hashes for hugedict-2.12.8-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 53aaa52aff67847c999981f8726d08b16d6dbc03c059159844bf8a054c1b017f |
|
MD5 | 57f330c8d73b3e703e0c09f6634cdf42 |
|
BLAKE2b-256 | 087a866dba5903239ba7f26aac6fdff0185c4635d0d1df9c4323d97b08cdb59a |
Hashes for hugedict-2.12.8-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b4b306c87e996f3c721f9dfad1d524138a467986d2e8ecdf674821cdcc2f550a |
|
MD5 | 798cce58e5aa6b407f42e4ac3685a74f |
|
BLAKE2b-256 | e7d82bf498828ea62246debf50108b1faba01ca68ce835a9fcb8d7d253a6d34d |
Hashes for hugedict-2.12.8-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a37adab35a0c77706f7352506259cc72214b25e876076531556abd301bd013b |
|
MD5 | 74b050697fc797a343800a7f36835264 |
|
BLAKE2b-256 | 6b262bc1b8725113b1a2b67e4078086f84ebab56dd8d6cef7791f029d30dd235 |
Hashes for hugedict-2.12.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c1142aab96793ec8caa9bbba69ae0f9cd21e7fd00d39e3e6a9a8c5028387ca7 |
|
MD5 | 709cf2d8856988254bfe74c4748d01bf |
|
BLAKE2b-256 | c97ff50077e2208261bb3de97a226129494bc3fb29839e1331e4a44234d8bd4b |
Hashes for hugedict-2.12.8-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3af3ab6143a0c30c7c5a1327a6316eb2e4c38a8c3ec0cad8bdda234875c03774 |
|
MD5 | 8ef8a78705cb4227ecbdc6e011065d53 |
|
BLAKE2b-256 | 2ddc816a594d41662932a861bea1b2bb092bb63b85d4acf9c5e06d80bb2e5ab9 |
Hashes for hugedict-2.12.8-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1d8f80bef3d3efb80f45f4531a59c3529377974b3e9e78f12bf2e20ae22854ef |
|
MD5 | 974e82e0f348110a86d3e701f4c237f1 |
|
BLAKE2b-256 | 3f24fe1b5f94ac16cb86f431593af9a99365cb4682442c844d801d61bc1ed3a2 |
Hashes for hugedict-2.12.8-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43d91b2acc1ce55b352d7dc62ec601b2e96614c26a29ab0eea62758f8527af67 |
|
MD5 | 074c9f6b4a118dccd4647956d7bd282f |
|
BLAKE2b-256 | 59cdf689219cdd71fd798cd09444c5132c81af90d17f88929e4d3caf9092aa35 |
Hashes for hugedict-2.12.8-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 219459d9ea48d08a4840e9d21969e099c8654abff96b644306ca76c5c6ce0a6b |
|
MD5 | af1de3ad177cad56547eff407ed17d79 |
|
BLAKE2b-256 | 839bcb1036065043ceca9786057c6749e058c458f4667572778710da0d2b2df5 |
Hashes for hugedict-2.12.8-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3b7ccbaa16965c3e22419a2876ec9e3304cc2e3d95c61dbf0e5067017943e521 |
|
MD5 | 227ac248dae036b37dfed4f3f6ab60f3 |
|
BLAKE2b-256 | f3c66a50490a54367574366aba4fcd1f7d32add9451a934e972228cd2964733f |