A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.7-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 67e3dc076223c28dc1489202463bba92e0ab37731ad13cc5f399fb570d8bcc43 |
|
MD5 | 746a694ddde731f6dc7325f7559afdf2 |
|
BLAKE2b-256 | c804a394cdc1d328117886d5a5f3d0d6deab49182f116c6b38c536125b00ccea |
Hashes for hugedict-2.12.7-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e031806c93fc0ae04f3315384937381e42bab001a43430afe4d1f9a41e4bf967 |
|
MD5 | 7a7926abf0d5a588b9c2f99431f74c3d |
|
BLAKE2b-256 | 728e4811147ac9ccb1a1b1ebea7579dc9bf25858f51c004eb3e3ba3c5d5673ca |
Hashes for hugedict-2.12.7-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 82746f7de1c7a98dc696f4aaa059ca57d79c874b272d2fd024ca5a49e78c8094 |
|
MD5 | 09084e47d0acf07b4cc8c6d75839550e |
|
BLAKE2b-256 | 2df59676afecbc2284233eba9dca02bb70e4b3ff96f4027c223e8dfc3476d842 |
Hashes for hugedict-2.12.7-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c09570cfb80cd812c525746517c8376871082b503b31485dd0d5b02c81c4e718 |
|
MD5 | 8bd5dba991f552ad45c7a97c2361cccf |
|
BLAKE2b-256 | 817daff0225af93442d417a1e0d7cfcb626e40c2fd2f3d39aa9f28326ea7351f |
Hashes for hugedict-2.12.7-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 45b0119ac07a4ba762c33f594635d19d9800d9d566d49d8997966326c0413fc9 |
|
MD5 | 140fa5353f007bd276708c3a333ab0fb |
|
BLAKE2b-256 | 8fa8d77f3e586560d1e502b5d1e31a1bde2bfffccc4d820546288ccb2b676a47 |
Hashes for hugedict-2.12.7-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6ae5e1b83f587302059a83ae053aa69007623e600135063b25be3baf60497272 |
|
MD5 | 160795efefbd668a89f64516878249cb |
|
BLAKE2b-256 | 8df5c44abb47c22e331a21b457e039950faca623fdedbdfcb52502febbbce154 |
Hashes for hugedict-2.12.7-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb4780f4cbe8061291911258c9cd022ee865c10f37aac1a2e2ee7358c4feff7b |
|
MD5 | 35363c5e7c846c697606fed10069f761 |
|
BLAKE2b-256 | c6c2e986a961117f9656d0c1e7f595479c2e7183ce30d5ece6a63fdfae6cb77f |
Hashes for hugedict-2.12.7-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | de54cea20ad11445736e263ab8dbd8343e2b1b123fa2593c9ad586247ac498e2 |
|
MD5 | 47e63d0e1f0cacf0ad11b7efae8d4844 |
|
BLAKE2b-256 | f03828837d128af49aaa44b88a93f36e56279ad712ecd0848372f7df3690e207 |
Hashes for hugedict-2.12.7-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c867024df94860d231df1db956b27ad937d7ba57344b057e34e5b07ff24ad824 |
|
MD5 | 2cc621ad0ef286d47b3e04716cd06201 |
|
BLAKE2b-256 | 47417d1d073e610dad9fc72c9898a1248ecd50ae0b6de1066d800a7ba8dfc843 |
Hashes for hugedict-2.12.7-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c00920c4b163ae2fa96fcf7bb33764a23bbd49ae51bb5e097ce9a9e371386db3 |
|
MD5 | 2868b32b5ade87ce77f331d15a66c88d |
|
BLAKE2b-256 | 2142bb6f87594359ed7c9214de67a3159a5921727315e87f5b7c5b1b89400124 |
Hashes for hugedict-2.12.7-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 804830a3ca3cb4a723386735c329e0a5d4af92fa071ef167848a39ffea8da060 |
|
MD5 | e31b4080d535cf414311b77e5d4f3b64 |
|
BLAKE2b-256 | 5a08ef6ac8e31246d27f3c605957415772e610a2c20bfc09156ee88ccde0c142 |
Hashes for hugedict-2.12.7-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac95768964c3a6c356d199c0b89d0100d71d9a4c5eb36936af2f1c3e075ff0f4 |
|
MD5 | ca7fbf54919fe4fb6d2b25a54c51f73b |
|
BLAKE2b-256 | a50e528bf9a465f61ce3a9bdd882c4cd0db277ee32f9968e1ab42df2a1308802 |
Hashes for hugedict-2.12.7-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e51cb03e9f940a608ae43b95d6863e6bec83f0d6f39185bf9c6ba17d3b377e43 |
|
MD5 | 4a2ec1a6dc5be60917c6599da9e413a6 |
|
BLAKE2b-256 | 1a4e03c736d57f3d89248ea68e7d9656d0ab940036c4110ef878c95efecea7b3 |
Hashes for hugedict-2.12.7-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1ddcfa5db2d12763f31488377df20f10555ac63a8d8c192ed41f1f857018af7 |
|
MD5 | e45e41893f89dc3e54a34c95720ad6eb |
|
BLAKE2b-256 | 0ef3b75847064126e19ad823c59756d65454509476a6b84b762c6898a554f8bf |
Hashes for hugedict-2.12.7-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b1d221610e6d11524b65d89851540880b4b2f99a77d432b16d4d69a170a491f1 |
|
MD5 | a25c31f50c37ef8a1ba119463ce0efa6 |
|
BLAKE2b-256 | 11431efb8cc2a9e0ab9a88512726145562e3f33bc9da339a1e97526ba321d6cb |
Hashes for hugedict-2.12.7-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84db15b111988e4c4fd33909ee85ea87ea5ec11dd82f47bc6b2ee6670d3eeea9 |
|
MD5 | a369c00f96cc7ccca2924a883c61fedd |
|
BLAKE2b-256 | 843ecbc34d333ebd612fc2076951371eb6ea3ab3a0e66d9aa17a50273a029911 |
Hashes for hugedict-2.12.7-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d6b1d1bb6e40ec7271d491a45416f7c0d864cb4e49e116804da87ea9fbe3d4b7 |
|
MD5 | 2184fc7822ed48eefa59797350bc91a0 |
|
BLAKE2b-256 | 6d97ad37b050e1a14dc835f0f33ab102e924a8411064afebef20d65129335df8 |
Hashes for hugedict-2.12.7-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f96f1f4da689082a588373f8d131fb1ec8fe91891f19f5d6f9a79dd1bcb59042 |
|
MD5 | 4137bda4e1efb94761c5bb453d3b3f5a |
|
BLAKE2b-256 | 787e3467e6795e652f19dc688e5eebd2d011aad1c83caa96d0b3b68bcf15e784 |
Hashes for hugedict-2.12.7-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48bba0956d0eaad1c37095240be986902167ff98047117d619f73303a4737964 |
|
MD5 | a10c9b2ca8d485c7026f856183264df0 |
|
BLAKE2b-256 | a02fc39f82c15ecc4bc222cdcda337ed9249375bb856d02fc49d4fef734e7a3d |