A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
![Documentation](https://pypi-camo.freetls.fastly.net/cb2a3c3438551102399986f02b7e576247103cac/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f68756765646963742f62616467652f3f76657273696f6e3d6c6174657374267374796c653d666c6174)
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.13.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f0e54ae4a46806ab3b116e54b629b3a5f6d51608904ebd6ec2c4c70a5f8927e |
|
MD5 | f620e6588c4f21a0c1c1d96d2c0427d3 |
|
BLAKE2b-256 | cfa1d7ffc88c461d895c74cdc330363b55d673be68f863be5f78b193a2f5efe2 |
Hashes for hugedict-2.13.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90260eaa5b5a4ed60e8e56718c76e37f17b87398f1c6899ae2c3ae947e4dd42b |
|
MD5 | ac9db5391622705699ce50691c77d263 |
|
BLAKE2b-256 | cd82af76a1952ee8a75a05c8f01042d6187c6c0d7e3cda865bf1578b8ceac59b |
Hashes for hugedict-2.13.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 522e04bbcd4b546e203326c2e0747b4093361aac6646cda524c61548eb5020ee |
|
MD5 | 6defe51763329a608301a2a49c79aa6c |
|
BLAKE2b-256 | 456fc20c7c2a32ff0e636bdee40ab14ca70c89835ad55f61e2e0614fabd0599c |
Hashes for hugedict-2.13.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b42189708e37c6080d793efefb6c15fa361d377cdff0033ad7d203b7eca0acf8 |
|
MD5 | e3e15e6bcd3834bcad33e9bad191146f |
|
BLAKE2b-256 | 2586759c798210fb7fb08d881ebfbf60264738c3025f538d2561f302db6b46a3 |
Hashes for hugedict-2.13.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4602ffc26293f86d0b3c4cff81403190502dfb2955826aa86c900947723b25c3 |
|
MD5 | efbbe3759163d682454cee8967fcb6c0 |
|
BLAKE2b-256 | e6663e1f37dd62b4bc3b073f8c587c9099e553ca20b88eb2743cd63c9c6705ba |
Hashes for hugedict-2.13.0-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 891420a85b321ee91972ef9919b43e0535472c80d1bea5876bda3ff46d2a969f |
|
MD5 | 32c8d7e6b8489e764defac47b987f301 |
|
BLAKE2b-256 | b0ba6d37c52af5879b96df340e7aaceeaa6aa7d3ee3154dd984eaca257abefd6 |
Hashes for hugedict-2.13.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 11a6f7c44760ee008c091a8afbc3621e45e9f2f260020555d24fa5fd2e1faa28 |
|
MD5 | 5e520e3bd65beb58bbaec7e2618801e8 |
|
BLAKE2b-256 | b9ca52a268184f9bf2712a1800d3f2410ec5134a038b23e51c9769e0c299842f |
Hashes for hugedict-2.13.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3eae6603a62dcf895a7c112854bfd6396ae633c4fc82e959d112a73e68df2039 |
|
MD5 | 96d0ce1279a2436b7eaae014aa559eb9 |
|
BLAKE2b-256 | 6fba6c2fdd4b39e63dac4f79b358761067bb45e348dd680045f5afbb93afc292 |
Hashes for hugedict-2.13.0-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e45059eff463833b21049545d0660a8b23d42ddb487a7b66fb2bf00eeeb304d9 |
|
MD5 | bb2070ac38f598bdcf9e69aa81f7aaea |
|
BLAKE2b-256 | 2761357b6677aa9424eedb18831bd650fff26fd670ce9314e4073268fabbd9bc |
Hashes for hugedict-2.13.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9661dc24ff2190756707e150a9b21accc1fe6936f0068259163fbf058cd1e477 |
|
MD5 | 59f9a5a4602880daffc55a97dc609870 |
|
BLAKE2b-256 | 110675f1d8d3b6742fca93ebe377031c9adbf9c185ac319fb469d1a3b0f1bc96 |
Hashes for hugedict-2.13.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 231cb215390650a4fb44c06095fa6c49cb66b44f2b7926cc2a6e4c014752824f |
|
MD5 | 202c0da08be23fd2eada388869410c53 |
|
BLAKE2b-256 | 8b9aa795ef2792907e16cc74ff675321d42737433b9d1b5442e299852b04892c |
Hashes for hugedict-2.13.0-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 026bbc4fba31a45f04b7785048c218dc755b52284334aabf307871a00ec1e98e |
|
MD5 | a8a8dc20f4661c316817a4953cce6018 |
|
BLAKE2b-256 | aa563897cc7c089b4959c4eb6d118021ea9115fe15a35a1634579a3ea8afcb8c |
Hashes for hugedict-2.13.0-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fa4bc0a8fceabbf51fe335d33726e9d52ac8d75a6e00935f5573169d516742b |
|
MD5 | 595adca03c2434c55c55c09df0429bf7 |
|
BLAKE2b-256 | af7bd884f29b22d88332602203017098b08ee8b012c07bb0bca26edd1190c4ee |
Hashes for hugedict-2.13.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13ba86ea66364b8255e3b8968e53343b4f4ef2f8a07bd70c43ab119319b94b7d |
|
MD5 | e1e35cdb9ea9d35548f29b74c3d36199 |
|
BLAKE2b-256 | a5c30ca7119dea29714b6de3efd710a8d70902c863e67219358e35a5a52e81a0 |
Hashes for hugedict-2.13.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5118527eccac0edac1a390a8fefab4b3190044e324b39b744cc0848713b6c57d |
|
MD5 | cf93423636e470f8a619bfb2c0bbca1b |
|
BLAKE2b-256 | c90a5a208d955b20484badc12ed0eab87e87f39c36b76aacd5d3ddb4fb941d6e |
Hashes for hugedict-2.13.0-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc7be1e71394844ea2f92b584edb65ea22a9e166fb86bf3e393af6507d9e38bb |
|
MD5 | 22b6d9f43af19fca93867ec07c0b757a |
|
BLAKE2b-256 | 8db1301afe1b5e8cd8b45f07b5971d18c20a2fcb87d4ed9404e8c26aefd60c6b |
Hashes for hugedict-2.13.0-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c643e613acb8077badbfb2cfe39f7f022bf6d1739587fd199045f2f0bd5249f7 |
|
MD5 | cab716a3ffb26a25df45edd1399724bd |
|
BLAKE2b-256 | b3bc812a25dfb1ac568d7248262161b53307f63b2737e61c3e64d08954b27788 |
Hashes for hugedict-2.13.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 08ef6e1ad52a2ac5d168a3b7072f5b2cd2c22943149abfbe0d4aead7b4e17c48 |
|
MD5 | 244b44ec7fabac727442fdbea8e1d3dd |
|
BLAKE2b-256 | cb9322687b3134e31f84e908245c8e33e0cbfe8d4a2fa4771eeaa4263a39baf4 |
Hashes for hugedict-2.13.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9cc7fedb2f9d5c65d323f4b5f68d23c65143fa99d8e79df85a1d480b3391c5c |
|
MD5 | ba8e8bc04cd1889461eba5af86e1b58d |
|
BLAKE2b-256 | d5753333f35916800e4d4897cfe25a22b935c2eb9c9253929d2589df22cd6b31 |
Hashes for hugedict-2.13.0-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 43add0d972afd5ff9f7039ba76f0e5ce9d55d886d0078dd36935b2ad1a86001c |
|
MD5 | 1a8fd94330e07a9e74a1aafc55d97152 |
|
BLAKE2b-256 | d58a5a1d0f03bf6c4b16a4b2df65da429eb609b2024598a40179f3b721170510 |
Hashes for hugedict-2.13.0-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca611a909712d6eecaafe97711d557bcc353ca3948093f2f95aef21cea9f6ad0 |
|
MD5 | 477eedbed10c609b94e36282cae0f2a9 |
|
BLAKE2b-256 | 0712308fc676ca1e31aab8854344594886499ab1040dc4305fd7af29059309b9 |
Hashes for hugedict-2.13.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 91897fb237c3d9ff539905c6b471c5de66b91e9243093d36677dbbaf488f8dad |
|
MD5 | a00ef1355a595f6ec89db6b73304fa44 |
|
BLAKE2b-256 | 4fb6173e089c1eb3a6d57d7cd4ec5c9b4afb09e9f15fe6792a98a103eaa1c680 |
Hashes for hugedict-2.13.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1081db9c68d95c91c25f5ad4c69cde88e1135d67bac6af0085ffa1a86eb7cf74 |
|
MD5 | a24ce34e085b1fe0da15f3231367a099 |
|
BLAKE2b-256 | f69c75c488e1013adb61120eaac4a33e92132c72ac599cf671c732603b686813 |
Hashes for hugedict-2.13.0-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7d44325189dd42d6719016b159529703d2bcf807d0569984266b8a537476a0f |
|
MD5 | 73fc4d6e5386b80ba939675c8b4be735 |
|
BLAKE2b-256 | afb37eb323ea360c3c4691db590d23fef2f5608740b1b8feff7f11746c90caa0 |
Hashes for hugedict-2.13.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c4ad658c24c131ae260a2a960444bd87e1302d2be3b70d7edb7da2be7f9b4bf8 |
|
MD5 | 4ac6b378d04fbe78e9b7686fbb407abe |
|
BLAKE2b-256 | 1c9b95b1b2700333fb7c0831ffb5368284873cc28eed995b94acb1cc952c0861 |
Hashes for hugedict-2.13.0-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e38b2f4de207734ca9a2658c7575eed6e44c3d0e13697cbd9f8f3698fc6cf413 |
|
MD5 | bdb8f1dbd85c4ef203f03e45d48d1890 |
|
BLAKE2b-256 | 4e03d81a71c282ada48eceb838c7357306def924473c77a03535a0d8271c94d1 |
Hashes for hugedict-2.13.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66efa1b31b569006e1dec4fb4f664a5850b39f8d3cfd1291ebb5d6c6bc2c7415 |
|
MD5 | 2ecdea2b6751eca00758490ece8e8587 |
|
BLAKE2b-256 | 54df11044f323b872d606844134bddd30836bb1976267826c12b0524e438d7cb |
Hashes for hugedict-2.13.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 05303995cf86e4a607f18f795e73b15fbd81b940fc9d4b69bd38be3176ac8e95 |
|
MD5 | ea2ffdc81aa42d5d877079236636f741 |
|
BLAKE2b-256 | 5989a8dc9d40f2c1fc81ada5fbc7a957721ed80ef61a3577581f0a8bdc5984e5 |
Hashes for hugedict-2.13.0-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5d621afa68d70bd1c757e89ddffea89e965e54783392dbd096565238edc3542b |
|
MD5 | 5a77b2efc359fd92a070861ca01e4b15 |
|
BLAKE2b-256 | 3f64710b02768e8390f1cc028f45e4a871494c1172dcedecfbd10417f815f8f3 |