A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.9-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f8ec43bee5d773ddfef92fac5b0affb58ca0e6516c183315d325e8d8a453632d |
|
MD5 | dc984015992b290ce8c00d947b5866f6 |
|
BLAKE2b-256 | 2c80efd6c54f0a7fdcc24a99eb35c0d9c28fe667db673603ef9b6e9f89f53f09 |
Hashes for hugedict-2.12.9-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14b98564664045cffedade446d99be50a34173215e96b0c18676e0cd11cf74ac |
|
MD5 | deeebfcb3f6f13891610130c3fcae271 |
|
BLAKE2b-256 | c2faadf515fbae1dc5263f6d1a117ec424579c50eac29b1a12bba853a5e1e643 |
Hashes for hugedict-2.12.9-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 50d3c5c796f53d98d00e3c1c832649e6edb1f1c087eba6a95edbe9802da98a46 |
|
MD5 | 3ba69cb36c86f4d02f8a67f549d9cae1 |
|
BLAKE2b-256 | cce136aff55627bf9cc29e423e1e6ac4837a96ad20a009565819df9b0f6eb083 |
Hashes for hugedict-2.12.9-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a3134f9a62a1b14ddbdb21e3be244a067b6a5287f5c6a194913f265dc88a48de |
|
MD5 | 88f75346075a27f6f9ed1b51423e148d |
|
BLAKE2b-256 | b6e9bef51f0c57e15a24a24fc87475c1869e1d572d631fb8f1731f82b5f8411a |
Hashes for hugedict-2.12.9-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8f0af97c7087032089b6b9d69d7536003f0a304084efef0af8c092fe1d7502d5 |
|
MD5 | 5f605b5b6ad47f2c475fdf0686d37c9e |
|
BLAKE2b-256 | 4ec20b2c02652c66a666e81d6a10046a099fab8f105de84eb4b7fc0c625cda11 |
Hashes for hugedict-2.12.9-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 930dde9331d38ef80a9bd8ca1049eb8954cd978fb3e53a597c61e470df1d08c3 |
|
MD5 | 7ea5817fda74409d92e31fbb56e0708f |
|
BLAKE2b-256 | 0bca22e3c85c28f4d7e0c84a4a5ab81340d27e67b37c69977e713616c1e20347 |
Hashes for hugedict-2.12.9-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 298bf5d8734b52fa59fa729051e30ff6dba4fff822b19f758fedf3c1917ee109 |
|
MD5 | 0d53b2276834457841f8f3ebf1d644ed |
|
BLAKE2b-256 | b4ce79bcc3c52760303fdaaa3f507f26953dd82be06323b53b0cc304ba7b04d7 |
Hashes for hugedict-2.12.9-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9bd95f01aad143a5e845225a56546dfb9cf3526bdcd4ced119a1f63a25b751d3 |
|
MD5 | c7c8f962aaadad3745b987bbca80b026 |
|
BLAKE2b-256 | c088746eff99acbc08cbb348cf0e42e3ffdf9e71368254a3a6769a390083cbf6 |
Hashes for hugedict-2.12.9-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1b546292a7e61cf23aeae9349ecd736cf7ad6c7aaa02f7833dc17c6fde088ca |
|
MD5 | dfb5f868b5822c6acb0e17f56b0af45e |
|
BLAKE2b-256 | b15cbbe68d73e7d2395766b665270b20fce3f096e19540d3cafefe8d47b4d276 |
Hashes for hugedict-2.12.9-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e12632fef7c319a86347c7e3dfe592e6d436f99d102353467616671ad6fdf01a |
|
MD5 | 86a62bbac5f05afacfd6109bb77a97c7 |
|
BLAKE2b-256 | fd0b56fc620fb0d6e1881bdecf7a90baaa685fb13602677933ecd2a918e13240 |
Hashes for hugedict-2.12.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 62886e14bb257e53f9e87ddb41ef64ba37449b5b331c54cfe61217ae2d972afd |
|
MD5 | eb7f4d005c8d7a6136babff8501e3972 |
|
BLAKE2b-256 | 9c697e8829a3984c368380aa465b942acbf19172f5efe692e05fd54bdf94dca6 |
Hashes for hugedict-2.12.9-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a298d47a77a6934d2dd17effc2153680a86fc7c0fc21eec697a883b9812ef57f |
|
MD5 | 77080f7a198a00552f5dea69a35bfdef |
|
BLAKE2b-256 | d1f6bd254463b639766f561f1e31e5d6565d0d2cc355062b1e846ec6e4a36e0c |
Hashes for hugedict-2.12.9-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1da608c44dca091edf18d7df90cc5e071097278c4da64bf618a32b8351e345f4 |
|
MD5 | 05d6777a170212d2596d794d51574d4b |
|
BLAKE2b-256 | 23fd9271de980a195080a91927b04de646b5762ba7504e7f4ab7e21fb0ea60c0 |
Hashes for hugedict-2.12.9-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b79bc64f15f1ef7e42ccdd78ff3b2403f8ca66ee8a295c25f32ab1a9fab2d7fe |
|
MD5 | 371a006e5b1164e05ec29e7b877a2562 |
|
BLAKE2b-256 | 197b2c2f10f9323ff0cdc1793fbb66b6c4a7872e4348a3150a45e5f27dacbf16 |
Hashes for hugedict-2.12.9-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58d5617e37d8c970cfedcbf261472f7a9f544999410389ef768dd81d30c687f1 |
|
MD5 | 1ac2c9f251f7668ea711daf3fce5d49e |
|
BLAKE2b-256 | 9090c73441087022c23d4ec804467ae4e08ddf1915822c8b4cca6c0188a9b1c3 |
Hashes for hugedict-2.12.9-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a9a3f6f775fcd5dffb2e1d9a0e85cb6f48761bcc59700d32483aac443c64805 |
|
MD5 | abf80a229c718a8551e9e697df3012c4 |
|
BLAKE2b-256 | a45d491f9ffdf5fe94777f9a0eb5b526a7889cbd199e876a01fb01e298e89113 |
Hashes for hugedict-2.12.9-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fab8ef5d6543e04a87863481a602c11fb2ca096334a0dfc24d6f17db389ca69c |
|
MD5 | f30716c997a48e4b215920f3e2bdc2c6 |
|
BLAKE2b-256 | 732fb889df518d5ff3b649f7ae8f0ac3dbf237921c5d33780f15190d21c73475 |
Hashes for hugedict-2.12.9-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1c8eef168ec3dcdbeeccfc528d7a8f5f0a6aeba2969c3dd40fbff22f72f2f47 |
|
MD5 | 309882eef64dd696701ece371d445636 |
|
BLAKE2b-256 | a07521e43f2068a2eceb11553585f23495b9dad9c13c6f410321ff2f32d9bb5c |
Hashes for hugedict-2.12.9-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4e5a24ed10e3119446576ffded6d9ce6e6ecc4aaad0a1c4d5522405414c4f33 |
|
MD5 | 695da085c04b43fe08dec7e6b2192d9b |
|
BLAKE2b-256 | e52887b01bfdd033a23bdbbe3750f1420929cd53922ce9531da1f2cb16cdc570 |