A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.13.4-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 765c16f3bb89a7e9059a77f921820c86096a678f91e74713da6d2de7ba49cd75 |
|
MD5 | cf5ad70196c460f4fc59075bc296524e |
|
BLAKE2b-256 | 50b568544059082ad24f8fdcdc0f6575653cba11572ce76d1830ff9846f5f755 |
Hashes for hugedict-2.13.4-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b14fc0739a81fecc4220281bb51bbf3e62d544596abee97e0f40875122dd7d93 |
|
MD5 | 5d1709ebf83b52b810b0a963d5fa1054 |
|
BLAKE2b-256 | c6880684a7a308dd80560a4a7b8d623e756fc4d210ad70a381f61f294ebffec6 |
Hashes for hugedict-2.13.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7621b39f5a42cd81914a53079497902a076bfc7d3df055aa52c403cc03137b3f |
|
MD5 | 5805b25dddb24892b6789c80e496b887 |
|
BLAKE2b-256 | ab23a3c816b803e855af308369b55423d31812c519dbb64fdc009161cc2769c0 |
Hashes for hugedict-2.13.4-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f27c8564b74adc16d64d08ff45b6bdb036f45fdb036f68cb9c098b0fc059be65 |
|
MD5 | 2004b33aa7e16e02eb24fd468b873bd3 |
|
BLAKE2b-256 | 34963608c91f14b5f702c07b5f03c38bf660b55d4e9c5e7ac318375096ce3ce0 |
Hashes for hugedict-2.13.4-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f27fde2df75d49e5bdfa66b9670d0f36a800f6d0812ac29681e4b862ad10e5d |
|
MD5 | 1ea45112d2dd5bba829d7f575e95470a |
|
BLAKE2b-256 | 9f0e3dca363bc445677916ceccd60b879d9c9ff777c920da804c2a6a3e9c9455 |
Hashes for hugedict-2.13.4-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5385d497cc818de33d2b4ec2a221ce054450d53bc54b1fd59922c850be8b041d |
|
MD5 | a1043c792440f7c8b925702cf1ce8a9a |
|
BLAKE2b-256 | 0d4fcaabb4d14cd514c121362e699f8f5ed09fb64c586643e79d7dd0c2f61b5e |
Hashes for hugedict-2.13.4-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66e4b161f535708525b31816cfd5f9159340ecb9e25f49ced8819e6034d8918a |
|
MD5 | 5d7220d34ed48a0972979d78d952e345 |
|
BLAKE2b-256 | 5ae038428ce4edeaedff9a90c869e67a27860ed49f8ee1cafc6cd34394b99cf1 |
Hashes for hugedict-2.13.4-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | abe97f016ce2281b1c157fd9a61791dd482ec09958ed0a1a6f377b19620a78b4 |
|
MD5 | 4e8c07261f55a9ab4fd8ead3463296b1 |
|
BLAKE2b-256 | f800ecbd52c72670e8575f30622aedd84787d6d47b62b823e21e348b7f88717b |
Hashes for hugedict-2.13.4-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae97cfb01a29556651a67c9cf876132a0168926182bcda3596cc52a804a45550 |
|
MD5 | a17e8e3d248e33af9fefc5012e2c9706 |
|
BLAKE2b-256 | 4fb638c1d2735ded852a87a772694245cd33e322db216dc36773f5091492b046 |
Hashes for hugedict-2.13.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7521ac43d4206c0f865fd32fc4dd0489b3d0125885edbdce59c4bcb12dd86652 |
|
MD5 | 6bf25df7e450793145edfa4ad55411e4 |
|
BLAKE2b-256 | 87ad7ac39cecacef0b13fc87bfc5756cd4269a6a08e5fe3680cf0539b0c42dad |
Hashes for hugedict-2.13.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 915c2c84c0e4d1172963193e5c3907972a08ab3e06471405e208d851e29734b7 |
|
MD5 | 9a5909465e1deac443fcde56e4ca24c0 |
|
BLAKE2b-256 | 98bfe3628802b01b5f011904d71a82b62173d4e3e9dcb807bc09aa9b28c03d6a |
Hashes for hugedict-2.13.4-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00a0f3825bfbba15ed47c31ec06ef0668af427ba23b05b5ab751dcc68845a1d0 |
|
MD5 | 22fc76049c7a6171ce002640f082a481 |
|
BLAKE2b-256 | 337fdb697961a456c5b59b3f2c30af2a1f454f58d63ff559912f44c133a56792 |
Hashes for hugedict-2.13.4-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc6c4c3967ea213d6b7e4bcc380178da2c81c582ec9ed19a6efb21ae9c205cc4 |
|
MD5 | e899b652fca620a430edfc610054d0a4 |
|
BLAKE2b-256 | a23b74a9e57e497eb294aef742b1626aab01b8042348eee39d9c00342753db39 |
Hashes for hugedict-2.13.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13c7c451da244a47c75b15baf46d14d2c0a20f13bf880b14764972ceb165eb9a |
|
MD5 | b38f2720d02fe55e48fcf8eed0393700 |
|
BLAKE2b-256 | 844534aa162fe759efff8426cf725175b7897ebc05a116454307e219e3f373c8 |
Hashes for hugedict-2.13.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d6d8f57567754576df273d7009d50b22979841d488dc0cae243edf680aecf444 |
|
MD5 | f63ab02ed1b35a7fde983fbe3cea0e8a |
|
BLAKE2b-256 | ad55027474ca747d5783dace88a898ef5b8267dbb2c45546326a3742a2a2c69e |
Hashes for hugedict-2.13.4-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 317e5c59c4a8485376c7c164fa641ee6709f75e21645f9562278fa5146424b7c |
|
MD5 | 4a888b493c3adc7a37886ce6b718426e |
|
BLAKE2b-256 | b07d20bffa8a92233187a1d8b7fc59ae1a7a931a597e951ea5d72dd80639f842 |
Hashes for hugedict-2.13.4-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c666a3c1a52f52ad86f5360186c6028d15e27af1d7e15d2b71622667810d4f24 |
|
MD5 | 40d4ca18901f34353eb7f51434ec17f5 |
|
BLAKE2b-256 | 7f95ac2b7a81a4c08db7d6063f7bb7fc5a4331dda2c384d7cad5a84741921f43 |
Hashes for hugedict-2.13.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a7dcc9e3db59dec347a81dcea9a2863bac6034a9250d49fcb35347dcdf01709 |
|
MD5 | 01846bbcc229b01481c20ea4c8bbf0d1 |
|
BLAKE2b-256 | bf5ca492b3e781f14c5f216214e86e6534ffd8cfc77fe961451c2966bf8c5339 |
Hashes for hugedict-2.13.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b585594f779106223df5cf22d2a7ac2de0f1a5e418863114fbf2d1a200e80906 |
|
MD5 | 1016ee1fd45c6f2966e1a98dd02aedb2 |
|
BLAKE2b-256 | 83a8485d6c5b26afc7ae6001648426d38d910420028ba91b77e8b7ad91060ef3 |
Hashes for hugedict-2.13.4-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c4b6b722227916a0b34c4bfe71b693dcaccfe88668a74c90afeb3e2d2254915 |
|
MD5 | 9e8c6dd428bff49ccc3b478175e52ed4 |
|
BLAKE2b-256 | ccee16b21bc009173da22742ae2c671c742d491312154bdcceed55f48171ec20 |
Hashes for hugedict-2.13.4-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01e9bb18db314254b5aba6133b87e94cd25799260293991de39b907bac19519d |
|
MD5 | a1583953b7410a851d4df8daf07f9c7b |
|
BLAKE2b-256 | 5a917812d75687fb1a2febcfe7cce9693661b15e12f1b68c153889dc15fd1954 |
Hashes for hugedict-2.13.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e88b447b9f71dc97ba47cd424a449ff90bc7d19b9dd3236458f4731656e9d160 |
|
MD5 | 4c68650084cef7726bdc556ba45006da |
|
BLAKE2b-256 | cfe4c6d984e3166016990ffd1eb03e3427020eaa7f95c4de07568cace7358fca |
Hashes for hugedict-2.13.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d10ca3b7cf68b410ecc357d429b550a3a70feee9eb97e78d646a6466715c341 |
|
MD5 | 6a9a40ea784d8624c38f44e496127e70 |
|
BLAKE2b-256 | e8a789ec1d6b4829e127e8815b5a212fa756645e314f470c24feb8db3888f18d |
Hashes for hugedict-2.13.4-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5f7924dddb97290f1cfb1f756df0657410793f765b9fa315ed0b1485f85da98 |
|
MD5 | 08469ed038c3f8779e24498dfc904749 |
|
BLAKE2b-256 | 21f4c144d15b5eebeb24b576e8ed7141a161be701452e8f4ab20225457f12010 |
Hashes for hugedict-2.13.4-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0db7981c430c37ab97f56b1381c7f3c5e1b5302791b260b12014875dad7d2c5e |
|
MD5 | ef7072c51b1d5d81438804d8cfe07c0e |
|
BLAKE2b-256 | 9501e6fd1b99ab28ba4c6af521c78075ab7d62b92448570dc18143530025b1e9 |
Hashes for hugedict-2.13.4-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bf0f7fb8071e9f2ecc48b27a1900cb74ee8015cae06dca786d455ba28d71da5a |
|
MD5 | 84545001de42d27a69c52ee347e1a7f0 |
|
BLAKE2b-256 | 744b571733d15edb3c92f21a00955fbde8fdf9ef4048bdd022e78080e31e188b |
Hashes for hugedict-2.13.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 195a3fbcc3e10c4166d9ed6d102aff5d76849d0cc2118af73f6fc3e461100eb0 |
|
MD5 | 6710fa6c83872d25e0d9eac75b60d378 |
|
BLAKE2b-256 | 22d75f8a102bf292c7f48bd964ef027a479705c3eb98c3ecefaafd82968596de |
Hashes for hugedict-2.13.4-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 38e8ac819b71c813dc6980b54dd8711059b1e1b139c22554a5de79091c54ef45 |
|
MD5 | 9f97d6109cfe38267bf7b59fcae7f553 |
|
BLAKE2b-256 | cbf91dbf564e785a2b0f2295664d8bc471ee4d1704973f167cddd933cb5bf554 |
Hashes for hugedict-2.13.4-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 935bdfbd2d364e2245039666429bc693e6955b5350bf7b9caf07e93a041229fc |
|
MD5 | bfa6b61b6088dc27b467a1a78ffa0588 |
|
BLAKE2b-256 | 03799dabdfda84d9161b5b738f6adbbcb32294451cb0197afff2aee448f72194 |