A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.6-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac9178466acd678cbecf5fd5e023f092a2e6c49909f5d1164d3fb1bf907a652a |
|
MD5 | bf692aa1f8a8362669e9b5d51108c025 |
|
BLAKE2b-256 | 28921bd40775c72f2d63a0897c312ae7192b9cf49e3ad9017dd3acc75c546abf |
Hashes for hugedict-2.12.6-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 00e219f402e1e4baecb4e1e90002914123b6f7fbdbd213743dcaf39c9bfbfe46 |
|
MD5 | 534bcf5a2cbbad16a4e6d8647469a5ef |
|
BLAKE2b-256 | 72e7155fc76650d1e94599af6024bce89cb1114adcc92e93ec44685f5db8a15e |
Hashes for hugedict-2.12.6-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f64e8ba5225053d93a57bebcd777b7b3ee202778b4fb0fe3caf92bbd02e6a9d |
|
MD5 | 0a2b0bd0db5d7b649e592ce97ed274bd |
|
BLAKE2b-256 | c60b4449c1c0b85c26beb3c785b7bebc8b07e371404cda98afefc634853df755 |
Hashes for hugedict-2.12.6-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e751943bd5c3d9e971b051447a6cd10da42e3ff4e3ca59fa3d45278353c5184e |
|
MD5 | afcc2b61bf4d284ac4f0c717b520755b |
|
BLAKE2b-256 | 9456fecaee98ef3f40ff01731ba0c4f6da9fbb6f988377c796cf2e4fcb2da5f3 |
Hashes for hugedict-2.12.6-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1facd289195e0aee64f53dd7e12fcad9d25d8d97acc434ccec6bcf0c979aecba |
|
MD5 | 6c64d59bfb89a55a7f1121fd474bb189 |
|
BLAKE2b-256 | eab61ff1f7a39c9c90c90b9e60732d76314a1748ffd53c4c79a73bc273b07b48 |
Hashes for hugedict-2.12.6-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e3404684572aebc87b28541aa92dc6a66ce838724fef24624880d0ad1174c8ab |
|
MD5 | f7edf21c6d6a28f18207c2ed5b89da73 |
|
BLAKE2b-256 | fb848bc9e98a8216953f60fab2391d2c4dda62d12f86af1d21f47e050058c52a |
Hashes for hugedict-2.12.6-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3013e9bbabaa977c74cc9596a6bfc45992dfb095c7fd3cf66fb7f5c92d18c2e6 |
|
MD5 | d53cefa463f5efb298370967525c30d9 |
|
BLAKE2b-256 | ef5edb15651e98e22b186bfb5481ae9d114977d978946e6d67d4fa4d651b1b1a |
Hashes for hugedict-2.12.6-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c47a9b4f693275190841c288abae031418351d77f8dba285891f0ca3edcb6307 |
|
MD5 | e6f2e434e1a0b5a37310a85e93004e86 |
|
BLAKE2b-256 | 16e5c68ee23ef26e22fc6ea019825faa42f3b40f78862db38f5590efea291422 |
Hashes for hugedict-2.12.6-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f7f3f620a9e2b6372f3fc34888fc17c3004a0e12b61673ef2f0ad49483952ae5 |
|
MD5 | 178fcd2ee52c948827d980ef47c2c42b |
|
BLAKE2b-256 | 113a636cdeca569e1aa9e3d5371b0ea286fa10ea32c5a1d0e8f63e49a3d5a643 |
Hashes for hugedict-2.12.6-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | df6984aa1fd026f86602f8007edd92ebd56eea00a39fc45891fbdb4a0dc48bad |
|
MD5 | 723c14d53e01613b8aee5fde2b7d5a6d |
|
BLAKE2b-256 | ab49e01cb1ab60558c057c5e081b711cee1d6c5bc268a01b0459df2a642c25f3 |
Hashes for hugedict-2.12.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e1fdafc5958410678e334cf2639578ec2337f87b6a8c048f1cb79d9f4f32159 |
|
MD5 | 53c26698782170e3eedd90f42fbcdec2 |
|
BLAKE2b-256 | f9e38ab3b2c7a5338253d81c1b5bbc8dc40ac486cfae67068468a6d1808309ec |
Hashes for hugedict-2.12.6-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9ff609e1e8bd03c09559c718802412b63f1138d0423ce809242881aba4bc141b |
|
MD5 | 725b2641e879b5e554c7f2c73f041371 |
|
BLAKE2b-256 | 648b96912b24fbc42198e3ddee192f332c7c410c029216b310871ebe28f3e085 |
Hashes for hugedict-2.12.6-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2db69f97acb8393d92c2064bace04861512a0677135e11cacf721960e4f3c14e |
|
MD5 | 7bfd4c6fe4c2815620c45a8ae7dec8ce |
|
BLAKE2b-256 | ae415369050efc6a8df52a8ce30d53a6a8c3bc869290e751ea151d6740a05ae9 |
Hashes for hugedict-2.12.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a482cc39a8841ae4419a8313f7f3f04e5d80dd6907d9918c2fcdb43493a5bf1 |
|
MD5 | 33298743798dd23dca3d6b0c3811984b |
|
BLAKE2b-256 | f5ae3d10a04e7fffad5c3ec0663b0561ae365b785c83a20162aee53c45eb692d |
Hashes for hugedict-2.12.6-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c79ff8b73f548b74219dfac44c8d3748b611ed31238ea286f37968196cb324c |
|
MD5 | caa8c1fae9e89adcaaaf1741b29a89b4 |
|
BLAKE2b-256 | 61c828520a75bff2a9443af250f93575bf8aabf7d8b28dcf8e84d7b742d1b9a9 |
Hashes for hugedict-2.12.6-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22fceed61e2e7d1ecdefc5f4441e5778ccf43a2089f153437b1e927da1658af4 |
|
MD5 | fbc601803f29979618be058d11a95bba |
|
BLAKE2b-256 | 7880e7e82637b82e727baa83f5462d13bb0c55ed61b922668d991f4b3bd4d882 |
Hashes for hugedict-2.12.6-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 025580ced08e9e5ac4914b3ad861a0b94abdd74e71a775df26418ea087a805db |
|
MD5 | f16272c20ecf605be0d83a3873e906cb |
|
BLAKE2b-256 | 0b8a6427e5aa0fca83bb516c0b5a411411b888fa5c70970e5c43c1adcb074725 |
Hashes for hugedict-2.12.6-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0ed45d949b83f5bf16fe1e134cc720b42e6fc6ebfe0770592caac0d9745d3ad1 |
|
MD5 | ea6f3442ebafbb3c499b35f7a0203e7b |
|
BLAKE2b-256 | 9ccfa811966dd2682f03e01457fde7a55d65add3225a4457ec4e9d390ca55033 |
Hashes for hugedict-2.12.6-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0561ee81606619fec6b5af424c9b7a4d72018b978ec1b61bc03bd42c1aa14278 |
|
MD5 | be1d21a835a23432fbc9e7b1096df4c9 |
|
BLAKE2b-256 | aaf8e95be53a31573d50cc2092f764d09a5566a9aca5dc4a3f8e34fdbdb5e455 |