A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.9.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 30a34e597a054436ba9b59af04e8481ce335812d34c91037786e6515cea9b53c |
|
MD5 | f355705bebf1130161e9a9a060807c2e |
|
BLAKE2b-256 | d9ec67623f8628a4bcad24dd3709e5b462a7b8edc3186d0970efc26ec8dcfcaa |
Hashes for hugedict-2.9.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | db5aa6a08d2d5e662d6d67a35925256bc28cd6069853fe8c6ddc410bccdc5e5b |
|
MD5 | c7eac3b3d577155bb671f9c88e908651 |
|
BLAKE2b-256 | 78398b3d764df6890897ae95f9b4a13de41dd3163e7fcd533b11027a42caf95c |
Hashes for hugedict-2.9.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | edacec118a1c9f7f30b15956cf2f638e74b902e61a26795cbe6dccfa28fa030c |
|
MD5 | de976f0cc70601fbc41316b319792242 |
|
BLAKE2b-256 | addbb0c6738c312cbd038afce136787e4843efc06b52d6dfdf2ef20e00216d4d |
Hashes for hugedict-2.9.1-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b19a5f2ae491307a04834bb220c84ec14b1ff277ee4c4f6891ba40032f021232 |
|
MD5 | b24863fdf9a75fbcbbd03dcb3c557cd6 |
|
BLAKE2b-256 | c8d0b69a276704f4379770eed92a0e22e43b8705ec2c8145b48e74731c3939c3 |
Hashes for hugedict-2.9.1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2c2785f3fa20159d22bbea1f9dd6678f8ffc8b3deee6baa27f31869d519b39da |
|
MD5 | b8cfbed09cbef6c234ba971d592a357d |
|
BLAKE2b-256 | 2c17de3eafcdb7baa540be08c8b4f9849d849c0618405efe0ac7394709bb721a |
Hashes for hugedict-2.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c8108945c5e1390bdcdedb492462905e4752f34042d0f03d3faa87bbbe55980 |
|
MD5 | 78a1f1e88b2a2e7728b56f47dfddcf6f |
|
BLAKE2b-256 | e02a5e43bb3acafd2b78320a701fe8dfebad23ead6ddea0306db426e11f765bd |
Hashes for hugedict-2.9.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4eba8cc2ddfb05d04e52d5266422a0600b6f4e6d31c55a78027127d0067816f8 |
|
MD5 | b19f48ccd8b8ac6f87509f52c8e73a78 |
|
BLAKE2b-256 | 5c168d95c14338670b91568dce76ff712fb163ca9b357c945287df73f7e72e5f |
Hashes for hugedict-2.9.1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4927dd93a3a35c0548398ccc2d6ed5852c8b4e18385ddb238e5a7c88f8149291 |
|
MD5 | 0a8ef8a6fa05b683cb19944ef86b2a5e |
|
BLAKE2b-256 | 8b402d639780b1155b0dd260859714c0e36df1f174b71b2854a379d9df804c15 |
Hashes for hugedict-2.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ced95113f452bd3fd7290eb1485e783729300748af84fb8047a6f48426784b20 |
|
MD5 | b6b2e7125ef028959a6d8ba833471e23 |
|
BLAKE2b-256 | b0204d44dd784e08b87fe07af2d16be72f0d986afca7bd17c268c477a97b2dab |
Hashes for hugedict-2.9.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 02aeeac776f7883a7b7b2efe93a3086ff1f21d22305c7d25793f388e6581e534 |
|
MD5 | aed7e0ca040381dca9512feee358f935 |
|
BLAKE2b-256 | d4043d0119bb2920a5435a66a9fab60f791a4643b043a10d080938241b7e6603 |
Hashes for hugedict-2.9.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 939de8a047ebf39b3411ca4e06ec0cae2a25109f70b75231f4e5d938b4fcd711 |
|
MD5 | 1a45b449c3fdeaa82f51a0770320bd3a |
|
BLAKE2b-256 | 70b2bbe75eb7fe9ea8af681c469cc056fcfbd9cc5a59ad1ba801b4ec1fac3635 |
Hashes for hugedict-2.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d9aff72eb9c816cdfafd416da981096345ea4d3f2d54c628a7fd25b6a032a91c |
|
MD5 | 07658ac75debc8cdb93d5f90aac49eb7 |
|
BLAKE2b-256 | d2dfe4ee6e767a69c826894244cf38bb795eaccfe1e22401b5c9463bf4f53e8a |
Hashes for hugedict-2.9.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a740c16150acdbfb01a6d11e418b1f478119a0aa306412fe29e34c54147e133 |
|
MD5 | ab5a05071f21ca090fe25999e2ee1a40 |
|
BLAKE2b-256 | 01b244b22d0a3a7de59ff05c050e3bd9bc32e4f4ddb8a7769b14af052bdcef18 |