A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.5.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab830730812ff08bafe46f17436f8edc91b42354ef94b248ff4796070a7aa080 |
|
MD5 | bf2b9e26dd8f94738959f5ba04f5a102 |
|
BLAKE2b-256 | 942826fe412d81e2af43cc341efd1d72a2ef18f134d0be3a9f466e3b0b834d0a |
Hashes for hugedict-2.5.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eea0e70e62d304419a54512333feb6afc36432e488251e08dd01209b46eae79d |
|
MD5 | 4f0aea007a0098bb201d07e9a1eebb65 |
|
BLAKE2b-256 | acc44949642c925129c453abf0b58b91fcb4c4654913e571951c207c337ef998 |
Hashes for hugedict-2.5.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4536aa8fe9c25dbfebd2b1d80517c462cd2c311fbdcc0e76558a4d3441c791d4 |
|
MD5 | 10fe0fdf793f584cd18cbdaf360a5aa8 |
|
BLAKE2b-256 | c8e646733593d8f8b999417594411cfbcf4288c65f19e9c6adc74a53db8bf641 |
Hashes for hugedict-2.5.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7f7e823839992a5ac6aeba514a9139d943265d7de678132975c95f6f808e89a4 |
|
MD5 | c6b3d4f1578196fa61c610852f2f3ace |
|
BLAKE2b-256 | b89090e3c69683bdfc5ead95b51cb1bb34661b3c4c446b55eb6af11d4d9c7ace |
Hashes for hugedict-2.5.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f0254505b0724d97441934c7ee968a6806f05e555a012005c427fa57fe33e37 |
|
MD5 | 3cc89c667795cc3a95d3d39bc44038b4 |
|
BLAKE2b-256 | 37257070347193d6f6bf9b109436fd4bb32c975f0959c16e2de1a896ddb283d6 |
Hashes for hugedict-2.5.2-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 48300d6dc03616193ef053ba64694b645335dcdf227de86db956f46256877d12 |
|
MD5 | 4879d56359c7c1af62183ee399b07afa |
|
BLAKE2b-256 | 72d6792c7844c16fb2a6a41f07f0700334056b489e3590f8e8ae6887e42db7f9 |
Hashes for hugedict-2.5.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6c1136c3543013500a29b5738f8bfcfaccd6735a48ce684964119a88e0f680bc |
|
MD5 | 37f423ebd03c4657a01951e822329ec6 |
|
BLAKE2b-256 | fa88167a34d6a3956ffc880856e2adc9e6765bd7aa97fb1e767063b90d25050e |
Hashes for hugedict-2.5.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2f0777524ce2cb1cfee722852f518d946d6a7065bf5aa05592ffe1a7c3a0567 |
|
MD5 | 714040181525016de65d97e1764b3440 |
|
BLAKE2b-256 | 8f9544f89897c3d35304371b623a164307adf02d33b0602f365958d6e5225620 |
Hashes for hugedict-2.5.2-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d51ebb4da6b4c2c58085b7728df22f966e8e42dd24f0e2d43eab004641052ad |
|
MD5 | 3a65233abb0f6bc916d79c60535ffaba |
|
BLAKE2b-256 | 2b5a172395afd93aa67f22473e4b0f8e8930e79085e469baf4ef3b00eac68b9d |
Hashes for hugedict-2.5.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fba2a88bf2f821ef995cc33eb05d31c4d8722bd31c5b91d0515d0b2cc6b94037 |
|
MD5 | 478b37fda3ac834c1afe7badde43436f |
|
BLAKE2b-256 | 47c5d02c339ed28b4da5198e706028e7478d4b97395aea8b3b03f7324e1eae74 |
Hashes for hugedict-2.5.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d67e580c4d1eb78c61a174fcd79803d981d0ddc89aa0bf1ed7bfe0eabb46419e |
|
MD5 | 81ef02bfafdb417703b6b5f82b880081 |
|
BLAKE2b-256 | d3bc9ca90142024ed4e417e3e1829b4bbdac0b8924005a78c9e6d5c46f6852aa |
Hashes for hugedict-2.5.2-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b563b43d549fe74af077e2cbe16b52a6c92c7d45fa28db42e82bbd72ae0757b |
|
MD5 | a3e4f98eadb39a997bf526ffa0660eb4 |
|
BLAKE2b-256 | 9540e6f247828c631823d6ea68f212918d34abad8ac9d50e138c920d2d604c66 |