A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.4.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04ce70e9d0551d1d3efb60b05082a8188d12f9a4caf80dd0226cea39d2f609bb |
|
MD5 | 4ad7621edb0962a0035c1e89bc7cefc1 |
|
BLAKE2b-256 | ed11609639fbd1b781dd5ef81930ad5e400be815b9b7249099e05d180657daa0 |
Hashes for hugedict-2.4.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f89eda8b0b53f0ecb8803481ac07d0d9e13415d69b43beb21d2db65fd5fe48b8 |
|
MD5 | 583ac18e63d9e1056f2b66dd8d2b750f |
|
BLAKE2b-256 | b9d2f3cf2312f8c01c116e087643b193762601c58c554de8dc4890653a4ef57d |
Hashes for hugedict-2.4.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca48cc5cbe3bbd7f50457206ae18782c412e1e429dfabf627b4e58eae2f49dc4 |
|
MD5 | b41772a9a8e7d33ebf2f3500ab5b9b4a |
|
BLAKE2b-256 | 987cf256861ef502b0160d885c950c95612ba4fdaeb4b89ac7c0c0e2f9c60244 |
Hashes for hugedict-2.4.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2d31a56f400e66de03112d79e551ca4bfc9aa0f7bc236548f0be74127cbbb94a |
|
MD5 | 5df4a32805c0d89cb8d6e7cfb3356cda |
|
BLAKE2b-256 | b27370fc59b3886c9f561a2d465c51950824a4cfd467dc7335ddcab274b271a9 |
Hashes for hugedict-2.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 291123a7c1f2ecdabbe2913bbd767cc605f2fa7d16db8580d5f8807729c3e609 |
|
MD5 | 48c909710b0171519c8b8f82e49597fa |
|
BLAKE2b-256 | 1f6b899ddb32125635e90babbdd04e3ac7a5caf5c08920c4e35d811e6dd40991 |
Hashes for hugedict-2.4.2-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5421a3da8b4ad48e44540b673b3ab5b3a9feb62c3da56d6843ff94186fad987b |
|
MD5 | f20be8a98a82ac117b59b20146ea2358 |
|
BLAKE2b-256 | 5028806fbba6a7231fdc7fe98d631a699cc340c330b478d2ed2fea3890ce44b9 |
Hashes for hugedict-2.4.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a398f5ba28dd3b41fabbb5a3b180e3fb787789666fc3db6fc3e4cf18bc348bbc |
|
MD5 | 0c380bd6518cb843cfc7c1e60ba5356a |
|
BLAKE2b-256 | 9f74115c2bc8edf7f570571f0d60137bb5bf9bff781d389c4a03e910af383c29 |
Hashes for hugedict-2.4.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 172cc4888a2faca1f0dde41ed2426470858ea7451ef180ab91ed204ffa160824 |
|
MD5 | 8364882cb510731af1411667d8b4c0e5 |
|
BLAKE2b-256 | 45a0fe8a73f000beb2ce4e7302d10fdfd66610a321902011746c3f7366d48c4e |
Hashes for hugedict-2.4.2-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fc24147a29c99d70ce103f0e2ba570c2b5184f05e762358d6fcd59d31147a5df |
|
MD5 | 21f1706edf3374cd57d365e299e88505 |
|
BLAKE2b-256 | 975f983bf5c846aa2e21fd2249347b2a7751f2ad49664e6c9ae1277b7f7a36c1 |
Hashes for hugedict-2.4.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ffc497e222215d922461c6a5f72ea20bbecb2263bcdff6765645725eb975936c |
|
MD5 | 39b9614e5562f296354f975fa49eac5a |
|
BLAKE2b-256 | 3f4a3ba8160fab1b6c196c538fe03e7b3e727e6c821466d4482596c33d50f866 |
Hashes for hugedict-2.4.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5c67cc4e766c4d9900344201bbc181a3d28b4d4831d3d6f2768477d97e911888 |
|
MD5 | 8800fab942b3985751b7c50c223c0ecf |
|
BLAKE2b-256 | 3023c9654e54c4d54c9e0b31290d85e2bef108abba5b5ad5dedbdc469cf03c47 |
Hashes for hugedict-2.4.2-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ed6980f28ae2ac136ee28c49bd3fa0fdc60648ea034fa23722da8b35d8214b66 |
|
MD5 | 430572694f6c7567ffa3f0737a6ee189 |
|
BLAKE2b-256 | 36aff04abb41c4c04f8fdfffb31b49e0c946562d807d684182c8e80831db8047 |