A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.5.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab2b86495a5a7156cc0d25fc5414f09683dac61319abc515b397c4516669b745 |
|
MD5 | 859138c88ab0fcd61ab78293a52a95e8 |
|
BLAKE2b-256 | 04dd61bc2affdcf3827a3ba19dc93110bedb28c61e9dfbc892b77d70b79bbd5a |
Hashes for hugedict-2.5.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4022325051863522f8619c2567fe415779273dda2647ea9de17e94321f55242 |
|
MD5 | 76c1e9fea7497e6c4314f37f1ef8d79b |
|
BLAKE2b-256 | 3495f0d4617bc938f87eae20ba1dfac274aa14b8358c9f66b4fe9bd0989c852a |
Hashes for hugedict-2.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cef0eb72ed9cf9b5fc6eb07e5c354cfc997a4063c0b72642a1a86148cb03bf2f |
|
MD5 | 14801af8903b6d40ac1f9ed6d8249ff8 |
|
BLAKE2b-256 | fd4d1f67d4085538de07737b2c7af91ee93f0674a1fcbf79a2e0ed64614311a0 |
Hashes for hugedict-2.5.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 743c03da089b1a59a6fa8d5d2d7b3aa94a9913fd5005aca93cc746b3037f240e |
|
MD5 | ef942c5fe0a82395742646d320e0ff49 |
|
BLAKE2b-256 | 49f0dc98f381bd9c1b9755d425b4318724c633a58916568e785ae3246947a026 |
Hashes for hugedict-2.5.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 236faa544473d9f8d278f1af88a1f0c573bd157fddbd1a0f7cca4704de05afee |
|
MD5 | 7c740321f69cd5c6ff4746ef239259fc |
|
BLAKE2b-256 | 476bfa3c473a70c7974180b5022d611aa16ac62bcdb19d6aa15fb27907b548e0 |
Hashes for hugedict-2.5.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4c207e6a7a64902e724ed40b6941f68d74faf4ace2ef4ceef0fb86f8b8abce97 |
|
MD5 | dd5da04b93f2f7b9cb9afd64feeb63a6 |
|
BLAKE2b-256 | 73d94b44fcfa0b63a6664abd38056ac2754a6d6226009f63a087ab881628f189 |
Hashes for hugedict-2.5.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 14b5bd2325ce3d2624114f36b34b389c67cab3fe4682a76dd54748c9f913399d |
|
MD5 | fdadf5aae4012f78a1a44fce58ea556a |
|
BLAKE2b-256 | 2aca911fcb9473200796b2d934703da1e08181f3383b3108e8a9fe04c39f7fcb |
Hashes for hugedict-2.5.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ccb36401ce7b8edce13a739fb98141badde22b564311a7107fae5eebf3c81b3 |
|
MD5 | 0b9e5093c26d071a64eeec4e6af73fa5 |
|
BLAKE2b-256 | a247e2af997c4ea61196b22642e2c1752ea7bd57dc6f6c15daa7cb0c05970f8a |
Hashes for hugedict-2.5.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9056dd5612db41f07dd7010c7c871e5e96ecb68b429aa7bf1acc5bcec0a7b0ab |
|
MD5 | 12dded5200d011722631aa310fcc3736 |
|
BLAKE2b-256 | 65d568307bb6722876e0a2bc96e66ed637cc383de2605a2ca46b4d1183cd5a50 |
Hashes for hugedict-2.5.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12a370f1bba851090801bb0e7802dd1ed3048b4c5be368c57606df77cfae97b5 |
|
MD5 | ee8070a72abb7707dc8858d4d46cd5c6 |
|
BLAKE2b-256 | ca613b34c3965153fa10167d1f660c0ebcdaf7eda846b5aa1d7c583c45eb64ca |
Hashes for hugedict-2.5.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc7d6f6b573030eff9abed426dd1831b9e2e3fbc1e20d56b70ee075fd4ae2f1a |
|
MD5 | 8b2888732e626680dcb70d1127c254a3 |
|
BLAKE2b-256 | dd5839e5ad89e55edb826c3700883217c73786c6c8a1c8088a9ece4c27041eb3 |
Hashes for hugedict-2.5.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 95d698db01a21d498942fca8e8e26801a7ddf3d621814066706c9c20cf79fd03 |
|
MD5 | 630a3bbc3174e9aa869929b93ad49c19 |
|
BLAKE2b-256 | ac2629521c6aaddeb777091d561345f8a5e5c84cfde4f838f317891c1fe9f55c |