A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.8.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6e76086b959751999a4a389a82b44f29173d69791c233a8967838cd1612a5c0 |
|
MD5 | 1abfc8c01d11f19887017c3e5da5e9c0 |
|
BLAKE2b-256 | 05546e9ea3e30ee4394fd3e116f15b6e4382affb402b113c3092de1d7ecd8fd7 |
Hashes for hugedict-2.8.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1486c71b3164005fc2e430ea97629110336e2a73c5bf853e4a4481858bf7a57e |
|
MD5 | 6d4bf6bf8750eb3447d835f9abe867fa |
|
BLAKE2b-256 | b8268a31794171a3c893c149a6200005ba7825bbd0e2cbbcca0c5c935e393d3c |
Hashes for hugedict-2.8.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e500fbfd730da6629ac8c3e4b37c4218f849a3e9482af88793eaa3213e0110ae |
|
MD5 | f57f322a1fbe05ac8c8948b196468e94 |
|
BLAKE2b-256 | 5365b0fdfec979e07bc2b9cdab0ea45ecab3056638f7de9fa6ccea0e4bc5a098 |
Hashes for hugedict-2.8.2-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a26ce8830461c7fa1905c354ac387d3d34dd48a3b4a56120bda5670f03abd08e |
|
MD5 | 085b81407b9b362913f1159244903a7c |
|
BLAKE2b-256 | a6361fe05e10e09965138b2fa1dde8b46f64d869daa8c1bcd5cf20dd16c6c517 |
Hashes for hugedict-2.8.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 29cd4167462120a3de2997b17e6ff9292723b8b54a4427c5c41099103196626e |
|
MD5 | 4a7fc696302c95f93ac700df09994109 |
|
BLAKE2b-256 | ea5aee2c3fac59880d2609debb2d1e58e2e942883d0c4ab062e57b3b00ba84c4 |
Hashes for hugedict-2.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84c0b436f74bceaaba1d477cec546501ff489dd90d6740a67250449ed5b5fc47 |
|
MD5 | 44fcde1c947e386e9a83f118df58c980 |
|
BLAKE2b-256 | a43efb478910b7b60f947e5f20c5b126f18bba77410e8c76f9bb33d27c875aed |
Hashes for hugedict-2.8.2-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f85c74936b799afcf7a0165ac6b06f72191c69885f21c4e16070f8aaa0060ed9 |
|
MD5 | 714e432b96d77e4a0fc847353f81e7fd |
|
BLAKE2b-256 | 4026d9d6486eb3229ba7b9185235b275b5f391e79ec6100943e9d6b2ef3ce7b5 |
Hashes for hugedict-2.8.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e75489c2db4ae0e6c2124542a1b56d7173a38c6431785e547b9f04bc8e1f5b3e |
|
MD5 | ee1e1c02061a7bf6bcc41f12bfb4a73c |
|
BLAKE2b-256 | ff484c6edf9dbfb70a3e112904f2a999d0a0c3294e2eecc9de1705fe81132bb8 |
Hashes for hugedict-2.8.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 092e4d57d38a2733c2f38d8f90ca93766be8d9ec8c81050ae0e959cee4e76342 |
|
MD5 | dbf0e4ed5aada8987661e4fa752b0429 |
|
BLAKE2b-256 | e8675529f2d4d340ca0c9759dfb5208b9161cd061a8684c970bb00c1cc184bdb |
Hashes for hugedict-2.8.2-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbb820532724f34600d06ae22be9ea017371953209309454089ab3361e918532 |
|
MD5 | 829fccdc7a3d749e23896df62648761d |
|
BLAKE2b-256 | 1e4cd61d4f9d152d19aa72075e0ae0087fff706400d4f730bf14fbccc8ee5bf8 |
Hashes for hugedict-2.8.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 61a4b1923cd710c299935630443e703c267a1593b52a8fc4cd70702352defbe7 |
|
MD5 | f84a161f225ed6d0639ca1aff6a0e4c4 |
|
BLAKE2b-256 | 47f52ab0b0c401be7d8aef1b1dee6b57302f1830b93817e799d7bafb724e8fc4 |
Hashes for hugedict-2.8.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cddd74673fba815372a67d886cc58c05a8e98325c860d74c377f4bf0bcb1cd3a |
|
MD5 | 7c0131758c2ee9d70402695fcc21b75d |
|
BLAKE2b-256 | 7c3ed4991cca7aea596cccbe77cd4d62ff5bcfae1a8429f378d3a5c530ca22ed |
Hashes for hugedict-2.8.2-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d604be39b2eec60f8fa86a315df362aa91d44a54c602e1298bdb73a8985ac550 |
|
MD5 | 5e445999972916822fc7966f8d7e2618 |
|
BLAKE2b-256 | 9db564c9b6d6df26581079916f9031c996d5b1d2d5b65a7ee0cfffa3014c13b5 |