A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.4.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 836f9a5582178809a15cfde9b0e6909bd550140aaa4db02619a322ef7fe24bc0 |
|
MD5 | 065760e37b484611e16c22dac609ec0a |
|
BLAKE2b-256 | 6a50d13cfe537a6890b55e3b9c205795e68ff1393b90119ab6f81e1ca00159cf |
Hashes for hugedict-2.4.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37758194498b20d90a1cae0bbc8c8613650f7378c3b5feb48de6690289d94a38 |
|
MD5 | 90e9ea72edb360573278219404028d71 |
|
BLAKE2b-256 | 49a82469a8f4bda91b2a255478dbf5589cc5fa597cd5f6d9bb40804253e9e0b8 |
Hashes for hugedict-2.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e49d7d6ac470ca4368ca66bc4f9a7ee0851ec0e95d8d507c710647ff272146d |
|
MD5 | be75d8c6c1368c967ab64ff61a32fb2d |
|
BLAKE2b-256 | a09204e59ee2d8dbd976a585410141301cff49643216eb4ee0b694b45fb82082 |
Hashes for hugedict-2.4.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c1894b5bf30a665fcc3aa1af8b8dd3e374ccdfbccd80d263020bfcc4397bf2f |
|
MD5 | 213c4c5e34262032c8f3d36ea9bd55c7 |
|
BLAKE2b-256 | ed2ebfba3047917ef9c67795d5093d9025b3994adf865119ed5f6a804cd8b359 |
Hashes for hugedict-2.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f2a10ad5bcf5cf6fd969f54b40547ca7ff7bf59cb7d89feeb04c913e95589b8 |
|
MD5 | 7c76adad5a91ad1e0d081fc01320f809 |
|
BLAKE2b-256 | 7655754a0ed915e8a64fd98dc7a01135d214686c9b98ebd453fec2463e868b83 |
Hashes for hugedict-2.4.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0731188823b59a1dcc7e57ce5edc1cce69cf81edd544a10bedab118b7557065f |
|
MD5 | ec0c4a9aedd1ec0d379d85f6acd623b9 |
|
BLAKE2b-256 | bf906bbd3fb903514987147360bc645e0c47cb0d188b1a45b421d642b156ab4a |
Hashes for hugedict-2.4.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb237703e1c46355a4a15316d62d6616fa6885347abf662eb8de956cd58e2bed |
|
MD5 | b2f4523a0e4d64e5f4eaedc964452894 |
|
BLAKE2b-256 | 73eccd0f0cd3420e1f5286f4b84b849d7b358b18617f49a6c09d2768af9d630f |
Hashes for hugedict-2.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d3c9e99a829f26857c38f928fd9cb904abe8b6b593b82a990af4ace51a45c64c |
|
MD5 | 253ffa76c37f544f78f4e36ea5636d81 |
|
BLAKE2b-256 | baca176083656510cca76b968ab459455bc5a9d949c8bea1cce182c1c6059286 |
Hashes for hugedict-2.4.0-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab7bf950b8c028c481894a9128b24eac2bcc2365c2b111b3024a205c4da2e1d4 |
|
MD5 | 199a373d8a0c79fac6b065e24031ac6f |
|
BLAKE2b-256 | 36d666d1d171e20d1499e9c29bc3b5324e2bafe48c2ebb31c2f6cfa56ecbf7ad |
Hashes for hugedict-2.4.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f4cb771ee02456bcfbce8b13d18c3d8722b2808ed4cf26975bc25a738e5a4b4 |
|
MD5 | b9b356ff9c5c7da99b6202063787e155 |
|
BLAKE2b-256 | fc15778c186acd9a75a6d3b5c913ba8f1251f1bc5f726fa5cb9b71641f9561d2 |
Hashes for hugedict-2.4.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5cbfd3e0b84d1afe999f67a42157e0f4c9ee637b49efe4739323c52de5eae3a4 |
|
MD5 | 999b57365ab000ab0a577749c501d86e |
|
BLAKE2b-256 | 217b63b869a2eb42ec7012114154c3739f71744bc4f867d497d8f48ffb6be94e |
Hashes for hugedict-2.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bc14258c31f00e52df22ce3ad112b04074721e0500344e05a98a23338625a14b |
|
MD5 | e324aef05997d41ed4923fa65a2ce519 |
|
BLAKE2b-256 | e6a6febf515b2c67abd3469f93ac2e1cdfe18fbae80aae6e5d6f0f97ed4518d3 |
Hashes for hugedict-2.4.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 974c99cee1d7be059d0b8f134c4682a55d25af2e1f5d9eccc8e148fc9742507c |
|
MD5 | 4639eb22ec93d68c0ef4f17061efa1ee |
|
BLAKE2b-256 | 1c57a0827f47c8bdf04122e68a4b3af5bebf9bd99f875ea7d40fce70e244c69f |