A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.7.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fb90822a81c4a0c20a7ee18e8fb03d07339cec5ede459c9864321cd7d5f5a6e |
|
MD5 | 76b04fb3534b7963dbf14dd210b608f4 |
|
BLAKE2b-256 | 8bd6a89771492830fac8ec296e3b2ec61c3e315f55176bf799c1f81728a80756 |
Hashes for hugedict-2.7.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3bb8ee4ded5dc84df26eaac3d475d503d9b9f88c15edb483b3063dbabc896175 |
|
MD5 | 761034ae846b1a321e137307baa41995 |
|
BLAKE2b-256 | c0bba26b8d9a1e33344a12c279f1246203d85776b8dd3ae400ab2a313a2e5e52 |
Hashes for hugedict-2.7.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4d736ab14ce8552401b9dc09e39e9d5a7cdd41d1b376f165f0fd6f51d5013996 |
|
MD5 | 40db2415e915f73bb1e3c9ea5b705097 |
|
BLAKE2b-256 | 55be9208e3da344df9cfd38884999a07673d7a7c62142d85e45fa39be0a24ac4 |
Hashes for hugedict-2.7.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c17cadae6a8b40d40c0acf5ae17b60679ef6bd2bc0e76fac36bd365f6bbe023 |
|
MD5 | 1eb2231a9d42c0f5f55c08dd9a9466f6 |
|
BLAKE2b-256 | eca7e1afd1e174fee23a001dc4c8b10cb6257d68445b5c787e1fe55eab362353 |
Hashes for hugedict-2.7.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 78fb2f652a65cc16dc3009fd151f06097ee9c27c88c7ffc53aa1bff56a653f59 |
|
MD5 | a2b75e537ea23c33697757377c98643e |
|
BLAKE2b-256 | abd6a8b175bf87148f91ce7deb8703f86c77195f1768fe5ad4639ceddc9f3353 |
Hashes for hugedict-2.7.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8730d19e4351d463412b1dc65c68bf8c92887a8173ce90c4e7f5cec8919acb73 |
|
MD5 | d0585222ba8b391176825c60181d7d14 |
|
BLAKE2b-256 | e5526837406bb56d6b63c458f303f346320384a026ebd384a4eba1d1349fc6ec |
Hashes for hugedict-2.7.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5aee42f37b62c7d814f8c0fc2babeb3762fa860896618a7a2486197efc6aaff9 |
|
MD5 | 5d6da5344e7fa5bfb2b30b3cc5d77a80 |
|
BLAKE2b-256 | dcacd00f204a0e2853546515ae33eb28ee1d8593c45b2ef5cdbbdad1fa309e05 |
Hashes for hugedict-2.7.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 34c0772e72cce03f87670fc36d5c8b4bb4879584e03dc88fd24b009c782c577b |
|
MD5 | 301873a965483c5046097c35e5bdbefc |
|
BLAKE2b-256 | d213d0b6d49e8cf493dd0e3b9f7660e7c8a41f985a6019a6b3b85ec7e38f24db |
Hashes for hugedict-2.7.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd0624f46d7d1317717f30b40abb29bd0ce46d3ece90fcb83e91138c8974e496 |
|
MD5 | 3135b2d33cae14bf5fdf54a64a163c41 |
|
BLAKE2b-256 | cc485944179150d498216f14b3465bc913ee5f1662255481ec20b71ecf2a008e |
Hashes for hugedict-2.7.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 84451657058c14cb3e3331cb2151a0eb310ddd8d3cc8ed7ba3f0eb6b8b00a009 |
|
MD5 | 05fc94e1b1f32a326afdd72505fce4a5 |
|
BLAKE2b-256 | 425d33ff3020619a30473709907c5b79933daa77e4e5cd4c4d5832ef9c817790 |
Hashes for hugedict-2.7.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8387ce02520b0538c14d654098d73cd67521a753e18baa4c386330b0f49dffc4 |
|
MD5 | 5d699da7b26b46260a55eec088651269 |
|
BLAKE2b-256 | 2c4293fd1d90d8fabd90e6c2dfb9e3b1267a2e6c9a4682554a46898600ed4bbe |
Hashes for hugedict-2.7.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e9b2612d349e7d20458be6d1563cd4b20cac07213dae40ac9f270c51e98c1f8c |
|
MD5 | c73a886dfb801e5eea7697517a75d138 |
|
BLAKE2b-256 | cb7dc098170c6aeaa5375cad5c777f77d29d4c92ccaaa3f682fdb29d09bfa7df |