A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.4.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1cff264031439dfa0b6a867bfe3fbcd20da4bc139d10d8b12e7edcad4bc3f5f7 |
|
MD5 | 1534d54e902ae2c6b105fd0fcd55da1b |
|
BLAKE2b-256 | 69808881af23abe7f69b90a0c3fca253ab01d3de2a0d0069f15e090e64800fa1 |
Hashes for hugedict-2.4.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a5eded9c12e26c0ed1f3757896dcf65267084ac19a55554df1dcb2496142807 |
|
MD5 | 6a45c7bd205995536a6ac7405227105a |
|
BLAKE2b-256 | b609c0300dc241b8f67ae476f97e9bb39e6d2d398b56f7d3011d2f6080031931 |
Hashes for hugedict-2.4.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8fd48ed865f2d882bd6067a41ea96090468b7411c057519c2ba6a3b300238416 |
|
MD5 | 788bda2d7b976a3450e971ff9d8d89ff |
|
BLAKE2b-256 | 106a43bcfb6930dff32e612b2d6d20e088ac5cbb45d2122708d756b4397ddc35 |
Hashes for hugedict-2.4.1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | efa6117ddf0b05731357643cbffd2659fdaa8b57af4a6c20e92eed6924f8b826 |
|
MD5 | cfcea64c58197ad0adc707c0cfc0d83f |
|
BLAKE2b-256 | 1804396a89f5572cff3a837a037aab2775e67d80e53771ca0835e54addd8be09 |
Hashes for hugedict-2.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 22b19753f5d8df1b14df84b4489498639c8e5d1f3a52daa215767b4604e7302c |
|
MD5 | bbb118c84dfaa4e3b85ad35a57e72e89 |
|
BLAKE2b-256 | 35e5ec7c7dbae5136e134f88ca2308deb2366bd24701bd3130f34b03a57e8706 |
Hashes for hugedict-2.4.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c5c3fb0bf885d296f3567ce2129cd8279c4c9d7ae24482efd9b058a2d5165897 |
|
MD5 | 8bbe4e48b97a9ba4923c1a6c3e13f4ba |
|
BLAKE2b-256 | 6e746c70662ccdfedf981fcf502e04ba33985f9199c80d571efc3d6d3e4f25e2 |
Hashes for hugedict-2.4.1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8d7220591517e20ca6d02eac10263240ec5c8f9551b4ee8be647bee8c3bbc455 |
|
MD5 | 3af6c828452f0705ddf7bedd93858adb |
|
BLAKE2b-256 | b29825b19872d71eef7db40d9fbacb9e1cb6be5cc2673c7db922785f1d031237 |
Hashes for hugedict-2.4.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87d20805883ebf0412d3cba7541a0b1cc0b63353c1c6b220a4e5b6f513035732 |
|
MD5 | 3ee2b374251108d05f616bb4b1148232 |
|
BLAKE2b-256 | b4e728932e5246e2f915eb4639a3062571525fea98f0d1ec041f9db5f298878b |
Hashes for hugedict-2.4.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3281f4bd051aac4c22f31d0878d6387f936e1085d625b2a2b4318add22c11196 |
|
MD5 | 27b0ed8cb6ae1197454b33357238f010 |
|
BLAKE2b-256 | 31f468e419b0b79d429edf628527e03cf58c88bfee9168c995303ccffbd36ac2 |
Hashes for hugedict-2.4.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ce0459bf566f432c56c2361d5ee8c3aba0f794e74a9a7bc809dc8edd69901075 |
|
MD5 | dcaf4729600c8a04383a7ac9bdf71612 |
|
BLAKE2b-256 | 781d5f67385a88e24fa1de8b5ff9120bd4c459b3fdb2bd7946918118d6d4967a |
Hashes for hugedict-2.4.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | edbbb43a2e4dcb35216d5ce8646641707749e6b7853802863670c3ae01bcd33c |
|
MD5 | 88f42f39cdfadbcbde0ed425e2068160 |
|
BLAKE2b-256 | 82016fb9158701f98ea6c6eaab8d2dc024707116ae651a5eb31d5efb35c676ca |
Hashes for hugedict-2.4.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76ef1703662ef17b0b2bf1231c203b2705d29dbd9dcfef41cde5bd6d41ab01cb |
|
MD5 | 1c00302fe8fd8baed4e5def27f0d57f7 |
|
BLAKE2b-256 | 7ad7362bd6b28496a41f5170aa92edb1edb489993e1e4b1e69686f15b4f5ec2b |