A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.8.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e27009224ed133b853bf79010ceadb42f9bf092e317c105db75ae0080619b99a |
|
MD5 | 54b3ff1418435b7641658fb82e3c514b |
|
BLAKE2b-256 | 30f4ec4f8fdc5a75aba30e31f3a4e566dc4ecb51f1f78569c1fa216b3b543e1c |
Hashes for hugedict-2.8.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 65edf8335896d87830d3e96ac65f0c860f25ad9caeaba5c3213a63352aa2948a |
|
MD5 | aa7323cedf4d646426a32e28efb2aa7c |
|
BLAKE2b-256 | 50dea8c7e00f6048492f8226a13ff9cc44cb7681f38de5869f960e1b5f68d691 |
Hashes for hugedict-2.8.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ae3c4ff416a05c5b7932964eec0b0fd140500cfc508b3a10c9df98500ea50d5d |
|
MD5 | 237a046ec14f80444493132c158fa91d |
|
BLAKE2b-256 | 6e087f21f3a7cfbc51da18ad40e0d6447648a259871d5f256a00274233caaf91 |
Hashes for hugedict-2.8.1-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1b922622b5e1028082aa908eda494864ecb86ce75c17bb0c316e0987e77db903 |
|
MD5 | 9d48fb7221fb96014beaceb42417a326 |
|
BLAKE2b-256 | 59d5371a7ca00523086fd94316931b08d70959abef8e3d190e88987105021d22 |
Hashes for hugedict-2.8.1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 92c094c0ef35082fafc70c1f90727d9fba677445ecc6a1dd182af0fb5c5cf1e3 |
|
MD5 | a316838496a0ca215cafabb7cea59643 |
|
BLAKE2b-256 | 33dcae2ccb2f6eaada8beb2244cb8e3f3869c7622585cfedcbe99247cec9b32a |
Hashes for hugedict-2.8.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7ade847a66520f3292c97b0925f5c7f5a1ef20f026f1f7d5d458fd643c4ffb5 |
|
MD5 | 8fd428ef2dc0747f6ba9097563cdf3f1 |
|
BLAKE2b-256 | fa961379db0d3815c8be25bb504e3243b9781df5140facd9b3efb4559249f3eb |
Hashes for hugedict-2.8.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 669139b4f0e96bba6d3447e29bcac686084b0b792fe82d1dc5bdbca638cb4e0c |
|
MD5 | fac13d24f79666fa43c82b97edaaa460 |
|
BLAKE2b-256 | 85dcc59f8b86fe7e9faaff2c04c3ce06c110f5da4492c8fdf603dbe58bec8543 |
Hashes for hugedict-2.8.1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8bec82bce567e60b71fd916fea1c9bc12812f95d1a6306b634cf9fcfe65c42f7 |
|
MD5 | 56d2931eaa07ee8ef9424f1c090d74ba |
|
BLAKE2b-256 | 0668d6faf212d986b15ef8de312f1387c4bf64765128c539d97331ac4aa2715c |
Hashes for hugedict-2.8.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 625faf5747af8950d77cbba3b30a2436be410f4902218925457a57ce6abcc0f5 |
|
MD5 | 4a02a45b0195a9f307df2b8155b46c45 |
|
BLAKE2b-256 | dbc6b9e3b0d80ae12f8ef624bc4e93f4c775c072c169d8112be0c876f7c2d961 |
Hashes for hugedict-2.8.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77f75b1519421d589e1d92673cf2385c8ad03b17855fb3dfa1c01a67894889ac |
|
MD5 | 8646617964c74082c625ccd8ea771fd1 |
|
BLAKE2b-256 | e08a34f915a8e6e4b999f4470e3ec3291667944e0810702c1a199fd2c67e77d7 |
Hashes for hugedict-2.8.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9e70ec02313c6afb7232bd053a45643c8755a7cfa21e2d3cc794f2714bcfb51 |
|
MD5 | 2d7e23fc0a7bcb43c0ca4c9c973f6778 |
|
BLAKE2b-256 | 45c3cde2dd235758d3326e030d423f7fca06b3ef5ecd4ee79a4b29741a7cd48a |
Hashes for hugedict-2.8.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5cd282dc1498d22dd7180c8cfc51347ab6af3b3bbcb6e41e5db52a969c33dbf2 |
|
MD5 | 987438aefdfd60768160239a070cc826 |
|
BLAKE2b-256 | 764bc2ba070331985fa928d457e169192efc3be36acda822903c2cafeb5664ca |
Hashes for hugedict-2.8.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c69df679aa2104d2a6d2b21b62f1b560d558754706843a12bdc676155229df56 |
|
MD5 | 4f37aa125be4929bbf3a8b5cc1d8a78a |
|
BLAKE2b-256 | c56f1b032a01f3e1d4afc8776e0553f83a9fdfd93bf42fb930ee6396dcf70c9e |