A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.6.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b47403dd789e1b2d8c44c118e044dae8363381729c0180a460824f4a2091a211 |
|
MD5 | fdbdc2191f88faff028b76f8dd594aee |
|
BLAKE2b-256 | c75599dca243783bbc93e0db82741bae77c78f0faa5e96dc2f0e47092f63e4bd |
Hashes for hugedict-2.6.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cce5e51f1477a6a03b5ef8d26bbcfc6330d633107ed6d9e9ca8b8ff21eff61c8 |
|
MD5 | 495e31a46fbf3efbf92580b0c7360f79 |
|
BLAKE2b-256 | 0d6f1dae493e6af2b0e89a7c14fbc74268eccb24ce359b90c6953aa555b68145 |
Hashes for hugedict-2.6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d132d2ea139a7dcdbeb06652ab483342eecf31217378c4ec13ee791b188f9f1e |
|
MD5 | 98d964eaa3120c37c1e40e947120f19a |
|
BLAKE2b-256 | 3dec8026ae56c12f1600f5e9316b8fb3355200da38e79b35a6df5f1d73393964 |
Hashes for hugedict-2.6.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1defe90e20dd831d2b7bd6cfbdd3762f4976b311af63637c3f5524682bcb2f8e |
|
MD5 | a245f0c56898117cc195ff119374aa98 |
|
BLAKE2b-256 | ddc269e34c8d539d93814f3d7d1f6ed0d55f45375b3bb7e757a6709e76058580 |
Hashes for hugedict-2.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e1a29a6daf1626da84243387173ca4d315a67430e6f2722cef5c9d736f08e54c |
|
MD5 | a76f702f9f9cf6ffe356ad4440b6e9b3 |
|
BLAKE2b-256 | 2a9dcf632e237a95092706f1f25a4908c3b11af87bbc67584615943270ec74c6 |
Hashes for hugedict-2.6.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4802c5467c6792c05ff519331d9d6cd66b8e94ec58c70bf141dab5ccce205290 |
|
MD5 | 8e2a9762ab412d409af2174507923a3c |
|
BLAKE2b-256 | 504d100cd7281826f3e55cc50590e3473ac5bce68beebd24b4520857caf9b946 |
Hashes for hugedict-2.6.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e69d7459217244059e57b61aaf21de70a541eb81343c31f5e80a2a9babdbc2ac |
|
MD5 | d78cb5c267ada91ef105c1e2d04a4f65 |
|
BLAKE2b-256 | df02b2676914514dadb09a139182a96b70ae8b5834ca2b8074b174208a19c268 |
Hashes for hugedict-2.6.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | caec8f20cec804653a5dc032663045fd189f79f14c524a334bcc37160b5dfb81 |
|
MD5 | 1fead7eda83f3cd24a348a17942047f8 |
|
BLAKE2b-256 | c970abe0f4211506307fc96a7543a5c36fd98b417d84a29c1b9b64aaf4a63bfd |
Hashes for hugedict-2.6.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eea1d2231adb8aa6ca5bbc14c333816d98899f6c3907fd63ee90120097e5030e |
|
MD5 | 6b50c3f6f11a82c9c301cd68e0ee12cc |
|
BLAKE2b-256 | 2b043926df83d43c5cafd617ce4eb2b285c10317de7af8668d8696f1d93f6a1b |
Hashes for hugedict-2.6.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a1753f8328d18b261a6de0f9349b15b120bd04ff073e953762210d32e1ac4ad0 |
|
MD5 | 386a586ffac35c4c3c2eeecff7db0aee |
|
BLAKE2b-256 | cc061cc0c2fad25d2529d68e51db52c5ac6defe146084416b6d12298ba21f665 |
Hashes for hugedict-2.6.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab19883b9a75cb467c3663bc5ef90678ad82591a6bcc9cb5641c26627c92b917 |
|
MD5 | 02f0511010f1ed7fae56429aec097597 |
|
BLAKE2b-256 | 169f50cc36ccb15c8925f2e0616d77674f8692a0fcd9f1e9df4035ee1b60668b |
Hashes for hugedict-2.6.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f7620c22eb864dd7facbe7b16fcdd2d383206bfeb5ac78a0ab27890cd5c640c0 |
|
MD5 | 50c978a65132a511bb4fb79a4358116d |
|
BLAKE2b-256 | e79859f8f81e27271a882bfa4279e38bd746f10e384a746c35ce565b8fd88c8b |