A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
![Documentation](https://pypi-camo.freetls.fastly.net/cb2a3c3438551102399986f02b7e576247103cac/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f68756765646963742f62616467652f3f76657273696f6e3d6c6174657374267374796c653d666c6174)
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.9.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 51747e5aa97049427a04772d9490cd453695aa9cf42d6fcf07cee3e1953f0dbc |
|
MD5 | 5f91c24c58cab851248496588f954b32 |
|
BLAKE2b-256 | 92614109a9c0c7bb01eefb965b5b386abacc7d6a554e4c79c6de0bdcb1d32d8a |
Hashes for hugedict-2.9.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 09a2b06354ea7051805175f864a39bf6c95ed58702d2828b6183e8e128172034 |
|
MD5 | 668d6924766cba0fc862d5c4191549a9 |
|
BLAKE2b-256 | aea2d6bba3bd27c9670c6f2bf505727cec8ddcf90fbb80e730c56f0c255557af |
Hashes for hugedict-2.9.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3f1849edffe9f00947c09aa9c4673fb9992207aa8df504f756d67c78b18e5cf |
|
MD5 | 421bd1ef27355cf5de0d2502db1b597e |
|
BLAKE2b-256 | 807807c19b715a2f2e4177223412a3c7ef973882801e0d0bc0a65902b5c5e61e |
Hashes for hugedict-2.9.2-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c65abfc9395db399d9aae77426cc4305c18bea71d3b13dabbfbcb206cfd83e4a |
|
MD5 | c2635124c18ad417d60fe4fbee17d919 |
|
BLAKE2b-256 | fb9a0da92869f3afdabe6e9714d1436431c01fc71d6d15ef8bb813d672ae07dc |
Hashes for hugedict-2.9.2-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24fe9c3deeb11a9f448672816f3453eb254c948d2b2b186f4062790267ad939a |
|
MD5 | 30c8ef96b93277bfd095489696c70d86 |
|
BLAKE2b-256 | 77ba21b61edbf60e9341523c5c000bf40ef32abc4d962271d836b3fc417970b0 |
Hashes for hugedict-2.9.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 55ade8fcbabb541aab9e6ce078894eb819165a41ef7620e4cac187a6d3e59fd4 |
|
MD5 | 05da03c2129e5bafe234e99b0013e1b5 |
|
BLAKE2b-256 | ca99fd9522fc7ba124d09b3703f695ed9ddf36e6505c37113a6d013c27382119 |
Hashes for hugedict-2.9.2-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 24af5dbf97a5ed37d82cbc32aa2a3df5f0c5eb8263edbca6994871694820e2d2 |
|
MD5 | 06baecd9a5a288801c6c34cbe3d0b6e7 |
|
BLAKE2b-256 | baf171a295508f4b067b864db983968a3ef8247be57089179cf7768e09ec48dd |
Hashes for hugedict-2.9.2-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 384a112324f68a468b887d2e5bb17fd7103f382034ebd28cf2631357bee94750 |
|
MD5 | fca6665fd35396fc8ab3bb628ad902ae |
|
BLAKE2b-256 | de94f4d0172253ac244251adb74e455c592b2589b82db4209a95b4b73f6a27c6 |
Hashes for hugedict-2.9.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8ac2b9470667f5dd51c87c409b0c826201b9794970799a41732cf411baeaa565 |
|
MD5 | 50ff4db6ec57138032da510561dce9b1 |
|
BLAKE2b-256 | e299d156798028977e4c333dacf329894c4cd97e8b3e592ef013929784b0d90d |
Hashes for hugedict-2.9.2-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6664b429a468610b9334646ffda48f0695ae6d198db362627cad06daf13b9bd6 |
|
MD5 | 1337ee83c3f5b3a5f000af5e073c7995 |
|
BLAKE2b-256 | 182616a48ed8b6576e872576a28740c4ca609f0a50ae82f222b5e8d4c5cb1bbf |
Hashes for hugedict-2.9.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c123bbad4cbcbab268226d576119d94ac00d2182fbd57d79bc266c336896238b |
|
MD5 | 1eb3de7d98f0aa41dd573d42c030d7fd |
|
BLAKE2b-256 | b0feba1edc15d94e773d9ef0a05ad4d1a7bb590b2c91886600d0238bd5dbe5c9 |
Hashes for hugedict-2.9.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | adc1d192c2991b2e084c32e16163e829510ad74864f68edcc63cf3a2972b28f7 |
|
MD5 | af5aa152cc585d5dc10a8671f9234569 |
|
BLAKE2b-256 | 163e4c54ef33c9a5a8f628670908f7c6ddb078b36b78e1b084ab6a7fbaf7ce37 |
Hashes for hugedict-2.9.2-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a4b1709a9119bdaedf1fb6a688a416625972951580fd7435231b19f55e8aecb0 |
|
MD5 | a88d3da147f3ef753ff8590c41b6deaa |
|
BLAKE2b-256 | 2ba93c122b3e3761e0137a4232cc75d228cd2a23890d5c835aaf791aeac56c9b |