A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.13.3-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 499bfee9a6fc5ea25c22b87123e7786ce8669be1b39f9ecf33bc3c7656f53043 |
|
MD5 | 1b430b75cc75502c0c817e1ddbd98361 |
|
BLAKE2b-256 | cbc889b1cdc9301bdc677022dcb9ee25f413bb1e7916fe584f053d17097d1225 |
Hashes for hugedict-2.13.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3dc07e0886c09278e63f1ab35693db1fe01a3836e04dd446e01cf8e803ea16e9 |
|
MD5 | 816f08842712105aae09f0079e21ee0b |
|
BLAKE2b-256 | 62710f84de447329f759e8452642c75fe4ac06eecedbab7f5f63b1eed0150a5f |
Hashes for hugedict-2.13.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7c5ed829faa71c2957e412030975b5dfe7c3306fc6ad0aa8bedc128fe002ccd6 |
|
MD5 | 5d5ea714f58c326ce69762f8c1277de4 |
|
BLAKE2b-256 | 94302f3cc36545924105194f42e7219aff19b6c364bda3cb2b60b65527897601 |
Hashes for hugedict-2.13.3-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e0ebf0e09f35f54e17a0db7685d0fcff1b0a484d64ebfb0c0e14eccc8ece2131 |
|
MD5 | 06d760226662cceb94823b56a7721851 |
|
BLAKE2b-256 | 876dae3819a703a48088dbd577113ada1b3a7dcb916b2151a41dee1a2367660d |
Hashes for hugedict-2.13.3-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a044969c3d36f900a06d0ab5ad905ae1d898a1c7a39154b749cc93993d86f49 |
|
MD5 | b4450f065e9e211624dc1ae14ca188b4 |
|
BLAKE2b-256 | 483997d5a2c6d9f2829865aa5c78649bf515436b401f89d0ef105a880c533137 |
Hashes for hugedict-2.13.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9595edb58e32b6166575f054ef2c4ff7bf02d375876f6412f9ec0a50299e7ae6 |
|
MD5 | 00f27385a4144ced23205f4768964eba |
|
BLAKE2b-256 | 5012cfe244b444c8b7713ceafdd4dd5dc1b448d4fc41da9c156dbb11179991f9 |
Hashes for hugedict-2.13.3-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d42a1c9d19f4a19e2fb297ea4eb7149f32ddad797d80ae140ef2b641a656dea6 |
|
MD5 | bea3818219cbf82be2e9f989d3d4b884 |
|
BLAKE2b-256 | 8c54a789f133e4092ce5124698d008cd96d00598bb33f449cdc7de89f061281b |
Hashes for hugedict-2.13.3-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7a493f2c68429e4ebad4a22319a36d2290ef04a981b4c63e1174fa9b789370e |
|
MD5 | 337d6afbd188756bb026c1e78e1d6a77 |
|
BLAKE2b-256 | e8c38f09f17c3d06e908d69e28239a94c5d9f49544273fbbbf6c5b49577f9da5 |
Hashes for hugedict-2.13.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 773c13a16a477061853ef47f9b780d4b190e605284da6d4af702cb67ae532b9d |
|
MD5 | 75154c77935edb397b773e6f9993d920 |
|
BLAKE2b-256 | 178ae301d1b15bee41f966be212da1afa4a5fa99381dfa0e1a4c4f3acc1e7803 |
Hashes for hugedict-2.13.3-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c57cef424bb7a3edbe1407e6bbb0643bc43fe1e91939fb4a9349fc2997dcbcf6 |
|
MD5 | 952d289f8231f9ee763dc905d2e4a701 |
|
BLAKE2b-256 | f17bf6a9c440cc38d866545c5236b89dff992242f3960b684470e15680b2f409 |
Hashes for hugedict-2.13.3-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20ac828335850c49de17db66619120f0464f1b7a4f5aaa72287d52dcba642284 |
|
MD5 | a8df2be3514e54a916e7e830d7445a15 |
|
BLAKE2b-256 | 2e70e81e3943598a37a6df9436b2fac9342c802e9dba9104563e7b132845a4e1 |
Hashes for hugedict-2.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a90c590f2feb2af05af7c680c7ab70eee32177a5c09e5fb6c9b5a543ed32f22 |
|
MD5 | b2df6f7414682c11515443d670c60a57 |
|
BLAKE2b-256 | bae72a4391cdb10fdf739a222674bc84b3d5e95cb69e3524465e6b2196c0038a |
Hashes for hugedict-2.13.3-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9b0608e731b2440eba88b10ac78a78ca52cd99fc0ba1ad8f4cc9e854c5400cf1 |
|
MD5 | fa4de4b19bc3db87627f2aae473c82cd |
|
BLAKE2b-256 | 5a2f3551cc02a269b072811c7345d233c048393e67efacb503e5e3f505fc07e8 |
Hashes for hugedict-2.13.3-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ffb9737eb36cfc9e9852ab05aa27250785c3922d503259360983aa20609c2d5d |
|
MD5 | 71e0e348900eb71e08caebb20f048edf |
|
BLAKE2b-256 | c346f864cbbeb1ec514a3f7b185038b574bdda57501d46995b6e9b79e710a773 |
Hashes for hugedict-2.13.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 585de086a0b6d90b3df250216a94445cf75ba378868db50180e1d548f1db61f8 |
|
MD5 | 57369984cbbf8680eecd73e6a837c949 |
|
BLAKE2b-256 | 5c443855b217f5ae74c056f486878dd467d948ccf71dfebe89eaf42bde29958b |
Hashes for hugedict-2.13.3-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec5039b9690a0e4654ab8d316ac6cbe03dd3c620f0eed3ff7deaab2997e2fd08 |
|
MD5 | f76e13101a57f8202eec9458e1054d0e |
|
BLAKE2b-256 | 1dd58bf45861a40e80583a74e6ff4d650d622d1beb6ae3a32e9dab66c26372d5 |
Hashes for hugedict-2.13.3-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fbcdafc707986067c48da23d38aa23c9d056d77e401c585677a01218c3f0a9d2 |
|
MD5 | 909122ceb9beb6e9047185b62e3431b6 |
|
BLAKE2b-256 | ed9db15d147db74f56f00d91d5f12b63affc16b7f867f2d89de42f4feb88e6ff |
Hashes for hugedict-2.13.3-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b10fbb8c0607de6c8677abcda930addc4d6bb0ae10ce72791368a4027ecdd3b |
|
MD5 | 2689366e36dfe92a387df51294297a59 |
|
BLAKE2b-256 | dec25a7bb305df9ccb3e4b357560254f1f165a01325175a8dcbc560daba6d4ff |
Hashes for hugedict-2.13.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 014685160a88d7226d102f117cbc697be0d269b9788648e3e22e54c51f4e9e7b |
|
MD5 | 0728bbe5ad46c767bfae1ada0fc047d9 |
|
BLAKE2b-256 | c1e66b103182eddc807072e24448704407b1d767399c9d4fdd26574596c2207d |
Hashes for hugedict-2.13.3-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a76d7a67632486ad9700b7cec094211bd02f30ba7730b407b5fdaf8f09397cc |
|
MD5 | 15b5ada1a9b9e273523589d73c240c1c |
|
BLAKE2b-256 | 147566bc74c4b38d6459640e0348f181c378e3b0bf37baae8196e01deb8f998b |