A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
![Documentation](https://pypi-camo.freetls.fastly.net/cb2a3c3438551102399986f02b7e576247103cac/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f68756765646963742f62616467652f3f76657273696f6e3d6c6174657374267374796c653d666c6174)
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.11.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bfca51ace200677ee1ad83b2b9e81bb6d035e931d89aee96ceea2cc0881f8e89 |
|
MD5 | fee069396fd3cfada4b6c13ac72ff6b7 |
|
BLAKE2b-256 | c632f2624b3fad706da2a4951fff4cf0be95c21035dca1ac7c756476e9146fb4 |
Hashes for hugedict-2.11.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2b99148afe1b9ff060ed7ce9700ef0bf0c6e82f9c4b55b37039426616c89e790 |
|
MD5 | 0161129ee6a53f6b036d06fa0467e58f |
|
BLAKE2b-256 | a7adce1fa534f289e8534645a748a817170a9118874d7ea98c49eb301e4f6334 |
Hashes for hugedict-2.11.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 23357016b8e721cd55d9fd503e22b27a673defcad5d10ccda8099dc7f554419e |
|
MD5 | 2e73e9440e3151644ff9439d82879e1d |
|
BLAKE2b-256 | be07ce94423fa8539a92c5c9d5cfe480b199135dd17be53dd93eaafe6a64ed08 |
Hashes for hugedict-2.11.0-cp311-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ac156e5e3631cc4ecc6fc48eb13a3e09367e9f51914f1fdc94e1d00478c4a98 |
|
MD5 | dd089d8a7c2b83e2ff49191a3857e2df |
|
BLAKE2b-256 | b5392a7b2b798f70247861af684b6cd19ea3bd7f8715d08e61837c2d4b022697 |
Hashes for hugedict-2.11.0-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | caafa9103e22990b0f483564c445aa65f54fc987b6601b4055b9b0f8b19e2d4b |
|
MD5 | d3987a69007c7353c56b7a8ff3eb9499 |
|
BLAKE2b-256 | 252fbfde3094e072e1cba841c791510583280e2d87dbcd1abb39c6fb7dae323f |
Hashes for hugedict-2.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd97314656d86aa6c4cf4469b7cfd10cd04f1e8d9ac3ce2d81df596524daa613 |
|
MD5 | a9c24c60bd6f02a0633100966555fd8d |
|
BLAKE2b-256 | fcfa4b4967cf7078dd81f0625060e88a7a6c634ea220b87a95115fb9c6dcce34 |
Hashes for hugedict-2.11.0-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7be844ad506634accb5850a28453a33b544d9fc5b09bcc2a5a87592143d2b034 |
|
MD5 | 616763851c7203db68bebdb1137a65a7 |
|
BLAKE2b-256 | 7f09eb519f753377903e868de8fc3e59059fb7d75480c4676f35480da8a3fe78 |
Hashes for hugedict-2.11.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 37a676f62fd5759baabeb68884c1e02c1958cab881b6b2371907563b842b078b |
|
MD5 | de8dcfaffd8fdd9fb0d4969153ef64fe |
|
BLAKE2b-256 | e54efca09d9b0b3f1d3e1cec93ed6899b51f1b37814ccea8f72851296a30a747 |
Hashes for hugedict-2.11.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c940fff006dbfcc7e69bb582aa3ced811e6c71dc6a2d670c9158d3ae0a11015b |
|
MD5 | e11e075583b9d0712cd73c036b7b719a |
|
BLAKE2b-256 | c70524727474ac01bdb8778501f6e32eb45fa6777cc88a1ef80331e2ffa32e61 |
Hashes for hugedict-2.11.0-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 52cc67f95678420ac101304743c3177e7898325bfc1b7a270e078329d37f49a2 |
|
MD5 | 2f744e65eee7203ede2086e79990c0ef |
|
BLAKE2b-256 | a6dbe6515386f3b7a5f80d43f29a175a5dff872735629ad6c311e71eee3a89b2 |
Hashes for hugedict-2.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cda9d8f2d4799b127d63c6b73dfe678331d0ca97a1bc1255cec6164c550aa2a7 |
|
MD5 | ab7395a3511143f0957c3c263d5e84f3 |
|
BLAKE2b-256 | fefbcd7225888904770a3e7f9aba9b20f84148b1158e104e7a7e32b833ec5999 |
Hashes for hugedict-2.11.0-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8a71e774403e242c03591d03b8b346555082fdaa30812619fe1a7e8d7ea38a7a |
|
MD5 | 7f1cd422e03600939c187c8e1c98d783 |
|
BLAKE2b-256 | 0686e07c74edf670f9df40ed3ba190d6335138ac00d650fe96789f782adcf5ef |
Hashes for hugedict-2.11.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d262185dfc3e9a13354387864053afeacac900707dd9d50237959f0dde3ccf59 |
|
MD5 | ef168c622c7503b5d4c7a264941d18c7 |
|
BLAKE2b-256 | d8d11d1d8b2a63c847ac367bbd1967610c9fbdcf745894d4338ed0d01b9db8cd |
Hashes for hugedict-2.11.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0897a3aa40abca85abe919ed3b354d82ca496b10f3a00622ef26a095430dde7 |
|
MD5 | ef97bca6670968e2c142299ca2f7e5de |
|
BLAKE2b-256 | 69689b97f422ed124a1777c50c1944603ed635870579aed11dae1c8d4eed44b9 |
Hashes for hugedict-2.11.0-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b444e1703a13eb08e9f6fe5a4dbe12e5aee695a22a3b8f92bd9e6c7e83945bc8 |
|
MD5 | 029d99721b9ca189a2ca96f070cc5029 |
|
BLAKE2b-256 | 0a90cedacc4a835eb5c8c77652d922985ccd86d7bae4a8b351b0c3f3bd0e125d |
Hashes for hugedict-2.11.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ad96bb50eb134dac5b965deb05a373bdf0943a7c52813d0fbbcc23e020d986a3 |
|
MD5 | 7ed4bf364f2c1145a61bc8ad2ab7f2d8 |
|
BLAKE2b-256 | 23a8a774225943fe6c7eec09d49b1b4ee5ba2df4bdd15159fbf848a1dbff859b |
Hashes for hugedict-2.11.0-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 333d6f3826502b33a37c680541b120cda953ee5339321f9b21c5aa774145bf43 |
|
MD5 | 24facef3189d1076d30a448bd3b956c0 |
|
BLAKE2b-256 | 3ccea47337802a80366dac3960386433babfd4af409b745a03adde290234f366 |
Hashes for hugedict-2.11.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c461a725e4c59cba6499b941bcbb30a347550a134fc8ec383ad58af4c3e2639 |
|
MD5 | dbe38c98894651578a0db2131c44e91a |
|
BLAKE2b-256 | b50720dfdbc15d73b344ac5e006d6d296521ff01212576c7d2b499ad1041ab35 |
Hashes for hugedict-2.11.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b1b79c53ab4117c60f4643347e5b09b0fede153b3b7e56d827d1c71db3c74bf |
|
MD5 | b59fa26649eed7ad4057c2cbd508b435 |
|
BLAKE2b-256 | 291e5129e96ff3ed14efa306168a404797a12bff5be649bcd8dca6007fba33dc |
Hashes for hugedict-2.11.0-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a33ec9cda6f21218b8bfbd1a86efd73413e3afbd607a392fa24186fc0805b8d4 |
|
MD5 | b048baacfb1aa48be0e496f09bd56ec1 |
|
BLAKE2b-256 | 8605879d697bcaef8e836f60ed8b6118fbc69b8faba81c5c81337b3fb4a0bc50 |
Hashes for hugedict-2.11.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 661560e6e037633c870ec3ccb923debc3f816c98cef2ef40544c9618f63de6db |
|
MD5 | 921b3be6ff2293177eac3cf993f4e723 |
|
BLAKE2b-256 | 9f576cf00038b2f8049983f1992f4bc322c342ca427aa2a84486ac03e5bc7d86 |
Hashes for hugedict-2.11.0-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 720f711a3fe4d210a130bb0bdcc63a92717dacace4318e5713768c06d4a9e234 |
|
MD5 | d09b283b9118378d5c9eaac7881c694f |
|
BLAKE2b-256 | 64df20f442edd52131362df6faf941e6873cb213360246dd4df54726b548a3ee |
Hashes for hugedict-2.11.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77592a1201a915dd6094b02793d108ba678e028aa85940998e98837a2e927ccd |
|
MD5 | ab7a23b50c8c02e2fd2d8627f2ef6064 |
|
BLAKE2b-256 | 80ee9727bd235b4fdd8b7f8a49510f47cd440669292c7e5a2d37474aaeeb69b1 |