A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 66f0f1855365d91e269dd55cecb63e38c58bd2574f06a5817fa53adfaa68d759 |
|
MD5 | dd615dc04847844bad8f224d2e58617c |
|
BLAKE2b-256 | 7e3d463a9af85a1015999c2bd42bccbbc037552d9b998240063cc5798532958b |
Hashes for hugedict-2.12.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 079b63981e6e105ace96fa4737d59c061cd3c1ac9b6f0fa7ad27a2a505dcac63 |
|
MD5 | 5e215c493b675bfd7b13bde6b24db9a4 |
|
BLAKE2b-256 | 34fe02cdc1c82e618f0cac7bdf9fb160e1b037f84b3e3a8c76f2b8a1f80459b9 |
Hashes for hugedict-2.12.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 89ba44361fd538ebb4e9521f4d2a19c9341a4c2db06f46383acae6ce9a7e8d9b |
|
MD5 | a0017f1bc1c4221901a589249278365a |
|
BLAKE2b-256 | 65c1fb80489173c631bb985e4f6c5d5f7f3ed9dd8e88c77833abef8aa6931e4b |
Hashes for hugedict-2.12.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fe67ce1ac7685a7bfca11f1bdc5e041f29fdc2fde43dd470b429b24c446500d |
|
MD5 | b13074eaa4bd48c887bc7a66e910d102 |
|
BLAKE2b-256 | 3cb81804ebda836b0d63d975fa7c1637bad24b4346400acccb497eee3a975efe |
Hashes for hugedict-2.12.1-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d36c8c0c031fd854affb1c0b8150f6a9ad9cbf841b7dface0b3e5c5c427a073 |
|
MD5 | 4b28b5dedb255483f7a9827e896d5dbf |
|
BLAKE2b-256 | 37429e250ec26c6e92649a61e59eb0d6f8802a582b6b18ce728da6cbd40a410c |
Hashes for hugedict-2.12.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 90a3c39f1a9e66ce8fe92eebd5b8749652027329d8625597f905e2f357e83c35 |
|
MD5 | e157335b12d7af0dfb9443db6c65ee55 |
|
BLAKE2b-256 | 61372292f4d5ece27af3b291132be34ca2ccdce7d26bc1523573057e05d8fb8c |
Hashes for hugedict-2.12.1-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd5cf1194722ca0d93391ad8f4e6e69bb0a05482a0bdc7b2dba3614f056dbe48 |
|
MD5 | b3e43b3c138885ab113ee47754809f4c |
|
BLAKE2b-256 | 8884f86effa87d1004d4e2645cb6e2590cc10b5f43a46334c70d8147a5641fd4 |
Hashes for hugedict-2.12.1-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7e66fd53a486f1f912bbf328f0b1a63540b445ff7a5b98bc06069c0d48e9d2e |
|
MD5 | e4d0d0f805cd5b17b653fa91815f2192 |
|
BLAKE2b-256 | 58d739468ec5dd86ec6dbd559b2544e49914a7b400292d0a858df96966a5a2e7 |
Hashes for hugedict-2.12.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2ac4022a731f661414741300c216caa8ca85b4f79a5ca34128c7b0a6205994bb |
|
MD5 | 4613de8ca5e10cc6303fe8e97f14d42c |
|
BLAKE2b-256 | 5e6141e01d6255c29fd08fcc86070ed167bf2908e4d11be29c29e09483f81977 |
Hashes for hugedict-2.12.1-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e8707dc5c9acc5283716bedf41ba77580e5775a6581253d15f36f2054c1991f4 |
|
MD5 | 2ab6c082ef8d8beeab84fb4e559b0582 |
|
BLAKE2b-256 | 2646d1e9cb41376c907e31d3c984e87e6915d6f674601e51e4f7e57493bd2da8 |
Hashes for hugedict-2.12.1-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1a209fb402ccfba1c0bd4999ff516c596e900636946fca9ebe2cc02d7ec939d2 |
|
MD5 | e29f1135d83053de7b0561c305ba8b92 |
|
BLAKE2b-256 | 3e4a370d536f94ddb8d91cf4cc840ee2d2141acf65ac75872fe81f1780d75043 |
Hashes for hugedict-2.12.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 44673d1193cbf1e3f00d1069a0636ed2c0e98cc2f5e5c165043aa974c4cc0ea6 |
|
MD5 | c7ce17d9c99914d461fe0144d756f780 |
|
BLAKE2b-256 | f0dc68bd422c8712924f6790b9b7ca8b080717aa53296555fe568beb113b05be |
Hashes for hugedict-2.12.1-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20d4def9bee178f86a239f6ef531590981a4c2d203b0173aa13b06d66cd4ffd2 |
|
MD5 | c03e35e0014a86bd2570ca70adce5272 |
|
BLAKE2b-256 | 63b3dcf210d67e90aace2287bb9e6af7bbb5fd133d7a669ae89bd1a92f5bed2d |
Hashes for hugedict-2.12.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ea58959e8cc349c33d75702d5f1d40b48c1df64474af29cc3544aa1c5f4dd419 |
|
MD5 | 22d56004954471f9acc479aa63ca9e47 |
|
BLAKE2b-256 | 0c874a938f03a0f680933dc1e54a980885fe08056e87829ef960d76fee9875d6 |
Hashes for hugedict-2.12.1-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6abb7ec28b57c97cb5b243a18d2b6882c96240207b9f45eb81463936270ce860 |
|
MD5 | 6563b941c56b477eec947bdb0b2371f3 |
|
BLAKE2b-256 | df1a96c12b99040debcb2a6f5dce29acd3717ca81d54551e37ed2bbc067cb303 |
Hashes for hugedict-2.12.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5d92a699ddefb9c096ca6760d382cfa812e940b700ea12beef77bcab876508f6 |
|
MD5 | fd078db87ae89f0fad71b341630ee059 |
|
BLAKE2b-256 | f744ac724eecefa5ff22330538136818ff60919f57705cbbc9a2aabf13a6d2ae |
Hashes for hugedict-2.12.1-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e4541cff29d8504277364388023b9e87ce60a96588c04206f935b9001e6ebc4e |
|
MD5 | a19e35d79a7a5401bf8b7f71b49eae83 |
|
BLAKE2b-256 | 3d8fdac7cd09bc352bdde475dfbbdc59c39c671ad1bc036eef7ab3e3a6b0eed7 |