A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.7.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1b545ef9395d5a345b29c842fe24ef1e6bd13bb1f52dd5f529541c0dcc616797 |
|
MD5 | 69e877ab83b33c1367206c4e32eed116 |
|
BLAKE2b-256 | a6098dfe3d2ff66fbb788a7c59802108f2bff816c45b107653d7055e9a8dfeef |
Hashes for hugedict-2.7.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 73724af764249ab6becfeac4766704af8a8af80ee397638efb4ec1f9a1c38acb |
|
MD5 | 365d61747cf7fbb7967a6bf8eca6bbd6 |
|
BLAKE2b-256 | 86f58cd9550527a2e404851395f5ae9d653de698140786956c0323a524541829 |
Hashes for hugedict-2.7.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb950fac475dc302379a19f920429d656fb9000914d2621d4eb37af2f9a0193a |
|
MD5 | 60b36bab5d3fbb636b423d975f5f7171 |
|
BLAKE2b-256 | 82bf02d929c5be7e3a1f9ed0fea41529023ee4f24b2a95a74d50c37bbce8280f |
Hashes for hugedict-2.7.1-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | be200b684c5ed6fd4a1da08366ea3231d3e1ac76e42201582fbdb360a24ece5a |
|
MD5 | 886f7254656c900be49920fffcc89890 |
|
BLAKE2b-256 | 06678be07ba2ed8193f77b9bea4ef24f8b853cf8831336ee817a1af3dcb80d65 |
Hashes for hugedict-2.7.1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a124c7cb3f3491e9493025be093ff3532476334814c1dc5e68412b79e23f9582 |
|
MD5 | 3db2aec5cab899e3aff1084c71084dbe |
|
BLAKE2b-256 | a95cfcd3c2a608729f4d7a8b661ab31ba3e810ab821ca46970b11ff9fc84d84c |
Hashes for hugedict-2.7.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | af68788ee6be87b66efbffcbf65ac00f639fd534e838f839a51a31dda30db49d |
|
MD5 | e20f86539864fd4301a5e8b0964a7064 |
|
BLAKE2b-256 | 956408fb9fc8a619a802cf0c0da1080db5f3bb1968fb019580d8f2933ec9d8e8 |
Hashes for hugedict-2.7.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7a116b4527943cf717a873c9ffe641efe490647ea217d83cdc804bb6bed5662a |
|
MD5 | df9a971ab1bff65b4587ab5425c7c9a6 |
|
BLAKE2b-256 | 9a1438122ea529ef77271b2279de50d3821f9956a2b8c73570326adca57578c7 |
Hashes for hugedict-2.7.1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d0ceb09df516cb2fbd98d558f40e3eb5b5b27c6426a760c4e439d36d29cda942 |
|
MD5 | 40539a8039ea00f531fd9daed25b798b |
|
BLAKE2b-256 | 0c3ce9a1d3e532245e41bc3c1429eaddb55248c5134dc30f14c1f4b5d5217fb3 |
Hashes for hugedict-2.7.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0cacc4e8299febd64d3bc434dd7db7673051c4cdce6ef343f2422f3fae7ffe22 |
|
MD5 | 5f8d95d54b9b8684c30f6907a9c2caec |
|
BLAKE2b-256 | a8a554063d5e3f37f700f66d4f4d6519a5372978d18edce3cd8dc753836563aa |
Hashes for hugedict-2.7.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1f75f09f0fc5a579ad4c218f825d8d5c49061111574aab1818e1de6aeb7facba |
|
MD5 | 2ff69b50fcf224cde851795c4edc6e19 |
|
BLAKE2b-256 | 3783a4715a898e628aebb9f4b0d42809329a1b113b468593566653240d1b4865 |
Hashes for hugedict-2.7.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ea3cbb104ac2d7ddfe98c0be62325e8888bc77c15d80c60ea0ed8a7971bbc6d |
|
MD5 | 96766a698395857b06baf02ff1848d1b |
|
BLAKE2b-256 | 1918068ca49a6831cfed036818de7c6c892ff310ccedb66030dd03b415da5860 |
Hashes for hugedict-2.7.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a44eb37b95647809a33f24fa74649d6ba0d03132bd764a12c6e66cd770dd3782 |
|
MD5 | 16582c3a0d3bf2b050aecf15c71ae005 |
|
BLAKE2b-256 | 7e6c6bc2c36fbcd6b74266b9d1f3c6dd691a85c43d36d475e6db44a65580cc76 |
Hashes for hugedict-2.7.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 657e6f464028bdcefac2be5913ebe200ccc32d561442d92796bd425ed5c75098 |
|
MD5 | e6875cceaccb83db9fd0a132f5d537ce |
|
BLAKE2b-256 | b202425d216e4c7fb2dab6adc13b08ebf763dc88f57a9696f5bc1056be5108b8 |