A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.3-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 577d0d331b9ab11d1c4440cb581ef66e9af10be36c6ddfd4dcd517dc8e1502d9 |
|
MD5 | 700ea10405a560c725e02e9bcf6f8df9 |
|
BLAKE2b-256 | 355ba48501c1d763ba39942c9c3894755f68e6b0187b005a3aed1874876b6f61 |
Hashes for hugedict-2.12.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9a15cfab4e237bdd225a7eca18be744a1a35493092c669b76e7eb115bfb130e |
|
MD5 | 3b311adbad42bc5fa8393c0e0bb0b790 |
|
BLAKE2b-256 | 3360271e3a7916b725d1dd1a1de237afda9ee59eb403c32ba82e80c02fd3c7f1 |
Hashes for hugedict-2.12.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aab4262660d624bc37d02f0b0cbf4700484e2322ce4c3c131cab3ff52b218468 |
|
MD5 | 5f7d11333b0a4ea60985e4dc440bac1b |
|
BLAKE2b-256 | a4fcc162d126318d49d843f3f4d85b7392c575b1a84a5a6575cbb2263a5815c5 |
Hashes for hugedict-2.12.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9e89feecfe688bedea7a5c03dcee140e54145e0971faad510d73e753d371e92d |
|
MD5 | e7d72d77a5020eab08dceeda5d59a1a7 |
|
BLAKE2b-256 | 5636bf243740949acbffa8323ff82dada6023b9cda322ea6390765d87a4d027a |
Hashes for hugedict-2.12.3-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b24c3354d16cb198d8311aa82f058d07e6b4b32347ce47e0339bef680800f64 |
|
MD5 | 1a3acfd4ebcd6cc958ad7d34eb555a25 |
|
BLAKE2b-256 | 0a9f30d4395711f3ca7b7aafc94f280fde0e3d72ae3f1493031df869a5f19c6e |
Hashes for hugedict-2.12.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e200719cc8a77ed82f38a6d77f47fc5487b9477644c75b5ba885b0c37de660e6 |
|
MD5 | fc11917d54c5050d149d8e336e4d4bbc |
|
BLAKE2b-256 | 5e411083bdf08e0935c9e0ffa09dbe9f847784bf6a76293ccfdc07653f0b6bd7 |
Hashes for hugedict-2.12.3-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4842076f93957c22a88359ecd903a77f937241a035ce9f7776378a38cde54b74 |
|
MD5 | 9556f11a59074274ea42a6f343dd2b2f |
|
BLAKE2b-256 | e1177a5ce221856296676a0fa55d1c6e02cd7c25e9ca4492e2a0f915beb30d83 |
Hashes for hugedict-2.12.3-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb8bad26c07906e1d2d6a655cacdbd77a374d951fe19c10e2f520d24de9214b4 |
|
MD5 | dd130c340e2a4b757e7b9c2f2c7db601 |
|
BLAKE2b-256 | eb94310563d68b1f47dd1b275420db51ae47149957ec601c1d1b88b35075c37a |
Hashes for hugedict-2.12.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 25c0eead093a20de398a11a3b9ae576e2618f9fe05fcd4fe1ecf8d8b17dc5bb6 |
|
MD5 | f39d7ee97f391963b4cf380e244bcfab |
|
BLAKE2b-256 | a38cc6d08bb50a30a76771b688e2704e650b5c780d08fe2f5894fad537a1a87c |
Hashes for hugedict-2.12.3-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1699310384623c9b02f9bc923f0d72aa97736e69f6fe2bca7dfc8819b350789 |
|
MD5 | e20e7e3da667d2e04201fb277025e432 |
|
BLAKE2b-256 | 26d47c188ea4d7cd5dd54d48d16c1292680c80db6482a78f105f341643548c5a |
Hashes for hugedict-2.12.3-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4bc18a70e2dfc9840b7cd6c247f494d223f02f40409a4fb4efb42c01ad6edc4f |
|
MD5 | b2eb7e7dd15fce7d489ad651597da3f5 |
|
BLAKE2b-256 | e05ec6116cd0fc5174e4f384f3f0026a0b16aa5239bcd1381533c4809ce9178b |
Hashes for hugedict-2.12.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7497399f88741316cb863589d6cd3fcd30e3ee9f673cf028f582fdebe24b3408 |
|
MD5 | 5189669db72ca0b560499721978539d1 |
|
BLAKE2b-256 | 35adb51927f070352fff3522b3f9fb54ca609525f69831ec079f97b7e925575f |
Hashes for hugedict-2.12.3-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3a89bc60f8000ff076fc0fab4e103c4b5e0222bcf72ef0a67f874214ed298837 |
|
MD5 | cbc90d23e32026edfb829716afd55dd7 |
|
BLAKE2b-256 | 37fe33a28aad68899e9c5959950eb972c1f39003142e3eb4306c40e417bd1351 |
Hashes for hugedict-2.12.3-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1d25cbe11a3f10dee6ad5c925fcabe908c5b0c6625f9feecb850de642ce46a7 |
|
MD5 | fad214ea368c852010abaf07f492b414 |
|
BLAKE2b-256 | cce8feccfc2fbf44221cca585cd462b65bd95fb4fda6477fa887d004ded8e3ac |
Hashes for hugedict-2.12.3-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dd9979abec3b3f156cafc6bc15787182bf810b652df84094bc5c7143b47a1449 |
|
MD5 | e9d42efa6f5a8824166462d23f010dfe |
|
BLAKE2b-256 | ee1906fccbbbad5ab30df2e2d51ba29645ff148f1fb89df69fba5a78b211971a |
Hashes for hugedict-2.12.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b830574307dcd7e163ad53e2a66931fbe86acd4f1b8e193a61aedf13b20d5f3 |
|
MD5 | 858cafffc2f17127f8cb9d00642492ce |
|
BLAKE2b-256 | 9e0bfc0209ae494e4089cbc7bf931fb3ce296abcd69904ea023369f06e9375d9 |
Hashes for hugedict-2.12.3-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12cb9dc5ccddc492263c09bac107b20a5998ee8fb6e652e9edc1179c01efbc75 |
|
MD5 | 98dab39c2b125723be1e5be383d0dec5 |
|
BLAKE2b-256 | 54013fc0bc7fd7bb587afed6dd5a479efb86d570c00f657f597006803e3838df |