A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=artial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.3.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8e7726e253f7433363d076a2b61476f3092d942a2df5f8e45a546f9a19070a29 |
|
MD5 | 730690fe45291da4b24978d8de6f403d |
|
BLAKE2b-256 | a0a5402512f266f9568f95dd44c5cf2cf8f14dada423b6a6eef425f7a39fc7d5 |
Hashes for hugedict-2.3.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 002c4fdebdfb07e79b01c943821ac72ee388610382402dbefb47d32c54e4ba12 |
|
MD5 | 24692914b8bd5c7c1aefc8d8d0a6c7fc |
|
BLAKE2b-256 | 01a23802477e855334cefed5f3c4760186350a4e13e9a59f94fd66766b91b11b |
Hashes for hugedict-2.3.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8b9fcec819114586177b853f33fc23245fba19dea7b77e34c15aa5c0b3a25864 |
|
MD5 | 5b717bb4b61dd282cf70a0d819e54ff4 |
|
BLAKE2b-256 | 5b9b0f0f7ccb89e1c277fb70bd95397458fc2ecca0429464e1a73fd6f9164a53 |
Hashes for hugedict-2.3.1-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 32a3a1f6a22fd893ea80fe91b1ece2c1722fa4111f07f6a887c70016873561ac |
|
MD5 | 1edf2f1136cb1bd6517ada55a2a266fa |
|
BLAKE2b-256 | 3708efbae8430956928c900067bf947b25cbdb91300b919d7d412a53b88359e4 |
Hashes for hugedict-2.3.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0543670ce5bcf44ef1edeba76ca68186839a621b07c6527d2649401a5ad75ae |
|
MD5 | 886ace9b0125ee0ff954a1cbb66f4c62 |
|
BLAKE2b-256 | 3e54dcc0f109f057ce9a9606ce374f6dcb8e8c195387a0e93729951b94462f59 |
Hashes for hugedict-2.3.1-cp310-cp310-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f9f46773fdefbea304c201b9504f3d063ef8f4150d20e3389694e02a1df45a9f |
|
MD5 | 2c9c1eb117cdebb66413b4a587430b9f |
|
BLAKE2b-256 | 4f58d0f33b49c3786b52800c38c7bd9164c590d0288c84866a29994b35288999 |
Hashes for hugedict-2.3.1-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0bddfa4c2f8647d0d34f7e374fc9a4e89b83d2c5205804cc83645a54a90b3793 |
|
MD5 | b5a1415842700b15b4ced307579b0d59 |
|
BLAKE2b-256 | b9252629f2d210ff528243c313281ee334650f50d4325e3984563e4febf46f63 |
Hashes for hugedict-2.3.1-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec64c0fccfe873bd06abc69ab4e9bfbccd7991ef9e9cc6fee0d81cc6afa9c237 |
|
MD5 | b335ab2ae995d8da22b9834724f1195e |
|
BLAKE2b-256 | c6f128f234bc093c37ec56e6bcbcee72fe1ae4c526b22814491d0d277ab27e81 |
Hashes for hugedict-2.3.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33f55bc9ba1470ec29b419ef08ebcb5180a9419aed0f54021c09d4a36433ae98 |
|
MD5 | c84290746db313f4b87cbd81f7397d03 |
|
BLAKE2b-256 | 16c7e120c41eb7671eecad9c5c55f280a3268ce5f23796ea1f32d6413d448b23 |
Hashes for hugedict-2.3.1-cp39-cp39-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0649791f88c037bb4c6ba98109aa50fb08c13dc933b0e5cf90eb02f9376ee54a |
|
MD5 | 8245a1b4f37d75c056ff70c5f0067cad |
|
BLAKE2b-256 | a713f98ca951df75a49fb4cdf77799792d39c3e171095cc91ee7050e9d5b96cc |
Hashes for hugedict-2.3.1-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ffbe573cc874de6d7e991b18913f1622c0d08f531da6c1fc78037c0f7fcc7005 |
|
MD5 | de16b29ee5bafe9c109d01eec1e359aa |
|
BLAKE2b-256 | e138e5aece400eaa9e185b10627cbdfdb281135425ff26fcf96f8a0a351be2a4 |
Hashes for hugedict-2.3.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd53f43c42bc9e74638bc90ff6031f1ca1c616b69c74ef1ee1fe7182a3f63bc8 |
|
MD5 | fe62c4c8c749b05550a8b231591796f3 |
|
BLAKE2b-256 | 30db588be58d005bee0c4072b4a2a313dd60b98d265f5ae7676a1bc86dd56353 |
Hashes for hugedict-2.3.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d605c4d2046893cc4afcd451d6f9947be3d69f1867bc08b9d09104c7c8072b54 |
|
MD5 | b92da61edd96b237907b47933546ea08 |
|
BLAKE2b-256 | bac394173bc2086d4f1043b6e53eb50c3d703b02f30f03d61d0bafc1ef891486 |
Hashes for hugedict-2.3.1-cp38-cp38-macosx_11_0_arm64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 94a981227d84f4033cad57c80e4bf665867e8e021f5a916931ae4c156089fa3f |
|
MD5 | 35e839bf3ff3983cd8da903ba7b47680 |
|
BLAKE2b-256 | 801aa314fa2f75ee43ecbf18e9d4e4d0c10b88d8f3be7070f41608c81d6e5f26 |
Hashes for hugedict-2.3.1-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c933a0857c8d1f45ccd58e9c10e12d977ced7881e27c75bde9ce3db0e8a752a |
|
MD5 | fde51ccc8eede5af833039d0a9f5576a |
|
BLAKE2b-256 | fd2ebd8ccbbe807919b5436aec7f9a2541d3c5a398062f5c3b2e9d62d2144d3b |