A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
![Documentation](https://pypi-camo.freetls.fastly.net/cb2a3c3438551102399986f02b7e576247103cac/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f68756765646963742f62616467652f3f76657273696f6e3d6c6174657374267374796c653d666c6174)
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.4-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0d8edd45806f114543d5351def006c22ba16fd8b3875b65aba60ba677408d7c4 |
|
MD5 | 4bd5cb235ee5c18bd111a4ae084da5ff |
|
BLAKE2b-256 | 51cac1b31edefccbe318077d1bf665a5064ed09b5ef8a09e689fcc93b1ede922 |
Hashes for hugedict-2.12.4-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0266abe19923092f02856c2440fb443f55370f6afcf3cf2a1fcfbc19522cb948 |
|
MD5 | ee7379058f8aca74ff72ba49f8cedcf0 |
|
BLAKE2b-256 | b16b2f41b5c3339891d07e763437979e59502826e0890338f00ae8b715da8ab4 |
Hashes for hugedict-2.12.4-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd7bb9703304a5d4c41924f40962b7e382c943ac66cac2bfbe9f857004f9123a |
|
MD5 | 7c692a58ff83e815b81a5ad1bf60ec7a |
|
BLAKE2b-256 | 01356c5ffb724e87b083eda5243f647570e4c9ba3955d22a92f9a453ec9e90db |
Hashes for hugedict-2.12.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 04e607454c12a608bcd1216461eeaec6079aa70b1b4eb7ec638da35266b100dc |
|
MD5 | 0696201045bfb8b6d9047c5ad1128ba9 |
|
BLAKE2b-256 | 02d8f65a2e8ce51c7702e327890d6387b839e660ac1f0b1dd4195659b3dfb283 |
Hashes for hugedict-2.12.4-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ee0f3eba82d239724e57877ea6eb1021eb55f83b950c3f21c31367ddbf36aa33 |
|
MD5 | 0578ee58f2e459c60a7b47b1296c842f |
|
BLAKE2b-256 | 284a61b399af9e22d5d45efd8e4bc86c511a3c8e78b49262c897dcfe2625fcfd |
Hashes for hugedict-2.12.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 17e83caf88cb9e17be6b63fec3d558a63b2a38f8bfb9a98403cce2bebc154c22 |
|
MD5 | c1dc101e1385af136723332bf4741398 |
|
BLAKE2b-256 | ff5646bd58c3eacbaf3fa6a4c402ede4bf716eec60216bbf47c7b6593d593da2 |
Hashes for hugedict-2.12.4-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cd3b68a728d23ea84931e98643baa75d35da8c7c9c38f0531ea1116a36ae7478 |
|
MD5 | 612c4007a9b07d03779b82fe7e2fa773 |
|
BLAKE2b-256 | 48a614f8be91f12d1f032813689e3499609671cabbf8fdb3ec2723b3b3ce3735 |
Hashes for hugedict-2.12.4-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb6801e1e3e728ac991c922be4a22647d58227419e3615dd6b1945dac71b0e4f |
|
MD5 | 5be7db9abd1cfc69688e1ca1e774bdcf |
|
BLAKE2b-256 | 15dfacd6bcbfaade72b827b0b8c787bc1edeee1457b518db3437c46bcc55cd71 |
Hashes for hugedict-2.12.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a05d770964c1add59f215c9893871d38d3f25c4ee9f6dd7792770887edd8ed26 |
|
MD5 | 1b53bb55680dbade5d33140697958120 |
|
BLAKE2b-256 | 012446eb4d073133385f9c02d375ef3a1923601813ee80524e67d53c1e35564c |
Hashes for hugedict-2.12.4-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fa0886d9f945a935d04d152f797c5faf41af5068ef23a9f035eae41f3fd84a61 |
|
MD5 | 80031d5bfce1bcf591aac5b28845d2e7 |
|
BLAKE2b-256 | fd0a2d53165cbc0120d4ec4675d1fd4ba0c750ac845ee9d52ff62e00fb26f739 |
Hashes for hugedict-2.12.4-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6a14cd584c8434e6cbe851a0518ebf1391de4bdba88cc0ab7f6ca1ac12f49b89 |
|
MD5 | e0716723fbf3f98d7d5c6934bbbbf7cc |
|
BLAKE2b-256 | bbb78518964985eb465bdcc9f87e11e7363e2401ae9dc9cad70e7463db9b1fed |
Hashes for hugedict-2.12.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 802fdb40ccb8a8f42aac149228f3c0b5a084117c9650a49383bbe85cbb9dc948 |
|
MD5 | ee646fec7c8ba75bd2840234253d072f |
|
BLAKE2b-256 | 4f9e60acefbaf1c446266e43c141e5ea1bf29980c8806ba8cd7b64fcc221706c |
Hashes for hugedict-2.12.4-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c3e209c7a106372faf3c75f469a48ebcba6545a1da7fec38de248cd606191f1 |
|
MD5 | 3f4364b9e05e07c869b96f0831c5b24b |
|
BLAKE2b-256 | c60c08455514f6d228f0cd56e45870ec49a866d5fa102d7f94198f5bdb01e8c9 |
Hashes for hugedict-2.12.4-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 20e7febb2fdb03981504720123e53872d1629197dd2d0ba8553bf86b90a1846f |
|
MD5 | 6bd2356fecf9a03c8c9af06aa94803a0 |
|
BLAKE2b-256 | bd7f1ee714aab53f2dd347cf156f0b16dcee9fcb3fa246ef38565f6e2769d594 |
Hashes for hugedict-2.12.4-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 022840f38191f68344876596301cd68bc935723200662ce68135f547c96342a4 |
|
MD5 | 96accf6da64b4f3e094bdb5bd8aafc2c |
|
BLAKE2b-256 | e022735b432832a7b5523847dc680987f3e00212e4e5a65cc05872c1c4061b0d |
Hashes for hugedict-2.12.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a7b10b670d1c95790f390b2f0b2c7d20ebbc9726650d6dbd6f6121d5e2d0a9bb |
|
MD5 | fe60d9f51cd1f3f3b0b423843084c711 |
|
BLAKE2b-256 | 78af397521c7007ff39a6640b15ae63628e844c52e19b2201808dda0db96b066 |
Hashes for hugedict-2.12.4-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1bc72867767f8a81f5de3aefded9c140c918b14f07b4bd5621b79d176b8d3081 |
|
MD5 | aa72480ca1e517ea344f753077a7afc9 |
|
BLAKE2b-256 | de0a79559fa63d350ec36190cc1f21d863d41868dd5a7f4b5ece7823b621f3f6 |