A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.9.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 33a55b4077ea650600ee0a53e52e393064c300bc1d0dbc94f30c58504e9110e8 |
|
MD5 | 27b91c75152f25064aac4a213c4cc5a6 |
|
BLAKE2b-256 | 7044c992f62209767bbbfef9920eefd824b824b5e4d6c292cd4cc8fa95f92d00 |
Hashes for hugedict-2.9.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c9bf6127b2c496c4d27355e8698d3205c94f63c68cba4abeed98fa67585c001 |
|
MD5 | a96c89cb3750bcc45e867c411632947b |
|
BLAKE2b-256 | a4dde8196bf9ff757f38a37cf38ca194cff496a3d278b820ea97a6053c631c0f |
Hashes for hugedict-2.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cfb6fea7616daf11ab2ed1ae66bcdde9f42273c95e8df142f70f75d395d8d774 |
|
MD5 | 81cccf4fe495a33f5d26b05ada1e103f |
|
BLAKE2b-256 | c3863c24fbdde8cebd79633871d4df4f7053e3b7d34bb3b016ac9ddf5e24ffed |
Hashes for hugedict-2.9.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e26ec2e603eb4743a3b7416e21a25e90d76736b657de893cefe8ba15722d1af1 |
|
MD5 | 6a52a8e52bd36a239aa9138d3f74df89 |
|
BLAKE2b-256 | 5448ff723f389e728a9b00b3e4365f643ef4935dc212a90b1c697bb15f9534a0 |
Hashes for hugedict-2.9.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4562fa895ae82eba4e0b196a0e882ef6b1f56fdc90624b8dcf20aff5feb0375e |
|
MD5 | d4deb84d5af4e2d7a06d6cd947255aa6 |
|
BLAKE2b-256 | 098897cf3cdb1401c69251d7ccb45fba90f85017b526a6240d49f4d154e92101 |
Hashes for hugedict-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cdcac4be5e9569151addbb9ec1bab49b48f6a87bce0ef1c26319cb2116808cfd |
|
MD5 | 9a62ea92b316ac1abc6c5de60c342b49 |
|
BLAKE2b-256 | f039b204b15fb898670888e8d3cbf6e9e4678965c8ddb80f3e8a88a444919729 |
Hashes for hugedict-2.9.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f867b4c8b450a02475551257fce1862def7b4967bdefe233c866f24cb30b877 |
|
MD5 | 68abbc7f88ff5680215a9df413ff89c8 |
|
BLAKE2b-256 | 34c4db863c16c1d099d3d5283aa5f8d7ab07961498497d30b42da01c954d7023 |
Hashes for hugedict-2.9.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1ec0ca912a7c3bff3897add8c2ca3be9de872fccd93d96a10d0ffc7ac549873b |
|
MD5 | 1d59814c9d4d9205f8e0d23a3dbc508c |
|
BLAKE2b-256 | cf07703ae5fa376ef60d705c9a4ebd2a4b4a4ab8ee79afd3ee483b10f1aabc2f |
Hashes for hugedict-2.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2e56dea881c0476110771678932c466da371f3ef0a071dd4800d6e172b861bd |
|
MD5 | 17c4ad41fb9d7e3e5cab2e1d4ee60258 |
|
BLAKE2b-256 | 58eee6d69d6e39c6a0fc343ef0d44a3c4458f2cc4aa6cc5a234a697d852ba10c |
Hashes for hugedict-2.9.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c2ddf4bb2513c02a593c56f2516dd46f28cbe93278ec00e839d23eb38326689c |
|
MD5 | aff927d9fbb7b66bca602ea814e495cf |
|
BLAKE2b-256 | 7a9720743237ebeb8efc67e8759f2f9d637a24e316604920eb2c99d4e97b30ea |
Hashes for hugedict-2.9.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f48fae86ab8d878d8a5fd3534377225efa196b7f030daa5012303f0d3f8e1fe3 |
|
MD5 | 8400d4ed2654d5a9a8ee8cfbb1227a2a |
|
BLAKE2b-256 | 37c1700fb5bab6d6e9ca91a3f7163ed108d59fdda8737c9583461a9faf45980b |
Hashes for hugedict-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e72d6d727bbb60f6db796ba908a4bd66fcde9b7fe4e83ffeee2531e9a2e938d3 |
|
MD5 | 80dc6e20cce1e972395fd7fddce487ef |
|
BLAKE2b-256 | b753df022a8c4ec0d5c22792c20232e9802940845654a5d71bc3366f8e5c3e17 |
Hashes for hugedict-2.9.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a6db782dfaa8e25377698a2d2debb73d9b58c7b79d34561978f572e19b715bd |
|
MD5 | 232ac1cf8a78fd4b44e26fcba8f760db |
|
BLAKE2b-256 | 263f483096cb6c368d138db82e610c71e9c331c9279eb6c6eec5bb3ea5001c07 |