A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distributions
Hashes for hugedict-2.3.3-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2feb57382261513b5ce7e841325dac33ab80d15cfb1a0ffda6e640e08739808c |
|
MD5 | b38ae40456b828078d1255bed9b18ac2 |
|
BLAKE2b-256 | 1e925a316dbdee292e76bf8e83a0e8b2b53639ee0931dc4020f5ae0f625d0cf6 |
Hashes for hugedict-2.3.3-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2a46de3ca4dd90bdb36fa9824b75c8c57528873f9dd485f16be9e1db3253183d |
|
MD5 | 77e77128bba4fe12b00057e9eaa1b27d |
|
BLAKE2b-256 | 2124b0610895a1b4a583d8f0ab6cabc560ae9e5b1432ae3c1473297801825f3b |
Hashes for hugedict-2.3.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2b7fdcd12b46101328b8623ee450459ab40afd28bdf96e3921d34750a0e523c |
|
MD5 | 93d88dfadfc4d5fca1a949b677005dda |
|
BLAKE2b-256 | d7c713caa4104b5cdcd96586f9be62cbd901c1587a9ef014464fc020b19d9e4e |
Hashes for hugedict-2.3.3-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e18499353fc637a6dc08129f716fb801bf33b15c4258723db9c6300557a0c2b6 |
|
MD5 | 44817f88d9fbd9488ebc68042f20e366 |
|
BLAKE2b-256 | 8bfc426f29d06b9a88c2241686edb985c7d9516556ef431fd7f8311a26b8e239 |
Hashes for hugedict-2.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8037476debecac759112af03bfd378a0eccd1d0c422edf0762e8f12699cc8d89 |
|
MD5 | cee192eff08d0b666f6295a259830da9 |
|
BLAKE2b-256 | b72ec27637713112d88026a511e5bd44350ddb7a61b4c28aa02daef2cf678519 |
Hashes for hugedict-2.3.3-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a203a2dc9ad22adb13378f7ec0b82e8bcb1d4ba27c93cd40c7bc3f9a5e7cd772 |
|
MD5 | 0ac8cde64fce990ed0748c7eba6fa57d |
|
BLAKE2b-256 | c650738ac4f03bf1bd54b2d5d2d1c09aa2d906b15d7f5e674ca1dbabb1bf5ae2 |
Hashes for hugedict-2.3.3-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87f03c7079c1a1c134f6fd616a4e4662448658452e0207ab5a6ee98e414abf24 |
|
MD5 | 23dd7da80fb3ff9d83bab7fcf539e0a4 |
|
BLAKE2b-256 | 2060f3afa6244063f8a8d19d51f58a775ce4f5fa98ea5f0567590571a5495d28 |
Hashes for hugedict-2.3.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c089731eef4b0d61854d9474eef6b2304f9c69f7cb288e98759cbdb8448cc070 |
|
MD5 | 3496bb358131c4a839a9e2ad14a42a80 |
|
BLAKE2b-256 | d332255756700b8b5a99159f735d8d71f7c65e70346e6e281d243860c1796c86 |
Hashes for hugedict-2.3.3-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b22428f07c1d66ce49f0b3de25ca8a5ba9974ecc3c368fae49b0d981d9d55338 |
|
MD5 | b81ff640fe54d61d27a6d97baaf113f4 |
|
BLAKE2b-256 | c7a815d03d32aa8a1a67dd67d04af3fe78d7c84e941348949ab8c0897124bffe |
Hashes for hugedict-2.3.3-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9c43fddd93377ec48fc0363e7ecca60d9ad89fe5719fde2230d67928b1045f80 |
|
MD5 | 7d90e240a75caea3d8b6f18e99394754 |
|
BLAKE2b-256 | 8cdfbdb2393308b21232ea7d733a4dc3d1c11e287e8de236c5bb36aaa6c374c9 |
Hashes for hugedict-2.3.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 58fde7bc1bcd47234d687d4bec5735e678cb6594ab90cae09a7d814c1646caec |
|
MD5 | 5db29e774e7a56cb32bc08714aae13f3 |
|
BLAKE2b-256 | 53b9340e7c7a259418a16ed1c19398c556448b7b92a8db32b62893445917af06 |
Hashes for hugedict-2.3.3-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 40f68c48741217299154584d706b112a6f93ebb03727da045b432fcf3587aefc |
|
MD5 | e28a69f1fec84dcac06b375f5f293a3e |
|
BLAKE2b-256 | 69d9ad1db14860a2aa8fdf38e03074f8c557717de3de5ffa9d5cb031c14ab425 |