A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features:
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=artial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.1.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c9c863743635330ca29a745b725d4944c6406e3b346a448bf267268e26cc6e4c |
|
MD5 | fbfb54a44811bae1b2e7c0695f33e571 |
|
BLAKE2b-256 | 97e191f4483ef57351583aa442bafb8aa59d2632543ff742895af74bb7656fa6 |
Hashes for hugedict-2.1.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a115a101da97d6dcf8fb201b1c929a4aaaac85203daebb2e0cf5433cecdb5ed6 |
|
MD5 | 0603878a7fa36347bb0b81b1ec88423f |
|
BLAKE2b-256 | 51dad420236883b52cccfe5968e5a795c58b6e58201f98a3ef0cdc5957ad214a |
Hashes for hugedict-2.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ba8c26198e060705d7004292723a8f0fc46034dd71c0e1722ac2a17c42917d29 |
|
MD5 | 0b0e8cfd9a7bbd5dd56a8ebab72c1797 |
|
BLAKE2b-256 | e8d6187842720dd2a6c9abd3fd473fe581b8d6b0554ac263b1c7ee10a10c3a4f |
Hashes for hugedict-2.1.0-cp310-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4341263aeee985b39782532e7ad4fdc3da6ac6fbbbf4f8d71f5288e4446cfd0c |
|
MD5 | 0cc196abd6a36d350ab783c97e403b2d |
|
BLAKE2b-256 | 46c0d52e50c872816c5987d84b72bb00ce693556ea264466af1f18e57341afbb |
Hashes for hugedict-2.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 77b87b76a4cb9c12bb9c7934566831c4ab56f1e7acb865781d32cb96b4528c97 |
|
MD5 | 0d51d384ca8db8e530b0cc896e6352b4 |
|
BLAKE2b-256 | 145df6e654ae32d6ca2ed2c4d9c9d77327ca3888e97f991eee0307c4d4bdb9e1 |
Hashes for hugedict-2.1.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b17cd118f5ba2f9d71d413508c4a5a48d0fa0a752efbdd533b04cbdd3a6cf8d9 |
|
MD5 | 728c67426abf23a094eb72622441e9d6 |
|
BLAKE2b-256 | 5a255c0d145b79df3e73e08d0b18b461ee8d242e17862804bdbf0cb7503d4db5 |
Hashes for hugedict-2.1.0-cp39-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d25ab561623e15619441252e362f22ebb6d68331c59345ab29babd51848287d0 |
|
MD5 | d08aa4f46738cbaefc0d3a335ce4feac |
|
BLAKE2b-256 | da91225f70b4cda1309f249b9f6c0107d4708165e54eea3a870f30a5e8daf836 |
Hashes for hugedict-2.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 18c482dada3e535a454f1c11b96e4834a9adefe82eb71c3e8da0d5ac34048e82 |
|
MD5 | e851da7f38cf974da5c9906c353d91bd |
|
BLAKE2b-256 | f894a12abe175ae0c996e00ddc5f22b36c62f880a4897bd8a50442cca90d6088 |
Hashes for hugedict-2.1.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e7a0fe75297492b4ab95b90a4a10514be3cb11e82dc0dec4edd1e34a8540467 |
|
MD5 | df5ce94e62e3c1ac384ea482f0e6fe77 |
|
BLAKE2b-256 | f1f4cc8631ea3bbb7e1b2bdeaf7034fb0ac9f8ae999153387545a9c75b7ec2b5 |
Hashes for hugedict-2.1.0-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 766175dcb0bef82b98ec0e017f602c6682fdf09c582f312c21ffd01ff7080015 |
|
MD5 | 12546211f641b63ead555c1de4e17b42 |
|
BLAKE2b-256 | b0b0d258c47423c12e95a002df7944e838bd9947366df35f85e7adbd7c130078 |
Hashes for hugedict-2.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ff9189f2bd35b4ac9d0914dc5bb4b9b06ab2d0260c4f45e21b3a0aaa78238f49 |
|
MD5 | 5f36589107d8e148df7c1ef893b9096c |
|
BLAKE2b-256 | d9434820e0a83d91dcb05a05a0d55248f9ddf972f2532195077a388bf5ae7524 |
Hashes for hugedict-2.1.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dbd8d09ab67e6e14f6f2ee9064fdca9a65d352381efce160de5505880340cc37 |
|
MD5 | 8d1766568a36766ca7bab68ec1f1af13 |
|
BLAKE2b-256 | 30db841e0ab0409d1159a08b57b1965d8f54c555f0aa6b964d7b34881e63f2f1 |