A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.10-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 457a053fd31e22faed2dffc8d29d487c5c338bd34f0a121cad06d9423399ff9d |
|
MD5 | 15434288bbd1a6bc4bbc81ea304f2657 |
|
BLAKE2b-256 | f33c32d0f8481aa14205f4018a9558fabf67e72a3f1349305d7a229261eec1c0 |
Hashes for hugedict-2.12.10-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f948f6af3c5e45c2a9c6d660a4b42c0d391d207faff3f27ce1b47be85f5cf14 |
|
MD5 | 6106c52d01b8e8912ace8807bce206ea |
|
BLAKE2b-256 | 6fb4bc39aefad4201677df8cd0509f92560fce3a747a0bb4f57905dc37096c1e |
Hashes for hugedict-2.12.10-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aff1d76cff7126c43aa686d9032cc6b698895ee9cc1e99de735cd8bd9e4bb452 |
|
MD5 | 0d75f8339b80b4e34d22421d6cd88f5d |
|
BLAKE2b-256 | cddf0cb7924143ed983335ba33b8f06845c5f99fd75ea94fe188f87c73ec93b9 |
Hashes for hugedict-2.12.10-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d850f938725f4a8674a71b875ec88303ba206c8b6006e2b5eee41b2eb435cc22 |
|
MD5 | 6efee286423dbe64e64c031de23bec5f |
|
BLAKE2b-256 | c6bf3cc0a85e2e812b1cb504e4d845c2629e01ae72a5bc4a70e9031229f6c723 |
Hashes for hugedict-2.12.10-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0320fff41c3978484e728a3a1e51634d224d0323e6a28f81f0c708d6abc14c4 |
|
MD5 | bd6e36c7ff009a71472f4002b7760f6a |
|
BLAKE2b-256 | 8a396b298c064e15718a34bef459006db5e7316a7b81c906a67e99b2269173b0 |
Hashes for hugedict-2.12.10-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e024da3c3bb8509e712a4fc84db7cf27ceb435f75bc599e4e5a9cd7c460b3296 |
|
MD5 | c93c18ae2d864eb092738393a04d419c |
|
BLAKE2b-256 | d8ba51bdeb3637a64ac0b485b59e7ad21538109694d1ac5af3fa3c1f97635f80 |
Hashes for hugedict-2.12.10-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0388e579250ff6ab31ed3e115c3ff73f922b751ade7ad9021c09ad6e4a84b80c |
|
MD5 | 5941060509d0ea1196fc548465364ab5 |
|
BLAKE2b-256 | c6bfa768ab3109133bd639384192a7c215939a3187d932d55bea61a141c0a877 |
Hashes for hugedict-2.12.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ab50a34c770a5b8ef9622c7aba15ecf133fc9ee8aa665fdba992b71825c03da |
|
MD5 | 5a50206c59e4a68184379bf0993f8fa0 |
|
BLAKE2b-256 | d03a1043e73fc7830d9ed657a922fe3a923c8b31e8b5ab1f9167721f30685b6e |
Hashes for hugedict-2.12.10-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 849ff95560e73e589c8124594c289bfcc24b1d99f6e7a53d24ab6bb2c4db5324 |
|
MD5 | a3cbd416e8468e50394aad795fd57822 |
|
BLAKE2b-256 | f686c13209929c7d46a891a8bac37b61ac248c994f1c31f6386f4fb62c477e31 |
Hashes for hugedict-2.12.10-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e93495294ea4585b35ddda02933f31c02da9bb0e3c2e2e9784aa8faf053df5d8 |
|
MD5 | 7cfd06bf5efca1a16487bfc3b01736df |
|
BLAKE2b-256 | 090367b49b40a2703821ca3a43d2824bbd394eef1b76cf941c145aaee73d4c45 |
Hashes for hugedict-2.12.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15689e366f259002fff20247a3a02fd4eec8dc76ade6e37ea72ff0a05ee51aec |
|
MD5 | 9efb2e2858779fe17b2a95d02bf30c17 |
|
BLAKE2b-256 | a13353718df700e9833752ec246d05f93bf30020fdb52390fc972d33e324bb94 |
Hashes for hugedict-2.12.10-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4eeb8c744c7f83d979e51c297fe39d7aad89fd1f4bc6fb63a45a5da13a1559eb |
|
MD5 | 646a99fc3d5cae7cdf2139237e29cab6 |
|
BLAKE2b-256 | 02c33ca0299323b3b5dbe665c14e65262dc81d4fcf3dd6a6a949220881f9108b |
Hashes for hugedict-2.12.10-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc7d7077524efa702352f91822f2b69e723233f6eefd711be903c820356646ce |
|
MD5 | 79ac8de3b032ccbdb4425fe002acfcee |
|
BLAKE2b-256 | 9d5499cf46a109008ce25e20ee3dbd4bf558723af101fad027a8979b6b4ae9eb |
Hashes for hugedict-2.12.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8372ea95953e3e7a5bf0e8a32b7ba093fe56df8a00d8c926721717d61535541f |
|
MD5 | e0c7a238608124f17f37c3a074105ead |
|
BLAKE2b-256 | 5f47e2cd833a9b8ae6863107ea240f98a9204f8a40044cd95b9ea5c054693712 |
Hashes for hugedict-2.12.10-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1294a5ed7859eb6392c7fac6c5fa6bfc6b9ff98cbfa0de232c9204f175f75493 |
|
MD5 | 7d9b3918ae771025e6ec4e60012c508d |
|
BLAKE2b-256 | 82a2756e77da2f62ab050ae00c44a83e3a60222ec98dc2a5171f9ac3812cdbe5 |
Hashes for hugedict-2.12.10-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4de62cffce90b1177fd9b36c9a6d5299f9bf2ca5529c90f7e4a91b9cfefbcb78 |
|
MD5 | e7c462b585a81b9faf4aa972982f2fde |
|
BLAKE2b-256 | 8655206bfea9db70b366b6878bd54b7dd9cc2958386c55f46cbb3cd4d64c4421 |
Hashes for hugedict-2.12.10-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a28a3b9f6609dfcfe0ba450513fc86fd56bea576ff715a0fcd01783c9ff536e |
|
MD5 | 4cabbfb5922dd6dd1f258ff89abc96ad |
|
BLAKE2b-256 | 39432861ece1efda172872fe387d981782efd744a623343c7303b0e1c3963032 |
Hashes for hugedict-2.12.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f92506d4fff38e08c3e976243211393a0b90b4d9df83e81a27dded0a27ebfa3 |
|
MD5 | 30831d9672b0ec1a10b8cbec555af120 |
|
BLAKE2b-256 | 2364c4b5aed4b78a16537021597a0e33abb7f95154b2aaff52fd0c5240fe1805 |
Hashes for hugedict-2.12.10-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74399ec180f3b0b8b2d1a227e6be204b4547043d87ed7bb059dd34c8999a3ee4 |
|
MD5 | 7c1356fd4b4c07323d00db69f6de0894 |
|
BLAKE2b-256 | a702ed945bd8104d18996a0d9fcb7e5f619d385e60d34a831047bbf36d7049a3 |