A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
![Documentation](https://pypi-camo.freetls.fastly.net/cb2a3c3438551102399986f02b7e576247103cac/68747470733a2f2f72656164746865646f63732e6f72672f70726f6a656374732f68756765646963742f62616467652f3f76657273696f6e3d6c6174657374267374796c653d666c6174)
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.10-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 457a053fd31e22faed2dffc8d29d487c5c338bd34f0a121cad06d9423399ff9d |
|
MD5 | 15434288bbd1a6bc4bbc81ea304f2657 |
|
BLAKE2b-256 | f33c32d0f8481aa14205f4018a9558fabf67e72a3f1349305d7a229261eec1c0 |
Hashes for hugedict-2.12.10-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 36883eb82b8066edb1fcc21521adfaa34a65645adc10578193ada91a0f0fef95 |
|
MD5 | 55fb835ef20ddca20be5825258a6f602 |
|
BLAKE2b-256 | be471b954f2bde6ccacb0748c68c46bc5777fc5915f35d4b8f997262f1deb1a2 |
Hashes for hugedict-2.12.10-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4f948f6af3c5e45c2a9c6d660a4b42c0d391d207faff3f27ce1b47be85f5cf14 |
|
MD5 | 6106c52d01b8e8912ace8807bce206ea |
|
BLAKE2b-256 | 6fb4bc39aefad4201677df8cd0509f92560fce3a747a0bb4f57905dc37096c1e |
Hashes for hugedict-2.12.10-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fe584b27feaaf764372b6e270abb21ec11a3a4ee6b106409e7311bac296fa23e |
|
MD5 | 0a07606970685d5137576c69168fd44e |
|
BLAKE2b-256 | fcf8860efaf29175f0cfc4caa8e2a99e10e98932e68b0e9eafbfdcaf3cbd4f8d |
Hashes for hugedict-2.12.10-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | aff1d76cff7126c43aa686d9032cc6b698895ee9cc1e99de735cd8bd9e4bb452 |
|
MD5 | 0d75f8339b80b4e34d22421d6cd88f5d |
|
BLAKE2b-256 | cddf0cb7924143ed983335ba33b8f06845c5f99fd75ea94fe188f87c73ec93b9 |
Hashes for hugedict-2.12.10-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f2eb526fd75928b0bb1ffdbcb94d8798135904e031b4a859463b168363e83932 |
|
MD5 | 3947d62ea34bb371f73a65ba72e11708 |
|
BLAKE2b-256 | cf01d2cbfebdff9187ffde17c0a2eb40ac0152b4d1d250fc4d71ba31c62072e0 |
Hashes for hugedict-2.12.10-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c48cc9fd59cf3905c43d2a9a231fb563cd8f742484478a0d6dab1e25d5174b0b |
|
MD5 | cdc54a70b7ef5a374ed55916c035b167 |
|
BLAKE2b-256 | dd94fa180672f51dd42a747832495946df9dd6154756efa3b6e6bff8fbf9ca22 |
Hashes for hugedict-2.12.10-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6f8433097fd1df7c28a4fd938e938ceb1e066b8f61f328d605cb091c7a3f303c |
|
MD5 | 132370e6828af2b1bbfd0e58844d89f9 |
|
BLAKE2b-256 | d8e80ddd65592e63af3c52e85f779ade93c71087f370ebf717eed28b9d868d32 |
Hashes for hugedict-2.12.10-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d850f938725f4a8674a71b875ec88303ba206c8b6006e2b5eee41b2eb435cc22 |
|
MD5 | 6efee286423dbe64e64c031de23bec5f |
|
BLAKE2b-256 | c6bf3cc0a85e2e812b1cb504e4d845c2629e01ae72a5bc4a70e9031229f6c723 |
Hashes for hugedict-2.12.10-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b0320fff41c3978484e728a3a1e51634d224d0323e6a28f81f0c708d6abc14c4 |
|
MD5 | bd6e36c7ff009a71472f4002b7760f6a |
|
BLAKE2b-256 | 8a396b298c064e15718a34bef459006db5e7316a7b81c906a67e99b2269173b0 |
Hashes for hugedict-2.12.10-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13ffe43799572677aecb4e85b4b5a0b65d29c963b256b53d9d6adc27f95452e6 |
|
MD5 | 27981cd25296d4b167b3ac31a3e08bbf |
|
BLAKE2b-256 | a9abad8834e441e8e42f56ff46b970af43c8236fd8166bd23b9b3aff8ef846bc |
Hashes for hugedict-2.12.10-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e024da3c3bb8509e712a4fc84db7cf27ceb435f75bc599e4e5a9cd7c460b3296 |
|
MD5 | c93c18ae2d864eb092738393a04d419c |
|
BLAKE2b-256 | d8ba51bdeb3637a64ac0b485b59e7ad21538109694d1ac5af3fa3c1f97635f80 |
Hashes for hugedict-2.12.10-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0388e579250ff6ab31ed3e115c3ff73f922b751ade7ad9021c09ad6e4a84b80c |
|
MD5 | 5941060509d0ea1196fc548465364ab5 |
|
BLAKE2b-256 | c6bfa768ab3109133bd639384192a7c215939a3187d932d55bea61a141c0a877 |
Hashes for hugedict-2.12.10-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ab50a34c770a5b8ef9622c7aba15ecf133fc9ee8aa665fdba992b71825c03da |
|
MD5 | 5a50206c59e4a68184379bf0993f8fa0 |
|
BLAKE2b-256 | d03a1043e73fc7830d9ed657a922fe3a923c8b31e8b5ab1f9167721f30685b6e |
Hashes for hugedict-2.12.10-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 96b079ffe08ded7693c849d57bee843b11cc64024fd34fbf1454da52cedfef29 |
|
MD5 | 86964235c3dc672c32aab3c3cfe8a3ae |
|
BLAKE2b-256 | 6721d96e6a9b2b1d78ffc125bc909861bd643c74c0920bef5983c6e84d54b727 |
Hashes for hugedict-2.12.10-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 849ff95560e73e589c8124594c289bfcc24b1d99f6e7a53d24ab6bb2c4db5324 |
|
MD5 | a3cbd416e8468e50394aad795fd57822 |
|
BLAKE2b-256 | f686c13209929c7d46a891a8bac37b61ac248c994f1c31f6386f4fb62c477e31 |
Hashes for hugedict-2.12.10-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e93495294ea4585b35ddda02933f31c02da9bb0e3c2e2e9784aa8faf053df5d8 |
|
MD5 | 7cfd06bf5efca1a16487bfc3b01736df |
|
BLAKE2b-256 | 090367b49b40a2703821ca3a43d2824bbd394eef1b76cf941c145aaee73d4c45 |
Hashes for hugedict-2.12.10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 15689e366f259002fff20247a3a02fd4eec8dc76ade6e37ea72ff0a05ee51aec |
|
MD5 | 9efb2e2858779fe17b2a95d02bf30c17 |
|
BLAKE2b-256 | a13353718df700e9833752ec246d05f93bf30020fdb52390fc972d33e324bb94 |
Hashes for hugedict-2.12.10-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a8f3973cff3af97a124d6437a77925fef438cc17f78c3ebfed0a4d681b2606b7 |
|
MD5 | 2ae4bbae9142e0e180c3d6c218da8703 |
|
BLAKE2b-256 | 12d419e08684d18866964f1938332d8a01cdf9966645690e69917fa2a5cf1813 |
Hashes for hugedict-2.12.10-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4eeb8c744c7f83d979e51c297fe39d7aad89fd1f4bc6fb63a45a5da13a1559eb |
|
MD5 | 646a99fc3d5cae7cdf2139237e29cab6 |
|
BLAKE2b-256 | 02c33ca0299323b3b5dbe665c14e65262dc81d4fcf3dd6a6a949220881f9108b |
Hashes for hugedict-2.12.10-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cc7d7077524efa702352f91822f2b69e723233f6eefd711be903c820356646ce |
|
MD5 | 79ac8de3b032ccbdb4425fe002acfcee |
|
BLAKE2b-256 | 9d5499cf46a109008ce25e20ee3dbd4bf558723af101fad027a8979b6b4ae9eb |
Hashes for hugedict-2.12.10-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8372ea95953e3e7a5bf0e8a32b7ba093fe56df8a00d8c926721717d61535541f |
|
MD5 | e0c7a238608124f17f37c3a074105ead |
|
BLAKE2b-256 | 5f47e2cd833a9b8ae6863107ea240f98a9204f8a40044cd95b9ea5c054693712 |
Hashes for hugedict-2.12.10-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f8fd94829eb142fccac774081f1d22e4cf735bd8b481538872d91c952da2d38 |
|
MD5 | 77269289bfae6eaa8911192a65dffa88 |
|
BLAKE2b-256 | 97c0b7d9d97f2bcb6b82b74d3b150c3d95d908bd64dc08d378a5da712f1eda7a |
Hashes for hugedict-2.12.10-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1294a5ed7859eb6392c7fac6c5fa6bfc6b9ff98cbfa0de232c9204f175f75493 |
|
MD5 | 7d9b3918ae771025e6ec4e60012c508d |
|
BLAKE2b-256 | 82a2756e77da2f62ab050ae00c44a83e3a60222ec98dc2a5171f9ac3812cdbe5 |
Hashes for hugedict-2.12.10-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4de62cffce90b1177fd9b36c9a6d5299f9bf2ca5529c90f7e4a91b9cfefbcb78 |
|
MD5 | e7c462b585a81b9faf4aa972982f2fde |
|
BLAKE2b-256 | 8655206bfea9db70b366b6878bd54b7dd9cc2958386c55f46cbb3cd4d64c4421 |
Hashes for hugedict-2.12.10-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9a28a3b9f6609dfcfe0ba450513fc86fd56bea576ff715a0fcd01783c9ff536e |
|
MD5 | 4cabbfb5922dd6dd1f258ff89abc96ad |
|
BLAKE2b-256 | 39432861ece1efda172872fe387d981782efd744a623343c7303b0e1c3963032 |
Hashes for hugedict-2.12.10-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9f92506d4fff38e08c3e976243211393a0b90b4d9df83e81a27dded0a27ebfa3 |
|
MD5 | 30831d9672b0ec1a10b8cbec555af120 |
|
BLAKE2b-256 | 2364c4b5aed4b78a16537021597a0e33abb7f95154b2aaff52fd0c5240fe1805 |
Hashes for hugedict-2.12.10-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 07c738530067101169e09da44a734f6c2a5002ad8c12f0f89dda2f20685b7223 |
|
MD5 | 5596a980d62dacb0d149ee320ee0db0d |
|
BLAKE2b-256 | ffbbe4f13162201c9b173601d689fcd85eb3718b7b7c465af36ecfff864c2ff2 |
Hashes for hugedict-2.12.10-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 74399ec180f3b0b8b2d1a227e6be204b4547043d87ed7bb059dd34c8999a3ee4 |
|
MD5 | 7c1356fd4b4c07323d00db69f6de0894 |
|
BLAKE2b-256 | a702ed945bd8104d18996a0d9fcb7e5f619d385e60d34a831047bbf36d7049a3 |