A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
- Create dictionary backed by Sqlite:
hugedict.sqlite.SqliteDict
- Chain multiple dictionaries:
hugedict.chained_mapping.ChainedMapping
- Cache a dictionary so previously accessed keys are stored in memory:
hugedict.cachedict.CacheDict
or calledhugedict.types.HugeMapping.cache
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.13.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3c70cb98d23a27a987ee60d4a9e5029c03024f790034bf5709404c59e97862a0 |
|
MD5 | 6f1d8be5a66e17cff50d433f9236db2e |
|
BLAKE2b-256 | 81c4b952e29b379938e38304632c8fd9f5d3c9e8b18e290663c30843fa5aea68 |
Hashes for hugedict-2.13.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3e9756ba90f0445527bc0a23e3707fcce584f6a57feec63c424754550573270b |
|
MD5 | da900c61a498ecf343fade57874c87f3 |
|
BLAKE2b-256 | 6ba2affe78fd9e94770e0469542c44db95b875f94085fe6e92f5351864c35bb4 |
Hashes for hugedict-2.13.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 396b5b3801b8038a068399f23aad28b6df3e8d106f41ef1696e80fe7acd55486 |
|
MD5 | bd39adcc2d2f890b97020a503116c656 |
|
BLAKE2b-256 | c3269e9149af9076aae696a76fb9be52e497bb21d9e94af972aa30c4cd766603 |
Hashes for hugedict-2.13.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 618ab779f6cd7c903116451c7e95e29a39b714adfec2136edce976061eca5e0a |
|
MD5 | f16d12aa6f134620959bd74beab75d4c |
|
BLAKE2b-256 | 23622e05178dbbc59ffd4ada19eb7428025be13094e5e14ada185ae29105513e |
Hashes for hugedict-2.13.1-cp312-cp312-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 06fe5f8f51cbc7434e6798ab0472cd6e83791ed060279859a30ece36edea4885 |
|
MD5 | 35ad1805b79d5e402b8850ae8fe6a420 |
|
BLAKE2b-256 | b4587de0df27396f04502b97c73613a2a1b48f27cfc67002151605e1029c1f9e |
Hashes for hugedict-2.13.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6d33f3068f36e9016927a30b268a7f06f069b9476adc89536931e649df6e69cb |
|
MD5 | de1348340aa71cbc26e9ead7e8e02302 |
|
BLAKE2b-256 | a3ee21181b4e854d8e388cfdc21e28b17888e234ac06b575d0f2e21a159e0929 |
Hashes for hugedict-2.13.1-cp312-cp312-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d84c9c6bad7e7f1650d10fe28e373de4ba7cab763e4ee84b0f04c8a11d48b9d1 |
|
MD5 | ef47f4638eff3f55feefc00708372ac0 |
|
BLAKE2b-256 | fd92ecc4707ccd698559272ecb442aa855681225dbf9d4dbc3cd9672b13409bc |
Hashes for hugedict-2.13.1-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 016735f4240813d00857a41b863832a3225b66d7689a89d08db536b78afc0763 |
|
MD5 | fca10f8a41871ceb0de864455745dec4 |
|
BLAKE2b-256 | 120980e53aebe5c10dd3bb25f3a05e00d292c8700f442e6d5d366bdaf39c441c |
Hashes for hugedict-2.13.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | cf3d60291f40e76c3755b65eb3e7f19d36cf163fe19b9e8e2992d0f2cdd5c8ba |
|
MD5 | 9f151a59759739d81846d33a7577717a |
|
BLAKE2b-256 | f31ed9ce74f95462247b9c3114e2185d2c1451cbb33278c9ee41801ab14e8bca |
Hashes for hugedict-2.13.1-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 87bbf129bd998a7c632f76aa04b1213eb83a08423a16cb059fe9724fd6c8d099 |
|
MD5 | 9a1cb4647ce1cfd618b8f07c68118a83 |
|
BLAKE2b-256 | 8f0ed7598cd7bf3b56872ad1245bb05775b5583df05204ee334560d40daa3ffa |
Hashes for hugedict-2.13.1-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b9964b611aa8cb2af23dc184d16d3eb2a741e2835c5f5be55870b0035ab245a4 |
|
MD5 | bbedd28ad5b959bb06dd6c469261e9b3 |
|
BLAKE2b-256 | e5cf8db7c7ee8b547707c99945a069e494014fc63981b18c037d820fc98ec694 |
Hashes for hugedict-2.13.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f659b5d9b6d549e1fa4d665f3952a4c35d5b3b335c1a3a4e8db0dd83e2414420 |
|
MD5 | 3316a17b54d285e65c64f667e32592d2 |
|
BLAKE2b-256 | f5ba3ea54fc628bd9a0c4d0ff40efdcdce5f8f0c89226a91d6aa0e0f6ff1a70f |
Hashes for hugedict-2.13.1-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc3ba0b77590d5e4ab6e40b8c38eeb2a8c5f983a3624fac03a76b03b19925e40 |
|
MD5 | 6ad3a6d3a5985c8662afb12cd092e725 |
|
BLAKE2b-256 | e70f8e37f5b3210f72ba08bd0e66863d93202f195d6906657c111b76612fe30c |
Hashes for hugedict-2.13.1-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f44c4b64f9515cb4b26dc235f7259215e3aa3a9327a8e6f05fa140de79b9d166 |
|
MD5 | 4090ee12a460ea08d124482443455532 |
|
BLAKE2b-256 | 2c8648280567a8a4fdd083dd53eacda76c96406576e7cdd65c51d93a65db1910 |
Hashes for hugedict-2.13.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 755641119781a02fdb9b2b5ae12dcefa6114b13617d87e3052dd311c6ca9d617 |
|
MD5 | bf553b080c8c1bdd5d68a9190434791f |
|
BLAKE2b-256 | 6d08cbf56f56164c163255a285b8a317157372d938224d3b0dd3d08f2fff7686 |
Hashes for hugedict-2.13.1-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1190bd429c35d7b82de8abec895f4712da16af544f73577f4816df2e8907d430 |
|
MD5 | 3e2ce46be1b1ba4860fec77d9070cb2e |
|
BLAKE2b-256 | ee3adab46acf01b8edbab93d781f80f6050423f5d1eb72ba195420a7f2a4ad4c |
Hashes for hugedict-2.13.1-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 76b274069b563ad2407755421fc61c6471d17055b288d47f19df9b4f08032f67 |
|
MD5 | b0b88b64b11905265156b4418e95f108 |
|
BLAKE2b-256 | ac04fbfb361a7fc5f731b33cf595ecbd321800282b1c37fae1a1b785586e9589 |
Hashes for hugedict-2.13.1-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 258967aa69d47024b14380d7d196421b1ca963df49902a80765620b6440ced08 |
|
MD5 | b05929b06164d7dcf6ac1799d1b9593b |
|
BLAKE2b-256 | fc986a8dd84e8e305a9465461d9763684fe0ad8795bc6e7d4441585a5d5c106c |
Hashes for hugedict-2.13.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fdac61cc2c1420ba80907d9d20981915fae7ed1001f3c48beb8ccbd26c0ff05d |
|
MD5 | c171549586b7af971bdb8ffbda7170ee |
|
BLAKE2b-256 | 947541b7517c4e450eccb6b04661a89f7453725f830f06018735c7fa7220080d |
Hashes for hugedict-2.13.1-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 366ee087e75495d9deff8e7cce6af4d227f6b3f782be9eedea2a5a570fc77d5b |
|
MD5 | 65ba8ea7007462df5d2fb9426408cb89 |
|
BLAKE2b-256 | 323f9c6c03e1c3c32697b309df1c4e6bc6379050f1c03d4a77d9ccd176d8cc0e |