A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.
Project description
hugedict
hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping
and typing.MutableMapping
interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.
Installation
From PyPI (using pre-built binaries):
pip install hugedict
To compile the source, run: maturin build -r
inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).
Features
- Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions
# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
path=dbpath, # path (str) to db file
options=RocksDBOptions(create_if_missing=create_if_missing), # whether to create database if missing, check other options
deser_key=partial(str, encoding="utf-8"), # decode the key from memoryview
deser_value=partial(str, encoding="utf-8"), # decode the value from memoryview
ser_value=str.encode, # encode the value to bytes
readonly=False, # open database in read only mode
secondary_mode=False, # open database in secondary mode
secondary_path=None, # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
-
Load huge data from files into RocksDB in parallel:
from hugedict.prelude import rocksdb_load
. This function creates SST files in parallel, ingests into the db and (optionally) compacts them. -
Cache a function when doing parallel processing
from hugedict.prelude import Parallel
pp = Parallel()
@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
time.sleep(seconds)
return seconds * 2
output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for hugedict-2.12.2-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 93eaba0bbff3174d591b808966c343542e68f4a00c9711835da1534b4b70e082 |
|
MD5 | 60006a4b35e6cff28efba63f2e2ce5aa |
|
BLAKE2b-256 | ea05a928daa6a1898236e45b09284044277b52e03a2ff8f1d3bd69e5ee35cec3 |
Hashes for hugedict-2.12.2-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 640f673942df1d76187f78d1c3b7f2d59f60adeeee90785ce74a02227ee1aae9 |
|
MD5 | 45d5af7478d907f364a8e6483264294c |
|
BLAKE2b-256 | 68b810aa3e169aa4e8b1479c856714f6d8d9cfc290ce8255a4da22123126c21e |
Hashes for hugedict-2.12.2-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3222b5d363fcf0c1da218ea510d2019c80191604c3853285a42ded092766a87b |
|
MD5 | c700111e27f5148a95725a4506b3c282 |
|
BLAKE2b-256 | 870a6c96da19a5e4f09ce525db3b48280968321c779eea188146f0ecd961e211 |
Hashes for hugedict-2.12.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6e1ddf9efa7035be922a4d367449837d3c42b8b2c82fabef199cce8118901417 |
|
MD5 | 3e29e7eb2bf40929207bd58d8942987e |
|
BLAKE2b-256 | c130b4dbd3aedc55dd27028ca26670595abee6e4efb701f990d7a20772e6d79b |
Hashes for hugedict-2.12.2-cp311-cp311-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7428ef0eb57bc75e1a750716b9e7a0fb5c334e600ebc0f9e74a44b86fc49ebe |
|
MD5 | 4ed4cc7f9029d0c4441e32cd32a2009f |
|
BLAKE2b-256 | dee42008dc42e729b7091547e5467a7301fccc52b034db76a4ac0bb4c6279aaa |
Hashes for hugedict-2.12.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f28783903044c3dbc1254162ed59d2ff5c7cdf24497ce50d69aa764b056fe087 |
|
MD5 | ecd24cda4b75239c53e844cd72b36dfd |
|
BLAKE2b-256 | 168e746a299abace85f3aded7a6b3a12607dbb98d057d9f867699b048b99c3da |
Hashes for hugedict-2.12.2-cp311-cp311-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e5d5f83b49b714c27594a9855e46715f19a98a6ea2a30c6700d6379fc29df3c |
|
MD5 | 6599f4b3fe0820127a13661d9b396777 |
|
BLAKE2b-256 | f198714abdfbc7f0c72c063f42d7ad1bc298ed57304e900687363bbe23c9e27c |
Hashes for hugedict-2.12.2-cp310-cp310-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f36c774723e522cae7b361b91e29ea0682ecfaec5e564bed8277038e9946b81b |
|
MD5 | 6fa1d0f58f27fbaa93101a541c1f296a |
|
BLAKE2b-256 | 916168fa086648b3b0c699f1a05f496dd010013091f09e85255d8b286876f239 |
Hashes for hugedict-2.12.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f6fa89483ef46c508152dcefcc899f245020f936df23eb981a4db921242e28d |
|
MD5 | 5b29fce96800e5ab354d68efe462f179 |
|
BLAKE2b-256 | ff3607f9c1e0dec5fca68f238e04163ebb6a20b8b0a83e9ff2324faca6e3b123 |
Hashes for hugedict-2.12.2-cp310-cp310-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 398ff253967aee4c1e0b546cd6e13e6d2db657e808c799efab002c17e0050c56 |
|
MD5 | 84c66530419fd9564f340bc60fb41fdc |
|
BLAKE2b-256 | 940653f8bf991ef4435c907d7829b908dc4654de6f00d345adb99d55a01c532a |
Hashes for hugedict-2.12.2-cp39-cp39-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d8bc75f89acebb6edf0736040f0a2daa35bd8c26a6b82b80d8f9a90b2aa2eef1 |
|
MD5 | 0fa8912953aeb2a4ee7b48734b33e3f0 |
|
BLAKE2b-256 | ac4ed5fd5f2611c3d8b2446fe29d54e97efde985d86fcb344fef09478103ba54 |
Hashes for hugedict-2.12.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b2a4e5a2a6c66e801d36cc5ff4c8612e8dad3f8aa74a241c4445bc13af0df423 |
|
MD5 | 90e31b1771f7b5201acd8cc37cc62cd0 |
|
BLAKE2b-256 | f78eae9f91d0c08d2803dc37b267e32dda5234329bd164254020be197229484a |
Hashes for hugedict-2.12.2-cp39-cp39-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | e7d7af4045548f2526aa3440ad6439c5614dc220ec1484093b734e0404b74bda |
|
MD5 | 3da3063dbea5c60da10a4832f69f6865 |
|
BLAKE2b-256 | 8c590a0530ebd5399a6fef6a7ea52ab4c4436ede58aa6277beb39c0f8daebd32 |
Hashes for hugedict-2.12.2-cp38-none-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5b3d74651373435f3413119dec74e4392e524b6e35eb8e248e3c8dbb8add83b6 |
|
MD5 | b9aac6ce5f5e7ae044f31bb29bdd58c5 |
|
BLAKE2b-256 | 0938282ecca7d2df95dacfd15689ded3e10494715de299ad9621ec37a9450154 |
Hashes for hugedict-2.12.2-cp38-cp38-manylinux_2_35_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7ea00dd0fa87edffe9d973aff1d4bf461f87e096293d001efa36a04b5a30d900 |
|
MD5 | d62e2967418cc58447d4d050d75a4319 |
|
BLAKE2b-256 | 01f735517f307ab2b88437c4101eb80c46abc98d645496b0f77b6c5b551dcafe |
Hashes for hugedict-2.12.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c6c03880692380ffab36b36321c49700131f4b369277d41a2756a596c29bbf96 |
|
MD5 | 80f68809ad174e151cc9e057a1055ae1 |
|
BLAKE2b-256 | c26103246549d79f7b1dc088cb6134503be985b952e8a3d0c51d03296c4c6af5 |
Hashes for hugedict-2.12.2-cp38-cp38-macosx_10_14_x86_64.macosx_11_0_arm64.macosx_10_14_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a9284453c2221a2c89c842473d3eaa41bcc174ddc44970e98bc4714dd1b9f1be |
|
MD5 | 88a15c278c44f3e5490617aff52e02f8 |
|
BLAKE2b-256 | f5c465a79e602b537a866c245129a0f3b8db7afadf0baff06ac3351d9d238fcf |