Skip to main content

A dictionary-like object that is friendly with multiprocessing and uses key-value databases (e.g., RocksDB) as the underlying storage.

Project description

hugedict PyPI Documentation

hugedict provides a drop-in replacement for dictionary objects that are too big to fit in memory. hugedict's dictionary-like objects implement typing.Mapping and typing.MutableMapping interfaces using key-value databases (e.g., RocksDB) as the underlying storage. Moreover, they are friendly with Python's multiprocessing.

Installation

From PyPI (using pre-built binaries):

pip install hugedict

To compile the source, run: maturin build -r inside the project directory. You need Rust, Maturin, CMake and CLang (to build Rust-RocksDB).

Features

  1. Create a mutable mapping backed by RocksDB
from functools import partial
from hugedict.prelude import RocksDBDict, RocksDBOptions

# replace [str, str] for the types of keys and values you want
# as well as deser_key, deser_value, ser_value
mapping: MutableMapping[str, str] = RocksDBDict(
    path=dbpath,  # path (str) to db file
    options=RocksDBOptions(create_if_missing=create_if_missing),  # whether to create database if missing, check other options
    deser_key=partial(str, encoding="utf-8"),  # decode the key from memoryview
    deser_value=partial(str, encoding="utf-8"),  # decode the value from memoryview
    ser_value=str.encode,  # encode the value to bytes
    readonly=False,  # open database in read only mode
    secondary_mode=False,  # open database in secondary mode
    secondary_path=None,  # when secondary_mode is True, it's a string pointing to a directory for storing data required to operate in secondary mode
)
  1. Load huge data from files into RocksDB in parallel: from hugedict.prelude import rocksdb_load. This function creates SST files in parallel, ingests into the db and (optionally) compacts them.

  2. Cache a function when doing parallel processing

from hugedict.prelude import Parallel

pp = Parallel()

@pp.cache_func("/tmp/test.db")
def heavy_computing(seconds: float):
    time.sleep(seconds)
    return seconds * 2


output = pp.map(heavy_computing, [0.5, 1, 0.7, 0.3, 0.6], n_processes=3)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

hugedict-2.9.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB view details)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

hugedict-2.9.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB view details)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

hugedict-2.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB view details)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

hugedict-2.9.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (7.2 MB view details)

Uploaded CPython 3.11 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

hugedict-2.9.0-cp310-none-win_amd64.whl (2.9 MB view details)

Uploaded CPython 3.10 Windows x86-64

hugedict-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB view details)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

hugedict-2.9.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (7.2 MB view details)

Uploaded CPython 3.10 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

hugedict-2.9.0-cp39-none-win_amd64.whl (2.9 MB view details)

Uploaded CPython 3.9 Windows x86-64

hugedict-2.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB view details)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

hugedict-2.9.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (7.2 MB view details)

Uploaded CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

hugedict-2.9.0-cp38-none-win_amd64.whl (2.9 MB view details)

Uploaded CPython 3.8 Windows x86-64

hugedict-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB view details)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

hugedict-2.9.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (7.2 MB view details)

Uploaded CPython 3.8 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

File details

Details for the file hugedict-2.9.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 33a55b4077ea650600ee0a53e52e393064c300bc1d0dbc94f30c58504e9110e8
MD5 27b91c75152f25064aac4a213c4cc5a6
BLAKE2b-256 7044c992f62209767bbbfef9920eefd824b824b5e4d6c292cd4cc8fa95f92d00

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 9c9bf6127b2c496c4d27355e8698d3205c94f63c68cba4abeed98fa67585c001
MD5 a96c89cb3750bcc45e867c411632947b
BLAKE2b-256 a4dde8196bf9ff757f38a37cf38ca194cff496a3d278b820ea97a6053c631c0f

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 cfb6fea7616daf11ab2ed1ae66bcdde9f42273c95e8df142f70f75d395d8d774
MD5 81cccf4fe495a33f5d26b05ada1e103f
BLAKE2b-256 c3863c24fbdde8cebd79633871d4df4f7053e3b7d34bb3b016ac9ddf5e24ffed

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp311-cp311-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 e26ec2e603eb4743a3b7416e21a25e90d76736b657de893cefe8ba15722d1af1
MD5 6a52a8e52bd36a239aa9138d3f74df89
BLAKE2b-256 5448ff723f389e728a9b00b3e4365f643ef4935dc212a90b1c697bb15f9534a0

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp310-none-win_amd64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp310-none-win_amd64.whl
Algorithm Hash digest
SHA256 4562fa895ae82eba4e0b196a0e882ef6b1f56fdc90624b8dcf20aff5feb0375e
MD5 d4deb84d5af4e2d7a06d6cd947255aa6
BLAKE2b-256 098897cf3cdb1401c69251d7ccb45fba90f85017b526a6240d49f4d154e92101

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 cdcac4be5e9569151addbb9ec1bab49b48f6a87bce0ef1c26319cb2116808cfd
MD5 9a62ea92b316ac1abc6c5de60c342b49
BLAKE2b-256 f039b204b15fb898670888e8d3cbf6e9e4678965c8ddb80f3e8a88a444919729

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp310-cp310-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 2f867b4c8b450a02475551257fce1862def7b4967bdefe233c866f24cb30b877
MD5 68abbc7f88ff5680215a9df413ff89c8
BLAKE2b-256 34c4db863c16c1d099d3d5283aa5f8d7ab07961498497d30b42da01c954d7023

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp39-none-win_amd64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp39-none-win_amd64.whl
Algorithm Hash digest
SHA256 1ec0ca912a7c3bff3897add8c2ca3be9de872fccd93d96a10d0ffc7ac549873b
MD5 1d59814c9d4d9205f8e0d23a3dbc508c
BLAKE2b-256 cf07703ae5fa376ef60d705c9a4ebd2a4b4a4ab8ee79afd3ee483b10f1aabc2f

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 d2e56dea881c0476110771678932c466da371f3ef0a071dd4800d6e172b861bd
MD5 17c4ad41fb9d7e3e5cab2e1d4ee60258
BLAKE2b-256 58eee6d69d6e39c6a0fc343ef0d44a3c4458f2cc4aa6cc5a234a697d852ba10c

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 c2ddf4bb2513c02a593c56f2516dd46f28cbe93278ec00e839d23eb38326689c
MD5 aff927d9fbb7b66bca602ea814e495cf
BLAKE2b-256 7a9720743237ebeb8efc67e8759f2f9d637a24e316604920eb2c99d4e97b30ea

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp38-none-win_amd64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp38-none-win_amd64.whl
Algorithm Hash digest
SHA256 f48fae86ab8d878d8a5fd3534377225efa196b7f030daa5012303f0d3f8e1fe3
MD5 8400d4ed2654d5a9a8ee8cfbb1227a2a
BLAKE2b-256 37c1700fb5bab6d6e9ca91a3f7163ed108d59fdda8737c9583461a9faf45980b

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e72d6d727bbb60f6db796ba908a4bd66fcde9b7fe4e83ffeee2531e9a2e938d3
MD5 80dc6e20cce1e972395fd7fddce487ef
BLAKE2b-256 b753df022a8c4ec0d5c22792c20232e9802940845654a5d71bc3366f8e5c3e17

See more details on using hashes here.

File details

Details for the file hugedict-2.9.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl.

File metadata

File hashes

Hashes for hugedict-2.9.0-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
Algorithm Hash digest
SHA256 2a6db782dfaa8e25377698a2d2debb73d9b58c7b79d34561978f572e19b715bd
MD5 232ac1cf8a78fd4b44e26fcba8f760db
BLAKE2b-256 263f483096cb6c368d138db82e610c71e9c331c9279eb6c6eec5bb3ea5001c07

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page