Skip to main content

multiprocessing_generator

A library to prefetch items from a Python generator in the background, using a separate process.

Install (no dependencies):

pip install multiprocessing_generator

Example:

from multiprocessing_generator import ParallelGenerator

def my_generator():
    while True:
        # ... download something long ...
        yield result

with ParallelGenerator(
   my_generator(),
   max_lookahead=100) as g:
     for elem in g:
          # ... do some heavy processing on that element ...

Up to 100 elements ahead of what is consumed will be fetched by the generator in the background, which is useful when the producer and the consumer do not use the same resources (for instance network vs. CPU).

The generator handles exceptions and more serious failures transparently.

Released under the MIT license.

See also: https://github.com/justheuristic/prefetch_generator

Metadata

Release files for multiprocessing-generator 0.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for multiprocessing-generator 0.3
File Size Uploaded Provenance Actions
multiprocessing-generator-0.3.tar.gz 4.6 kB Without provenance Details for multiprocessing-generator-0.3.tar.gz

Release files / multiprocessing-generator-0.3.tar.gz

Download URL multiprocessing-generator-0.3.tar.gz
Size 4.6 kB
Tags Source
SHA-256 checksum
How to use checksums
778895fd84f2ce8c7bb5420500e885c87a6b4092f65af6d079f9482696d93175
BLAKE2b-256 checksum
How to use checksums
5bb10c5a75bf2113101addd86a93aadc40a87c3807288f7c055523f3023403f3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.4.1 importlib_metadata/3.7.3 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.59.0 CPython/3.9.1+

Security for multiprocessing-generator 0.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.3 This release

1 release file No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page