Skip to main content

Prefetch elements from a Python generator in the background, from a separate process

Project description

A library to prefetch items from a Python generator in the background, using a separate process.

Install (no dependencies):

pip install multiprocessing_generator

Example:

from multiprocessing_generator import ParallelGenerator

def my_generator():
    while True:
        # ... download something long ...
        yield result

with ParallelGenerator(
   my_generator(),
   max_lookahead=100) as g:
     for elem in g:
          # ... do some heavy processing on that element ...

Up to 100 elements ahead of what is consumed will be fetched by the generator in the background, which is useful when the producer and the consumer do not use the same resources (for instance network vs. CPU).

The generator handles exceptions and more serious failures transparently.

Released under the MIT license.

See also: https://github.com/justheuristic/prefetch_generator

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
multiprocessing-generator-0.2.tar.gz (3.9 kB) Copy SHA256 hash SHA256 Source None May 29, 2017

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page