Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

a simple tool to compute arbitrary generator in a background thread

Project Description

#based on http://stackoverflow.com/questions/7323664/python-generator-pre-fetch

This is a single-function package that transforms arbitrary generator into a background-thead generator that prefetches several batches of data in a parallel background thead.

This is useful if you have a computationally heavy process (CPU or GPU) that iteratively processes minibatches from the generator while the generator consumes some other resource (disk IO / loading from database / more CPU if you have unused cores).

By default these two processes will constantly wait for one another to finish. If you make generator work in prefetch mode (see examples below), they will work in parallel, potentially saving you your GPU time.

We personally use the prefetch generator when iterating minibatches of data for deep learning with tensorflow and theano ( lasagne, blocks, raw, etc.).

This package contains two objects
  • BackgroundGenerator(any_other_generator[,max_prefetch = something])
  • @background([max_prefetch=somethind]) decorator

the usage is either

#for batch in BackgroundGenerator(my_minibatch_iterator): # doit()

or

#@background() #def iterate_minibatches(some_param): # while True: # X = read_heavy_file() # X = do_helluva_math(X) # y = wget_from_pornhub() # do_pretty_much_anything() # yield X_batch, y_batch

More details are written in the BackgroundGenerator doc help(BackgroundGenerator)

Release History

Release History

This version
History Node

1.0.0

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
prefetch_generator-1.0.0.tar.gz (3.2 kB) Copy SHA256 Checksum SHA256 Source Aug 13, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting