Skip to main content

Limits the number of values generated by an iterator until acknowledged.

Project description

Run Status Coverage Badge Supported Python versions License


multiprocessing.pool.ThreadPool and multiprocessing.Pool have the wonderful imap and imap_unordered methods which allow to consume an iterable incrementally and will yield results as they become available.

This is wonderful as it allows to process inputs or results larger than available memory.

It has an important flaw which is that it does not limit the number of results that are not consumed. This causes it to use memory without bound.

This package implements a wrapper over an iterable which will limit the number of results generated until they are acknowledged. It allows a number of messages to be processed concurrently but no more than that.


it = itertools.count()           # The input iterable.
it = BoundedIterator(10, it)     # Allows concurrent processing of up to 10 values.
results = Pool().imap(str, it)   # Will begin consuming `it` and producing results.
time.sleep(5)                    # No matter what else we do, no more than 10 results will be available.
for res in results:              # Consume normally.
  it.processed()                 # Acknowledge a value was processed so that a new one can be generated.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bounded-iterator, version 0.1.2
Filename, size File type Python version Upload date Hashes
Filename, size bounded_iterator-0.1.2-py2.py3-none-any.whl (4.9 kB) File type Wheel Python version 3.6 Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page