concurrent-iterator

Intro

Classes to run producers (iterators) and consumers (coroutines) in a background thread/process.

There are many libraries to create pipelines with stages running in separate processes, a nice one is parallelpipe, but this library does something different. It will lift the entire pipeline up to the point of the Producer into a separate process or thread. It's a more coarse library but easier to integrate since things keep looking as normal generators.

Implementations

There are currently 3 implementations:

• dummy.Producer: non-concurrent implementation
• thread.Producer: uses a background thread to run the generator
• process.Producer: uses a background process to run the generator

dummy.Producer is useless in practice.

thread.Producer is useful for IO bound generators.

process.Producer is useful for CPU or IO bound generators. It has the complications of dealing with processes (different memory spaces, logging, etc). For logging, module multiprocessing-logging can be used.

Limitations: process.Producer/process.Consumer with generators, coroutines and other unpicklable objects require the fork start method. Python 3.14 changes the default on Linux from fork to forkserver, and spawn/forkserver cannot pickle generators (TypeError: cannot pickle 'generator' object). In those cases process.Producer now raises RuntimeError with a clear message. Use thread.Producer/thread.Consumer, a picklable iterable (e.g. iter([1,2,3])), or force fork via multiprocessing.set_start_method('fork', force=True) or multiprocessing.get_context('fork').Process/Queue where fork is available (Linux).

Usage

Basic example:

from concurrent_iterator.thread import Producer

...

items = Producer(slow_generator, maxsize=5)

for item in items:
    [Do some time consuming task]

In the previous example, while doing some time consuming task, the slow_generator will continue running in a background thread and will pre-calculate up to 5 values.