parallel-iterator 1.0.0

A parallel iterator execution library for Python

parallel-iterator

English | 中文

A parallel iterator execution library for Python, providing various executors for concurrent and parallel task processing.

Installation

pip install parallel-iterator

Or install from source:

git clone https://github.com/lh9171338/parallel-iterator.git
cd parallel-iterator
pip install -e .

Quick Start

import time
from parallel_iterator import ProcessExecutor

# Define a task function
def process(a, b, delay=0.1):
    time.sleep(delay)
    return a + b

# Prepare data
a_list = [1, 2, 3, 4, 5]
b_list = [10, 20, 30, 40, 50]

# Execute in parallel using multiple processes
executor = ProcessExecutor(process, nprocs=4)
results = executor.run(a_list, b_list, delay=0.1)

print(results)  # [11, 22, 33, 44, 55]

Executors

Sequential Execution

Executor	Description
SequentialExecutor	Execute tasks sequentially in a single process

Process-based Executors

Executor	Description
ProcessExecutor	Execute tasks in parallel using multiple processes
AutoProcessExecutor	Process executor that automatically falls back to sequential execution when nprocs=1
BoundedProcessExecutor	Process executor with bounded resource slots (e.g., GPU ports)
AutoBoundedProcessExecutor	Auto-switching bounded process executor

Thread-based Executors

Executor	Description
ThreadExecutor	Execute tasks in parallel using a thread pool
AutoThreadExecutor	Automatically fallback to sequential execution when nworkers=1

Async Executors

Executor	Description
AsyncExecutor	Execute async tasks concurrently in a single process
AsyncProcessExecutor	Execute async tasks across multiple processes
AutoAsyncProcessExecutor	Auto-switching async process executor

Parallel Executors (Data Splitting)

Executor	Description
TaskParallelExecutor	Split data and process each item with a single-task function
BatchParallelExecutor	Split data and process batches with a batch function
AutoTaskParallelExecutor	Auto-switching task parallel executor
AutoBatchParallelExecutor	Auto-switching batch parallel executor

Usage Examples

Using ProcessExecutor

from parallel_iterator import ProcessExecutor

def compute(x, y, op="+"):
    if op == "+":
        return x + y
    return x - y

executor = ProcessExecutor(compute, nprocs=4)
results = executor.run([1, 2, 3], [4, 5, 6], op="+")
# results: [5, 7, 9]

Using ThreadExecutor

from parallel_iterator import ThreadExecutor

def io_task(url, timeout=10):
    # Simulate I/O operation
    return f"Downloaded: {url}"

executor = ThreadExecutor(io_task, nworkers=8)
results = executor.run(["url1", "url2", "url3"], timeout=5)

Using AsyncExecutor

import asyncio
from parallel_iterator import AsyncExecutor

async def async_task(item, delay=0.1):
    await asyncio.sleep(delay)
    return item * 2

executor = AsyncExecutor(async_task)
results = executor.run([1, 2, 3, 4, 5], delay=0.1)
# results: [2, 4, 6, 8, 10]

Using BatchParallelExecutor

from parallel_iterator import BatchParallelExecutor

def process_batch(items_a, items_b, factor=1):
    return [a + b * factor for a, b in zip(items_a, items_b)]

executor = BatchParallelExecutor(process_batch, nprocs=4)
results = executor.run([1, 2, 3, 4], [10, 20, 30, 40], factor=2)
# results: [21, 42, 63, 84]

Executor Selection Guide

Scenario	Recommended Executor
CPU-bound tasks	ProcessExecutor
I/O-bound tasks	ThreadExecutor or AsyncExecutor
Need GPU/port binding	BoundedProcessExecutor
Async functions + multiprocessing	AsyncProcessExecutor
Batch data processing	BatchParallelExecutor
Single-task function + data splitting	TaskParallelExecutor
Uncertain about process count	Use Auto* series executors

API Reference

• API Reference (English)
• API 参考文档 (中文)

License

This project is licensed under the MIT License - see the LICENSE file for details.