parallel-iterator

English | 中文

A parallel iterator execution library for Python, providing various executors for concurrent and parallel task processing.

Installation

pip install parallel-iterator

Or install from source:

git clone https://github.com/lh9171338/parallel-iterator.git
cd parallel-iterator
pip install -e .

Quick Start

import time
from parallel_iterator import ProcessExecutor

# Define a task function
def process(a, b, delay=0.1):
    time.sleep(delay)
    return a + b

# Prepare data
a_list = [1, 2, 3, 4, 5]
b_list = [10, 20, 30, 40, 50]

# Execute in parallel using multiple processes
executor = ProcessExecutor(process, nprocs=4)
results = executor.run(a_list, b_list, delay=0.1)

print(results)  # [11, 22, 33, 44, 55]

Executors

Sequential Execution

Process-based Executors

Thread-based Executors

Async Executors

Parallel Executors (Data Splitting)

Usage Examples

Using ProcessExecutor

from parallel_iterator import ProcessExecutor

def compute(x, y, op="+"):
    if op == "+":
        return x + y
    return x - y

executor = ProcessExecutor(compute, nprocs=4)
results = executor.run([1, 2, 3], [4, 5, 6], op="+")
# results: [5, 7, 9]

Using ThreadExecutor

from parallel_iterator import ThreadExecutor

def io_task(url, timeout=10):
    # Simulate I/O operation
    return f"Downloaded: {url}"

executor = ThreadExecutor(io_task, nworkers=8)
results = executor.run(["url1", "url2", "url3"], timeout=5)

Using AsyncExecutor

import asyncio
from parallel_iterator import AsyncExecutor

async def async_task(item, delay=0.1):
    await asyncio.sleep(delay)
    return item * 2

executor = AsyncExecutor(async_task)
results = executor.run([1, 2, 3, 4, 5], delay=0.1)
# results: [2, 4, 6, 8, 10]

Using BatchParallelExecutor

from parallel_iterator import BatchParallelExecutor

def process_batch(items_a, items_b, factor=1):
    return [a + b * factor for a, b in zip(items_a, items_b)]

executor = BatchParallelExecutor(process_batch, nprocs=4)
results = executor.run([1, 2, 3, 4], [10, 20, 30, 40], factor=2)
# results: [21, 42, 63, 84]

Executor Selection Guide

API Reference

• API Reference (English)
• API 参考文档 (中文)

License

This project is licensed under the MIT License - see the LICENSE file for details.