threaded_map_reduce 0.1.0

A threaded version of map reduce

threaded-map-reduce

A Python library that implements map, unordered map, and map-reduce using threads.

This library is intented to be performant for CPU-bound tasks. The map implementation has been tested to be much more performant than the map method of the futures.ThreadPoolExecutor class of the standar library.

---

Features

• Parallel map with deterministic order: map
• Parallel unordered map for maximum throughput: map_unordered
• Parallel map-reduce: map_reduce
• It groups items into chunks to reduce the parallelization overhead
• No external dependencies

---

Installation

pip install threaded-map-reduce

(Or, if you use uv:)

uv pip install threaded-map-reduce

---

Quick Start

1. Parallel map (ordered)

from threaded_map_reduce import map

def square(x):
    return x * x

nums = range(1000)
result = list(threaded_map(square, nums, num_computing_threads=4, chunk_size=100))
print(result[-10:])

---

2. Parallel map (unordered)

Faster, but order is not preserved:

from threaded_map_reduce import map_unordered

nums = range(1000)
result = list(map_unordered(square, nums, num_computing_threads=4, chunk_size=100))
print(sorted(result))
print(result[-10:])

---

3. Parallel map-reduce

Useful for reductions such as sums, counts, or any associative operation.

from operator import add
from threaded_map_reduce import map_reduce

def square(x):
    return x * x

nums = range(0, 1000)
result = map_reduce(square, add, nums,
                    num_computing_threads=4,
                    chunk_size=100)
print(result)

---

API Summary

threaded_map(map_fn, items, num_computing_threads, chunk_size)

Runs map_fn over every item in parallel and yields results keeping input order.

map_unordered(map_fn, items, num_computing_threads, chunk_size)

Same as above, but yields items in any order.

map_reduce(map_fn, reduce_fn, items, num_computing_threads, chunk_size)

Maps items in parallel, reduces mapped chunks using the provided reducer function, and returns a single result.

---

License

MIT License.