This library is used to enable batch processing by implementing python multiprocessing
Project description
batch_processor
install
pip install batch_processor
quickstart
from threading import Thread
from batch_processor import BatchProcessor
def batch_func(batch):
return [v + v for v in batch]
def create_bulk_request(n, processor):
print(n, processor.process(n))
processor = BatchProcessor(batch_func, worker_num=2, batch_size=32)
threads = []
for i in range(200):
t = Thread(target=create_bulk_request, args=(i, self.processor))
t.start()
threads.append(t)
for t in threads:
t.join()
This piece of code generates discrete 200 incomming requests that can be processed in batches whose size is 32. There are 2 workers processing these requests. They take batches and double each integers in the batch, then return results in batches.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
batch_processor-0.0.2.tar.gz
(4.8 kB
view hashes)
Built Distribution
Close
Hashes for batch_processor-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2e4b90de2b501e478f4b589e1a3a79fb6f488d682165f589d3be4dfebce2e3c3 |
|
MD5 | ff8cb03b7fd1e48b810fa8354b7111b0 |
|
BLAKE2b-256 | 42105072f6f5afb54ad3dd50fefb72fd9446364347f961639c78b0ce2f3eb58d |