batch-processor 0.0.3

This library is used to enable batch processing by implementing python multiprocessing

batch_processor

install

pip install batch_processor

quickstart

from threading import Thread
from batch_processor import BatchProcessor

def batch_func(batch):
    return [v + v for v in batch]

def create_bulk_request(n, processor):
    print(n, processor.process(n))

processor = BatchProcessor(batch_func, worker_num=2, batch_size=32)

threads = []
for i in range(200):
    t = Thread(target=create_bulk_request, args=(i, self.processor))
    t.start()
    threads.append(t)

for t in threads:
    t.join()

This piece of code generates discrete 200 incomming requests that can be processed in batches whose size is 32. There are 2 workers processing these requests. They take batches and double each integers in the batch, then return results in batches.

test

• run test cases

  python tests/test_runner.py
  

• generate coverage

  coverage run tests/test_runner.py
  

• generate coverage html

  coverage html
  

build

build the distribution

python setup.py sdist bdist_wheel

upload to registry

python -m twine upload dist/*