This library is used to enable batch processing by implementing python multiprocessing
Project description
batch_processor
install
pip install batch_processor
quickstart
from threading import Thread
from batch_processor import BatchProcessor
def batch_func(batch):
return [v + v for v in batch]
def create_bulk_request(n, processor):
print(n, processor.process(n))
processor = BatchProcessor(batch_func, worker_num=2, batch_size=32)
threads = []
for i in range(200):
t = Thread(target=create_bulk_request, args=(i, self.processor))
t.start()
threads.append(t)
for t in threads:
t.join()
This piece of code generates discrete 200 incomming requests that can be processed in batches whose size is 32. There are 2 workers processing these requests. They take batches and double each integers in the batch, then return results in batches.
test
- run test cases
python tests/test_runner.py
- generate coverage
coverage run tests/test_runner.py
- generate coverage html
coverage html
build
build the distribution
python setup.py sdist bdist_wheel
upload to registry
python -m twine upload dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
batch_processor-0.0.3.tar.gz
(5.2 kB
view hashes)
Built Distribution
Close
Hashes for batch_processor-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | f1d5b81791fb1c697ef06d5bf6c9464b8c9dccf3acf0cc297b9fe6aed5aedc12 |
|
MD5 | de4a9f8335b09f7684a292e0cb870739 |
|
BLAKE2b-256 | b8d8be86cf2aa1f4f1e32f7721dfd182e5602f09fbd9987e8b4d6dea485bc185 |