Python library for efficiently batch-processing workloads asynchronously with batch-size- and time-based flushes.
Project description
Timed Batch Worker
Python library for efficiently batch-processing workloads asynchronously with batch-size- and time-based flushes using short-circuiting events.
Installation
pip install timed-batch-worker
Usage
from timed_batch_worker import TimedBatchWorker
# Handle the batched items
def handle_batch(items):
http.post('/api/logs/bulk', data=items)
# Create a worker that flushes the queue when:
# 1. The batch contains 20 objects, or
# 2. Five seconds have passed since the last flush
worker = TimedBatchWorker(
handler=handle_batch,
flush_interval=5,
flush_batch_size=20,
)
# Worker started in separate thread to avoid throttling main event loop
# in case the batch handler includes synchronous workloads.
worker.start()
# Add objects to the current batch at any time
@app.get('/users/create')
def create_user():
...
# Hand off an object to the worker
worker.enqueue({
'event': 'USER_CREATED',
'data': user
})
...
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for timed_batch_worker-0.0.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fd87d810c3e01d603541ff6f289996b20b46f71e364e1aac7b060b0a198f6020 |
|
MD5 | a941d02efd883a3e978cd9a31a1868d2 |
|
BLAKE2b-256 | 9cd514865be26e7ade6dbf14a880d7ec7824c9bb9c2d67b2be0161d41cea1528 |