Consumer/publisher loop for workers in an object recognition pipeline
Project description
catflow-worker
Consumer/publisher loop for workers in an object recognition pipeline
Setup
- Install pre-commit in your virtualenv. Run
pre-commit install
after cloning this repository.
Develop
Install:
pip install --editable .[dev]
Configure:
export CATFLOW_S3_ENDPOINT_URL="your-endpoint-url"
export CATFLOW_AWS_ACCESS_KEY_ID="your-access-key-id"
export CATFLOW_AWS_SECRET_ACCESS_KEY="your-secret-access-key"
export CATFLOW_AWS_BUCKET_NAME="your-bucket-name"
export CATFLOW_AMQP_URL="amqp://username:password@hostname:port/"
export CATFLOW_AMQP_EXCHANGE="catflow"
Run the sample loop:
$ python -m catflow_worker
[*] Message received (ingest.video): da0b7b0a-c5e9-44dd-b67a-4fef9c5d86bd.mp4
[-] Content-Type binary/octet-stream
[-] Content-Length 92598
[*] Message received (detect.video): da0b7b0a-c5e9-44dd-b67a-4fef9c5d86bd.mp4
[-] Content-Type binary/octet-stream
[-] Content-Length 92598
Build
pip install build
python -m build
Test
pytest
Usage
Define an async handler function:
async def example_handler(
msg: str, key: str, s3: Any, bucket: str
) -> Tuple[bool, List[Tuple[str, str]]]:
The handler function
- gets: a message, the key the message was sent to, an S3 client, the name of an S3 bucket to use
- and returns: an error status, and a list of (key, message) tuples to publish
Then start it like:
worker = await Worker.create(handler, queue_name, topic_key)
await worker.work()
See catflow_worker/main.py
for a working example.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
catflow-worker-0.1.4.tar.gz
(8.8 kB
view hashes)
Built Distribution
Close
Hashes for catflow_worker-0.1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | b269e40006eea1b82a2be5af8b9c0002078956aec18f71d18a7509793584d6b5 |
|
MD5 | 0302f70461869030360c3a6591fe7553 |
|
BLAKE2b-256 | 8adf7f71cabcce9ae61c4a68511d60c0b258352e4d1615dd388a7ce45f849796 |