Distributed Task Queue - backed by Kafka
Project description
Kueue
Python Distributed Task Queue - backed by Kafka
Motivation
Kueue was born out of the lack of a library that leverages Kafka for distributed task processing
Features
- simple
- intuitive api
- extensible
Code Example
import time
from kueue import task, TaskExecutorConsumer, KueueConfig
KueueConfig(
kafka_bootstrap='localhost:9092'
)
@task(topic="my-topic")
def sleepy_task(sleep: int):
time.sleep(sleep)
print("done sleeping", sleep)
return sleep
sleepy_task.enqueue(args=(15,))
consumer = TaskExecutorConsumer(["my-topic"])
consumer.start()
# prints "done sleeping, 15"
Installation
pip install kueue
Development
Install poetry and run poetry install
at the root of the repository. This should create a virtual environment with all the necessary python dependencies.
Tests
The test framework makes heavy use of pytest
fixtures in order to spin up full integration environment consisting of a kubernetes cluster using kind and pytest-kind and kafka using strimzi
pytest
License
MIT © Jose Rojas
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file kueue-0.1.2.tar.gz
.
File metadata
- Download URL: kueue-0.1.2.tar.gz
- Upload date:
- Size: 7.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.1.4 CPython/3.9.0 Darwin/19.6.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 773e007ba5232ac0f6119b0c347baca2721e293aeac2aa0f5a38f4796f90589c |
|
MD5 | 3d1bce2341c98c0cbcfb6263129e5251 |
|
BLAKE2b-256 | c81bd501c6ead2eb0fa51a366404d0de94e5e7cc867c899c8eb6528391d4dcdb |
File details
Details for the file kueue-0.1.2-py3-none-any.whl
.
File metadata
- Download URL: kueue-0.1.2-py3-none-any.whl
- Upload date:
- Size: 7.6 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.1.4 CPython/3.9.0 Darwin/19.6.0
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5be83b9f4621d256662990e28db8dc91603901efd7abe551582f9dab170b913f |
|
MD5 | b7844009b84c159366d139aa3b128320 |
|
BLAKE2b-256 | 82e7574e3b30042541b16c85e5afaa60d714eeb33594728cc3a49eda303efc15 |