Skip to main content

Distributed Task Queue - backed by Kafka

Project description

Kueue

Build

Python Distributed Task Queue - backed by Kafka

Motivation

Kueue was born out of the lack of a library that leverages Kafka for distributed task processing

Features

  • simple
  • intuitive api
  • extensible

Code Example

import time
from kueue import task, TaskExecutorConsumer, KueueConfig

KueueConfig(
    kafka_bootstrap='localhost:9092'
)

@task(topic="my-topic")
def sleepy_task(sleep: int):
    time.sleep(sleep)
    print("done sleeping", sleep)
    return sleep

sleepy_task.enqueue(args=(15,))

consumer = TaskExecutorConsumer(["my-topic"])
consumer.start()
# prints "done sleeping, 15"

Installation

pip install kueue

Development

Install poetry and run poetry install at the root of the repository. This should create a virtual environment with all the necessary python dependencies.

Tests

The test framework makes heavy use of pytest fixtures in order to spin up full integration environment consisting of a kubernetes cluster using kind and pytest-kind and kafka using strimzi

pytest

License

MIT © Jose Rojas

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

kueue-0.1.2.tar.gz (7.5 kB view details)

Uploaded Source

Built Distribution

kueue-0.1.2-py3-none-any.whl (7.6 kB view details)

Uploaded Python 3

File details

Details for the file kueue-0.1.2.tar.gz.

File metadata

  • Download URL: kueue-0.1.2.tar.gz
  • Upload date:
  • Size: 7.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.4 CPython/3.9.0 Darwin/19.6.0

File hashes

Hashes for kueue-0.1.2.tar.gz
Algorithm Hash digest
SHA256 773e007ba5232ac0f6119b0c347baca2721e293aeac2aa0f5a38f4796f90589c
MD5 3d1bce2341c98c0cbcfb6263129e5251
BLAKE2b-256 c81bd501c6ead2eb0fa51a366404d0de94e5e7cc867c899c8eb6528391d4dcdb

See more details on using hashes here.

File details

Details for the file kueue-0.1.2-py3-none-any.whl.

File metadata

  • Download URL: kueue-0.1.2-py3-none-any.whl
  • Upload date:
  • Size: 7.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.4 CPython/3.9.0 Darwin/19.6.0

File hashes

Hashes for kueue-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 5be83b9f4621d256662990e28db8dc91603901efd7abe551582f9dab170b913f
MD5 b7844009b84c159366d139aa3b128320
BLAKE2b-256 82e7574e3b30042541b16c85e5afaa60d714eeb33594728cc3a49eda303efc15

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page