Skip to main content

Distributed task queue with full async support

Project description

PyPI - Python Version PyPI PyPI - Downloads


Documentation: https://taskiq-python.github.io/

What is taskiq?

Taskiq is an asynchronous distributed task queue for python. This project takes inspiration from big projects such as Celery and Dramatiq. But taskiq can send and run both the sync and async functions, has integration with popular async frameworks, such as FastAPI and AioHTTP.

Also, we use PEP-612 to provide the best autosuggestions possible. All code is type-hinted.

Installation

This project can be installed using pip:

pip install taskiq

Or it can be installed directly from git:

pip install git+https://github.com/taskiq-python/taskiq

Usage

At first you need to create a broker. Broker is an object that can communicate to workers using distributed queues.

We have different brokers for different queue backends. For example, we have a broker for NATS, Redis, RabbitMQ, Kafka and even more. Choose the one that fits you and create an instance.

from taskiq_nats import JetStreamBroker

broker = JetStreamBroker("nats://localhost:4222", queue="my_queue")

Declaring tasks is as easy as declaring a function. Just add a decorator to your function and you are ready to go.

import asyncio

from taskiq_nats import JetStreamBroker

broker = JetStreamBroker("nats://localhost:4222", queue="my_queue2")


@broker.task
async def my_task(a: int, b: int) -> None:
    print("AB", a + b)


async def main():
    await broker.startup()

    await my_task.kiq(1, 2)

    await broker.shutdown()


if __name__ == "__main__":
    asyncio.run(main())

The message is going to be sent to the broker and then to the worker. The worker will execute the function. To start worker processes, just run the following command:

taskiq worker path.to.the.module:broker

Where path.to.the.module is the path to the module where the broker is defined and broker is the name of the broker variable.

If you have tasks in different modules, you can ask taskiq to automatically import them by passing the --fs-discover flag:

taskiq worker path.to.the.module:broker --fs-discover

It will import all modules called tasks.py in the current directory and all subdirectories.

Also, we support hot reload for workers. To enable it, just pass the --reload flag. It will reload the worker when the code changes (To use it, install taskiq with reload extra. E.g pip install taskiq[reload]).

Also, we have cool integrations with popular async frameworks. For example, we have an integration with FastAPI or AioHTTP. You can use it to reuse dependencies from your web app in your tasks.

Read about all features in our documentation: https://taskiq-python.github.io/

Local development

Linting

We use pre-commit to do linting locally.

After cloning this project, please install pre-commit. It helps fix files before committing changes.

pre-commit install

Testing

Pytest can run without any additional actions or options.

pytest

Docs

To run docs locally, you need to install yarn.

First, you need to install dependencies.

yarn install

After that you can set up a docs server by running:

yarn docs:dev

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

taskiq-0.11.18.tar.gz (54.3 kB view details)

Uploaded Source

Built Distribution

taskiq-0.11.18-py3-none-any.whl (79.6 kB view details)

Uploaded Python 3

File details

Details for the file taskiq-0.11.18.tar.gz.

File metadata

  • Download URL: taskiq-0.11.18.tar.gz
  • Upload date:
  • Size: 54.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.1.3 CPython/3.12.3 Linux/6.11.0-1018-azure

File hashes

Hashes for taskiq-0.11.18.tar.gz
Algorithm Hash digest
SHA256 b83e1b70aee74d0a197d4a4a5ba165b8ba85b12a2b3b7ebfa3c6fdcc9e3128a7
MD5 c47857fca5049601a8804b7ec6cbe178
BLAKE2b-256 914d0d1b3b6c77a45d7a8c685a9c916b2532cca36a26771831949b874f6d15c3

See more details on using hashes here.

File details

Details for the file taskiq-0.11.18-py3-none-any.whl.

File metadata

  • Download URL: taskiq-0.11.18-py3-none-any.whl
  • Upload date:
  • Size: 79.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/2.1.3 CPython/3.12.3 Linux/6.11.0-1018-azure

File hashes

Hashes for taskiq-0.11.18-py3-none-any.whl
Algorithm Hash digest
SHA256 0df58be24e4ef5d19c8ef02581d35d392b0d780d3fe37950e0478022b85ce288
MD5 fd4d796d1f0bfc7fb2dced9f1533e367
BLAKE2b-256 4ad546505f57c140d10d4c36f6bd2f2047fb0460e4d5b9b841dc3b93ab8c893d

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page