celery-aio-pool 0.1.0rc8

Celery worker pool with support for asyncio coroutines as tasks

Celery AsyncIO Pool

Free software: GNU Affero General Public License v3+

Getting Started

Installation

Using Poetry (preferred)

poetry add celery-aio-pool

Using pip & PyPI.org

pip install celery-aio-pool

Using pip & GitHub

pip install git+https://github.com/the-wondersmith/celery-aio-pool.git

Using pip & A Local Copy Of The Repo

git clone https://github.com/the-wondersmith/celery-aio-pool.git
cd celery-aio-pool
pip install -e "$(pwd)"

Configure Celery

Option 1: Using Celery's Out-Of-Tree Worker Pool (preferred)

Configuring Custom Worker Pool

Celery now supports the configuration of out-of-tree worker pool classes, allowing you to configure custom pools like celery-aio-pool directly:

• Set the environment variable CELERY_CUSTOM_WORKER_POOL to the name of your desired worker pool implementation.

  • NOTE: The value must be formatted as package:class

  • export CELERY_CUSTOM_WORKER_POOL='celery_aio_pool.pool:AsyncIOPool'
    

• Start Celery with --pool=custom to use the configured pool:

  • celery worker --pool=custom --loglevel=INFO --logfile="$(pwd)/worker.log"
    

To verify the pool configuration, use celery inspect stats:

celery --app=your_celery_project inspect stats
->  celery@freenas: OK
    {
        ...
        "pool": {
           ...
            "implementation": "celery_aio_pool.pool:AsyncIOPool",
    ...

Option 2: Monkey-patching Celery (non-preferred) for older versions of Celery (<5.3)

Using celery-aio-pool's Provided Patcher

• Import celery_aio_pool in the same module where your Celery "app" is defined
• Ensure that the patch_celery_tracer utility is called before any other Celery code is called

"""My super awesome Celery app."""

# ...
from celery import Celery

# add the following import
import celery_aio_pool as aio_pool

# ensure the patcher is called *before*
# your Celery app is defined

assert aio_pool.patch_celery_tracer() is True

app = Celery(
    "my-super-awesome-celery-app",
    broker="amqp://guest@localhost//",
    # add the following keyword argument
    worker_pool=aio_pool.pool.AsyncIOPool,
)

Developing / Testing / Contributing

NOTE: Our preferred packaging and dependency manager is Poetry. Installation instructions can be found here.

Developing

Clone the repo and install the dependencies

$ git clone https://github.com/the-wondersmith/celery-aio-pool.git \
  && cd celery-aio-pool \
  && poetry install --sync

Alternatively, if you prefer not to use Poetry, celery-aio-pool is fully PEP-517 compliant and can be installed directly by any PEP-517-compliant package manager.

$ cd celery-aio-pool \
  && pip install -e "$(pwd)"

TODO: Coming Soon™

Testing

To run the test suite:

$ poetry run pytest tests/

Contributing

TODO: Coming Soon™