celery-aio-pool 0.1.0rc7

Celery worker pool with support for asyncio coroutines as tasks

Celery AsyncIO Pool

Free software: GNU Affero General Public License v3+

Getting Started

Installation

Using Poetry (preferred)

poetry add celery-aio-pool

Using pip & PyPI.org

pip install celery-aio-pool

Using pip & GitHub

pip install git+https://github.com/the-wondersmith/celery-aio-pool.git

Using pip & A Local Copy Of The Repo

git clone https://github.com/the-wondersmith/celery-aio-pool.git
cd celery-aio-pool
pip install -e "$(pwd)"

Configure Celery

Using celery-aio-pool's Provided Patcher (non-preferred)

• Import celery_aio_pool in the same module where your Celery "app" is defined
• Ensure that the patch_celery_tracer utility is called before any other Celery code is called

"""My super awesome Celery app."""

# ...
from celery import Celery

# add the following import
import celery_aio_pool as aio_pool

# ensure the patcher is called *before*
# your Celery app is defined

assert aio_pool.patch_celery_tracer() is True

app = Celery(
    "my-super-awesome-celery-app",
    broker="amqp://guest@localhost//",
    # add the following keyword argument
    worker_pool=aio_pool.pool.AsyncIOPool,
)

Using (Upcoming) Out-Of-Tree Worker Pool (preferred)

At the time of writing, Celery does not have built-in support for out-of-tree pools like celery-aio-pool, but support should be included starting with the first non-beta release of Celery 5.3. (note: PR #7880 was merged on 2022-11-15).

The official release of Celery 5.3 enables the configuration of custom worker pool classes thusly:

• Set the environment variable CELERY_CUSTOM_WORKER_POOL to the name of your desired worker pool implementation implementation.

  • NOTE: The value of the environment variable must be formatted in the standard Python/Celery format of package:class

  • % export CELERY_CUSTOM_WORKER_POOL='celery_aio_pool.pool:AsyncIOPool'
    

• Tell Celery to use your desired pool by specifying --pool=custom when running your worker instance(s)

  • % celery worker --pool=custom --loglevel=INFO --logfile="$(pwd)/worker.log"
    

To verify the pool implementation, examine the output of the celery inspect stats command:

% celery --app=your_celery_project inspect stats
->  celery@freenas: OK
    {
        ...
        "pool": {
           ...
            "implementation": "celery_aio_pool.pool:AsyncIOPool",
    ...

Developing / Testing / Contributing

NOTE: Our preferred packaging and dependency manager is Poetry. Installation instructions can be found here.

Developing

Clone the repo and install the dependencies

$ git clone https://github.com/the-wondersmith/celery-aio-pool.git \
  && cd celery-aio-pool \
  && poetry install --sync

Optionally, if you do not have or prefer not to use Poetry, celery-aio-pool is fully PEP-517 compliant and can be installed directly by any PEP-517-compliant package manager.

$ cd celery-aio-pool \
  && pip install -e "$(pwd)"

TODO: Coming Soon™

Testing

To run the test suite:

$ poetry run pytest tests/

Contributing

TODO: Coming Soon™