Skip to main content

Celery Crossover aims to make it really easy to execute tasks in another service.

Project description

Build Status PyPI PyPI Coverage Status

Celery Crossover (Alpha)

About

Celery crossover aims to make it easier to execute celery tasks from a diffent source code base in the most simple and yet reliable way.

Use Case

Quick Examples

1) Simple Alice task execution triggered by Bob

Lets suppose Bob is a service that needs Alice to execute the task plus defined on Alice with the following code:

Alice:

  • celery_config.py
# -*- coding: utf-8 -*-
from kombu import Exchange, Queue

BROKER_URL = 'redis://localhost:6379/1'
CELERY_QUEUES = [
    Queue('alice_queue', Exchange('alice_queue'), routing_key='alice_queue')
]
  • alice.py
# -*- encoding: utf-8 -*-
import crossover
from celery import Celery

app = Celery('tasks')
app.config_from_object('celery_config')

# The line bellow will make Alice's Tasks usable by other services.
crossover.register_router(app)


@app.task(name='plus', queue='alice_queue')
def plus(x, y):
    _add = x + y
    return _add

Bob:

All that Bob need to do is:

  • exec_task_on_alice.py
# -*- encoding: utf-8 -*-
from crossover import Client

alice_broker = "redis://localhost:6379/1"
alice_client = Client(broker=alice_broker)

alice_client.plus(x=340, y=210)

2) Alice task execution triggered by Bob with callback (Auto Callback)

In the same scenario described above, lets suppose Bob need to be notified (have a task executed) after Alice is done with the plus task. For this case, all we have to do is decorate the Alice's task plus with@crossover.callback(auto_callback=True) to have its returned value sent back to Bob. Also, Bob have to define and send to Alice a task to be called by Alice's callback. That way, Alice and Bob could would be:

Alice:

  • alice_celery_config.py
# -*- coding: utf-8 -*-
from kombu import Exchange, Queue

BROKER_URL = 'redis://localhost:6379/1'
CELERY_QUEUES = [
    Queue('alice_queue', Exchange('alice_queue'), routing_key='alice_queue')
]
  • alice.py
# -*- encoding: utf-8 -*-
import crossover
from celery import Celery

app = Celery('tasks')
app.config_from_object('alice_celery_config')

# The line bellow will make Alice's Tasks usable by other services.
crossover.register_router(app)


@app.task(name='plus', queue='alice_queue')
@crossover.callback(auto_callback=True)
def plus(x, y):
    _add = x + y
    return _add

Bob:

  • bob_celery_conf.py
# -*- coding: utf-8 -*-
from kombu import Exchange, Queue

BROKER_URL = 'redis://localhost:6379/1'

CELERY_QUEUES = [
    Queue('bob_queue', Exchange('bob_queue'), routing_key='bob_queue')
]
  • bob.py
# -*- encoding: utf-8 -*-
import crossover
from celery import Celery
from celery.utils.log import get_task_logger

logger = get_task_logger(__name__)

app = Celery('tasks')
app.config_from_object('bob_celery_conf')

crossover.register_router(app)

@app.task(name='plus_callback', queue='bob_queue')
def plus_callback(result):
    logger.info('Got Addition callback = {0}'.format(result))
  • exec_task_on_alice.py
# -*- encoding: utf-8 -*-
from crossover import Client
from bob import plus_callback

alice_broker = "redis://localhost:6379/1"
alice_client = Client(broker=alice_broker)

alice_client.plus(x=340, y=210, callback=plus_callback)

3) Alice task execution triggered by Bob with callback (No Auto Callback)

In this case, everything is the same as 2) from Bob's perspective but lets suppose Alice's task cant calculate or determine a response right away so it needs to pass (or persist) the callback metadata for further execution. It can be done by using de decorator @crossover.callback with bind_callback_meta=True which will give to the task function the callback metadata as its first parameters. Following an example of its usage:

  • alice.py
# -*- encoding: utf-8 -*-
...

@app.task(name='plus', queue='alice_queue')
@crossover.callback(bind_callback_meta=True)
def plus(callback_meta, x, y):
    logger.info('Execution actual addition task.')
    calculate_addition.delay(callback_meta, x, y):


@app.task(name='calculate_addition', queue='project_1')
def calculate_addition(callback_meta, x, y):
    _add = x + y
    logger.info('Addition {0} + {1} = {2}'.format(x, y, _add))
    crossover.CallBack(callback_meta)(result=_add)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

celery-crossover-1.1.16.tar.gz (28.7 kB view details)

Uploaded Source

Built Distribution

celery_crossover-1.1.16-py3-none-any.whl (6.8 kB view details)

Uploaded Python 3

File details

Details for the file celery-crossover-1.1.16.tar.gz.

File metadata

  • Download URL: celery-crossover-1.1.16.tar.gz
  • Upload date:
  • Size: 28.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.5.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.5

File hashes

Hashes for celery-crossover-1.1.16.tar.gz
Algorithm Hash digest
SHA256 654af875548a11222f226d03f92beab1d3e9d93dd2ecf69aa8db6b3fdbeaf749
MD5 959a95d38a3d5975f8c5edbd7533db47
BLAKE2b-256 fbb8efa4c35a27f6687e042693451ebc2eba486f0eee4777e394df459bb928b5

See more details on using hashes here.

File details

Details for the file celery_crossover-1.1.16-py3-none-any.whl.

File metadata

  • Download URL: celery_crossover-1.1.16-py3-none-any.whl
  • Upload date:
  • Size: 6.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.1 importlib_metadata/4.5.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.61.1 CPython/3.9.5

File hashes

Hashes for celery_crossover-1.1.16-py3-none-any.whl
Algorithm Hash digest
SHA256 19378f8d57aeac13c93ba4be4f1ec5341353af355913fc7eae2bd94dcf436da5
MD5 54ad2b34cf83ad7961516547ae1329e2
BLAKE2b-256 c76ebf941187e1ab1791649837c3aeadbe4a88eaa8f7e7b5d4e9d10523fac658

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page