A service for handling deferred and periodic tasks on beanstalk worker machines
An app for handling deferred and periodic tasks on beanstalk worker machines
- Installation and Setup
- Development, Testing and the FakeTaskServer
The core of this app is the task decorator.
When a decorated task function is called instead of running immediately an SQS message is queued. When unqueued by a worker that message will cause the original function to be called.
The arguments may be any combination of JSON serializable types, datetimes and decimals, including keyword arguments. Any other argument types will cause an error.
Queuing of SQS messages happens in transaction on_commit so if you exception out of a transaction any tasks called in that transaction will not be run.
Task functions can also be invoked by cron or a management command as needed.
Installation and Setup
1: Installed Apps
2: Environment Variables
Software section of the Beanstalk configuration of your worker environment add the environment variable
WORKER and set it to
3: Django Settings
Add the following settings to your Django settings:
BEANSTALK_WORKER = bool(os.environ.get("WORKER", False)) BEANSTALK_TASK_SERVICE = "beanstalk_worker.services.TaskService" BEANSTALK_SQS_URL = <SQS queue URL> BEANSTALK_SQS_REGION = <Amazon region>
You can find the SQS queue URL in the
Worker section of the Beanstalk environment configuration.
For test and development you can omit the SQS settings and should set
BEANSTALK_TASK_SERVICE = "beanstalk_worker.services.FakeTaskService"
In your top level URL's add
if settings.BEANSTALK_WORKER: urlpatterns.append(url(r"^tasks/", include("beanstalk_worker.urls")))
This will add the URLS
/tasks/cron. You can move the base of these URL's if you do so other instructions on this page will need updating appropriately.
DO NOT include these URL's in a production web server, only the worker.
For test and development you won't have seperate web and worker machines so always include the URL's.
5: Beanstalk Worker Configuration
Worker section of the Beanstalk configuration of your worker environment set
HTTP path to
MIME type to
Declare a task function
from beanstalk_worker import task @task def my_task(message="hi"): print(message)
Call the task from anywhere in your code
This will queue an SQS message instructing a worker to run the actual function.
The arguments may be any combination of JSON serializable types, datetimes and decimals, including keyword arguments
Call the task from CRON
In cron.yaml add
- name: "my_project.my_app.tasks.my_task" url: "/tasks/cron/" schedule: "0 0 * * *"
my_project.my_app.tasks.my_task should be replaced with the fully qualified name of your task function.
Arguments are not currently supported for cron.
Call the task from the command line
./manage.py run_task my_project.my_app.tasks task "['hello world']"
my_project.my_app.tasks my_task should be replaced with the fully qualified name of your task function, also note the space between module name and function name.
Development, Testing and the FakeTaskServer
The FakeTaskServer will internally queue tasks but will not run them unless instructed to.
DEBUG = True is set in the Django settings an additional URL is exposed at
/tasks/run_all/ if you poke this URL the
FakeTaskService will run all queued tasks. This can be handy for local development.
In tests you can acquire the running task service instance with
from beanstalk_worker import task_service. This class has two helpers
clear whcih will discard all queued tasks and
run_all which will immediately run all queued tasks.
While running a task the FakeTaskServer will patch
True. This lets you
assert settings.BEANSTALK_WORKER in code that should only ever be run on a worker.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size django_beanstalk_worker-0.0.1-py3-none-any.whl (6.7 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size django-beanstalk-worker-0.0.1.tar.gz (5.3 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for django_beanstalk_worker-0.0.1-py3-none-any.whl
Hashes for django-beanstalk-worker-0.0.1.tar.gz