A Cron-like system for running tasks
Project description
# CronQ
A cron-like system to run your application tasks across any node, instead of one
special snowflake. This is done by keeping your tasks in MySQL and publishing
them over AMQP to workers that will run your tasks and eventually save the
results back into the DB. This was started as a hackathon project at
[SeatGeek](http://seatgeek.com)
# Installation
pip install cronq
Workers that need to be started
* `cronq-runner` - executes each task
* `cronq-injector` - checks for new tasks in the DB and publishes to AMQP
* `cronq-results` - Saves events from the runner
These take the envvars `RABBITMQ_HOST` and `CRONQ_MYQL`. `CRONQ_MYQL` should
be a MySQL Connector DSN starting with `mysql+mysqlconnector://`.
`cronq-runner` will also use the variable `CRONQ_QUEUE` to determine which
queue to consume. The default is `cronq_jobs`.
The web view is a WSGI app run from `cronq.web:app` and requires the same envvars.
## Runner
The runner requires `/var/log/cronq/` to exist and be writable by the user
executing the runner.
## Categories
Categories allow you to replace a set of jobs with a single API call
```
curl -v 'localhost:5000/api/category/example' -f -XPUT -H 'content-type: application/json' -d '
{
"category": "example",
"jobs": [{
"name": "Test Job",
"schedule": "R/2013-05-29T00:00:00/PT1M",
"command": "sleep 10",
"routing_key": "slow"
}]
}'
```
This adds / updates a job named `Test Job` in the `example` category. The time
format is ISO 8601. Any jobs no longer defined for the example category will be
removed. This allows you to script job additions / removes in your VCS.
A cron-like system to run your application tasks across any node, instead of one
special snowflake. This is done by keeping your tasks in MySQL and publishing
them over AMQP to workers that will run your tasks and eventually save the
results back into the DB. This was started as a hackathon project at
[SeatGeek](http://seatgeek.com)
# Installation
pip install cronq
Workers that need to be started
* `cronq-runner` - executes each task
* `cronq-injector` - checks for new tasks in the DB and publishes to AMQP
* `cronq-results` - Saves events from the runner
These take the envvars `RABBITMQ_HOST` and `CRONQ_MYQL`. `CRONQ_MYQL` should
be a MySQL Connector DSN starting with `mysql+mysqlconnector://`.
`cronq-runner` will also use the variable `CRONQ_QUEUE` to determine which
queue to consume. The default is `cronq_jobs`.
The web view is a WSGI app run from `cronq.web:app` and requires the same envvars.
## Runner
The runner requires `/var/log/cronq/` to exist and be writable by the user
executing the runner.
## Categories
Categories allow you to replace a set of jobs with a single API call
```
curl -v 'localhost:5000/api/category/example' -f -XPUT -H 'content-type: application/json' -d '
{
"category": "example",
"jobs": [{
"name": "Test Job",
"schedule": "R/2013-05-29T00:00:00/PT1M",
"command": "sleep 10",
"routing_key": "slow"
}]
}'
```
This adds / updates a job named `Test Job` in the `example` category. The time
format is ISO 8601. Any jobs no longer defined for the example category will be
removed. This allows you to script job additions / removes in your VCS.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
cronq-0.0.25.tar.gz
(126.1 kB
view hashes)