This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Celery CloudWatch

Monitor your celery application from within AWS CloudWatch!

Metrics

The following events are tallied per task:

  • CeleryEventSent
  • CeleryEventStarted
  • CeleryEventSucceeded
  • CeleryEventFailed

You can then see how many tasks/day, tasks/week etc are being completed.

Also, we record the total waiting/running per task:

  • CeleryNumWaiting
  • CeleryNumRunning

Also, statistics on task duration are sent in the metrics:

  • CeleryQueuedTime
  • CeleryProcessingTime

These metrics are sent with all supported stats (No. Events, Sum, Max, Min), allowing you to gain insight into your task processing and match requests and capacity.

Getting Started

  1. Set up an IAM Role for your instance.

    It must include a policy to perform ‘PutMetricData’, eg:

    {
      "Version": "2000-01-01",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": [
            "cloudwatch:PutMetricData"
          ],
          "Resource": [
            "*"
          ]
        }
      ]
    }
    

    (Note: Alternatively, you can set up a User with the same policy and provide access details that way)

  2. Install via python-pip (and upgrade pip & boto)

    sudo apt-get install -y python-pip
    sudo pip install --upgrade pip boto
    
    # Install directly
    sudo pip install celery-cloudwatch
    
    # OR, install in a virtualenv
    sudo apt-get install -y python-virtualenv
    mkdir /var/python-envs
    virtualenv /var/python-envs/ccwatch
    source /var/python-envs/ccwatch/bin/activate
    pip install celery-cloudwatch
    
  3. Create your own boto.cfg at /etc/boto.cfg-

    [Credentials]
    # if not using an IAM Role - provide aws key/secret
    aws_access_key_id = xxx
    aws_secret_access_key = yyy
    
    [Boto]
    cloudwatch_region_name = my-region
    cloudwatch_region_endpoint = monitoring.my-region.amazonaws.com
    
  4. Create your own config file in /etc/ccwatch.yaml

    ccwatch:
      broker: null
      camera: celery_cloudwatch.CloudWatchCamera
      verbose: no
    camera:
      frequency: 60.0
      verbose: no
    cloudwatch-camera:
      dryrun: no
      namespace: celery
      tasks:
        - myapp.mytasks.taskname
        - myapp.mytasks.anothertask
        - myapp.mytasks.thirdtask
        - name: myapp.secondarytasks
          dimensions:
            task: myapp.secondarytasks
            customDim: value
        - name: myapp.tertiarytasks
          dimensions:
            task: myapp.tertiarytasks
            customDim: value
    
  5. Install upstart

    Create a file /etc/init/celery-cloudwatch.conf-

    description "Celery CloudWatch"
    author "nathan muir <ndmuir@gmail.com>"
    
    setuid nobody
    setgid nogroup
    
    start on runlevel [234]
    stop on runlevel [0156]
    
    exec /var/python-envs/ccwatch/bin/ccwatch
    respawn
    

    then-

    sudo initctl reload-configuration
    sudo service celery-cloudwatch start
    
  6. Start Celery your celery workers with the -E (or CELERY_SEND_EVENTS=1 and CELERY_TRACK_STARTED=1) options, and, start celery clients with CELERY_SEND_TASK_SENT_EVENT=1

  7. All done! head over to your CloudWatch monitoring page to see the results!

Release History

Release History

1.1.0.post0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.0a

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
celery-cloudwatch-1.1.0.post0.tar.gz (10.9 kB) Copy SHA256 Checksum SHA256 Source Sep 22, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting