This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Summary

asynchronous workers in your instance

How to use this cube

Add the worker cube to the dependencies of your cube. In your schema, extend the cube’s schema if necessary. Common extension often involve adding attributes and relations to the CWWorkerTask entity which will bear information pertaining to the task (data, relations to other entities in the database…).

Then extend the CWWorker entity class. It is meant to have do_xxxx methods, where xxxx matches the value of the operation attribute of CWWorkerTask entities. These methods are called with 2 arguments: a session and a task. You generally use the task to get additional parameters about what needs to be done. The method should return a Unicode string which will be used as a message in the transition information for the CWWorkerTask.

Here is an example of a CWWorker method which will asynchronously delete an entity from the database, this is interesting in case the entity has many composite relations and its deletion will trigger lengthy chained deletions. The entity is at the end of an added CWWorkerTask relation called target:

def do_delete_entity(self, session, task):
    entity = task.target[0]
    session.execute('DELETE Any X WHERE X eid %(eid)s', {'eid': entity.eid})
    return _('Success')

To trigger the deletion, all you need to do is to create a CWWorkerTask with the correct operation and target (which in this case may require overriding cubicweb.web.views.editforms.DeleteConfFormView and setting up a custom Controller):

task = self._cw.create_entity('CWWorkerTask',
                              operation=u'delete_entity',
                              target=some_entity)

Instance setup

You need to configure your instance to start the worker. This is done by setting long-transaction-worker to True in your instance configuration file (this is in the [WORKER] section). This will start a periodic task (you can also configure the period with worker-polling-period) which will look for pending tasks in the database. When a task is found, the worker will grab it and start working on it. The worker-max-load option sets the maximum number of tasks that can be run simultaneously by a worker. It defaults to 2 and you may want to set it to 1, but setting higher values will degrade performances.

You can setup your instance as usual and configure it with a worker, but the efficient way of doing things is to setup 2 instances (or more) sharing a common database. The first instance will have the long-transaction-worker option set to False and will concentrate on web serving, and will create new CWWorkerTask. The other instances can be repository only (i.e. cubicweb-ctl create -c repository -a somecube myworkerinstance) and will have long-transaction-worker set to True. That will ensure that the workers and the web serving processes are not fighting over Python’s Global Interpreter Lock and provide maximum performance.

Note about connections pool size: each task processed by a worker can use typically up to 3 connection from the pool. If you are running a worker in the same instance as the one which does web serving, you will probably need to set a larger connections-pool-size value than the default (4): 7 or 8 should be fine.

Release History

Release History

3.2.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.1.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.0.5

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.0.4

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.0.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.0.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

3.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
cubicweb-worker-3.2.0.tar.gz (16.9 kB) Copy SHA256 Checksum SHA256 Source Jun 19, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting