cubicweb-worker 3.1.0

asynchronous workers in your instance

Summary

asynchronous workers in your instance

How to use this cube

Add the worker cube to the dependencies of your cube. In your schema, extend the cube’s schema if necessary. Common extension often involve adding attributes and relations to the CWWorkerTask entity which will bear information pertaining to the task (data, relations to other entities in the database…).

Then extend the CWWorker entity class. It is meant to have do_xxxx methods, where xxxx matches the value of the operation attribute of CWWorkerTask entities. These methods are called with 2 arguments: a session and a task. You generally use the task to get additional parameters about what needs to be done. The method should return a Unicode string which will be used as a message in the transition information for the CWWorkerTask.

Here is an example of a CWWorker method which will asynchronously delete an entity from the database, this is interesting in case the entity has many composite relations and its deletion will trigger lengthy chained deletions. The entity is at the end of an added CWWorkerTask relation called target:

def do_delete_entity(self, session, task):
    entity = task.target[0]
    session.execute('DELETE Any X WHERE X eid %(eid)s', {'eid': entity.eid})
    return _('Success')

To trigger the deletion, all you need to do is to create a CWWorkerTask with the correct operation and target (which in this case may require overriding cubicweb.web.views.editforms.DeleteConfFormView and setting up a custom Controller):

task = self._cw.create_entity('CWWorkerTask',
                              operation=u'delete_entity',
                              target=some_entity)

Instance setup

You need to configure your instance to start the worker. This is done by setting long-transaction-worker to True in your instance configuration file (this is in the [WORKER] section). This will start a periodic task (you can also configure the period with worker-polling-period) which will look for pending tasks in the database. When a task is found, the worker will grab it and start working on it. The worker-max-load option sets the maximum number of tasks that can be run simultaneously by a worker. It defaults to 2 and you may want to set it to 1, but setting higher values will degrade performances.

You can setup your instance as usual and configure it with a worker, but the efficient way of doing things is to setup 2 instances (or more) sharing a common database. The first instance will have the long-transaction-worker option set to False and will concentrate on web serving, and will create new CWWorkerTask. The other instances can be repository only (i.e. cubicweb-ctl create -c repository -a somecube myworkerinstance) and will have long-transaction-worker set to True. That will ensure that the workers and the web serving processes are not fighting over Python’s Global Interpreter Lock and provide maximum performance.

Note about connections pool size: each task processed by a worker can use typically up to 3 connection from the pool. If you are running a worker in the same instance as the one which does web serving, you will probably need to set a larger connections-pool-size value than the default (4): 7 or 8 should be fine.