odoo12-addon-queue-job 12.0.3.0.1.99.dev1

Job Queue

This addon adds an integrated Job Queue to Odoo.

It allows to postpone method calls executed asynchronously.

Jobs are executed in the background by a Jobrunner, in their own transaction.

Example:

from odoo import models, fields, api

class MyModel(models.Model):
   _name = 'my.model'

   def my_method(self, a, k=None):
       _logger.info('executed with a: %s and k: %s', a, k)


class MyOtherModel(models.Model):
    _name = 'my.other.model'

    @api.multi
    def button_do_stuff(self):
        self.env['my.model'].with_delay().my_method('a', k=2)

In the snippet of code above, when we call button_do_stuff, a job capturing the method and arguments will be postponed. It will be executed as soon as the Jobrunner has a free bucket, which can be instantaneous if no other job is running.

Features:

• Views for jobs, jobs are stored in PostgreSQL

• Jobrunner: execute the jobs, highly efficient thanks to PostgreSQL’s NOTIFY

• Channels: give a capacity for the root channel and its sub-channels and segregate jobs in them. Allow for instance to restrict heavy jobs to be executed one at a time while little ones are executed 4 at a times.

• Retries: Ability to retry jobs by raising a type of exception

• Retry Pattern: the 3 first tries, retry after 10 seconds, the 5 next tries, retry after 1 minutes, …

• Job properties: priorities, estimated time of arrival (ETA), custom description, number of retries

• Related Actions: link an action on the job view, such as open the record concerned by the job

Table of contents

• Installation

• Configuration

• Usage

  • Developers

  • Tips and tricks

  • Patterns

• Known issues / Roadmap

• Changelog

  • Next

  • 12.0.1.1.0 (2019-11-01)

  • 12.0.1.0.0 (2018-10-02)

• Bug Tracker

• Credits

  • Authors

  • Contributors

  • Maintainers

Installation

Be sure to have the requests library.

Configuration

• Using environment variables and command line:

  • Adjust environment variables (optional):

    • ODOO_QUEUE_JOB_CHANNELS=root:4 or any other channels configuration. The default is root:1

    • if xmlrpc_port is not set: ODOO_QUEUE_JOB_PORT=8069

  • Start Odoo with --load=web,queue_job and --workers greater than 1. [1]

• Using the Odoo configuration file:

[options]
(...)
workers = 6
server_wide_modules = web,queue_job

(...)
[queue_job]
channels = root:2

• Confirm the runner is starting correctly by checking the odoo log file:

...INFO...queue_job.jobrunner.runner: starting
...INFO...queue_job.jobrunner.runner: initializing database connections
...INFO...queue_job.jobrunner.runner: queue job runner ready for db <dbname>
...INFO...queue_job.jobrunner.runner: database connections ready

• Create jobs (eg using base_import_async) and observe they start immediately and in parallel.

• Tip: to enable debug logging for the queue job, use --log-handler=odoo.addons.queue_job:DEBUG

[1]

It works with the threaded Odoo server too, although this way of running Odoo is obviously not for production purposes.

Usage

To use this module, you need to:

1. Go to Job Queue menu

Developers

Configure default options for jobs

In earlier versions, jobs could be configured using the @job decorator. This is now obsolete, they can be configured using optional queue.job.function and queue.job.channel XML records.

Example of channel:

<record id="channel_sale" model="queue.job.channel">
    <field name="name">sale</field>
    <field name="parent_id" ref="queue_job.channel_root" />
</record>

Example of job function:

<record id="job_function_sale_order_action_done" model="queue.job.function">
    <field name="model_id" ref="sale.model_sale_order" />
    <field name="method">action_done</field>
    <field name="channel_id" ref="channel_sale" />
    <field name="related_action" eval='{"func_name": "custom_related_action"}' />
    <field name="retry_pattern" eval="{1: 60, 2: 180, 3: 10, 5: 300}" />
</record>

The general form for the name is: <model.name>.method.

The channel, related action and retry pattern options are optional, they are documented below.

When writing modules, if 2+ modules add a job function or channel with the same name (and parent for channels), they’ll be merged in the same record, even if they have different xmlids. On uninstall, the merged record is deleted when all the modules using it are uninstalled.

Job function: channel

The channel where the job will be delayed. The default channel is root.

Job function: related action

The Related Action appears as a button on the Job’s view. The button will execute the defined action.

The default one is to open the view of the record related to the job (form view when there is a single record, list view for several records). In many cases, the default related action is enough and doesn’t need customization, but it can be customized by providing a dictionary on the job function:

{
    "enable": False,
    "func_name": "related_action_partner",
    "kwargs": {"name": "Partner"},
}

• enable: when False, the button has no effect (default: True)

• func_name: name of the method on queue.job that returns an action

• kwargs: extra arguments to pass to the related action method

Example of related action code:

class QueueJob(models.Model):
    _inherit = 'queue.job'

    def related_action_partner(self, name):
        self.ensure_one()
        model = self.model_name
        partner = self.records
        action = {
            'name': name,
            'type': 'ir.actions.act_window',
            'res_model': model,
            'view_type': 'form',
            'view_mode': 'form',
            'res_id': partner.id,
        }
        return action

Job function: retry pattern

When a job fails with a retryable error type, it is automatically retried later. By default, the retry is always 10 minutes later.

A retry pattern can be configured on the job function. What a pattern represents is “from X tries, postpone to Y seconds”. It is expressed as a dictionary where keys are tries and values are seconds to postpone as integers:

{
    1: 10,
    5: 20,
    10: 30,
    15: 300,
}

Based on this configuration, we can tell that:

• 5 first retries are postponed 10 seconds later

• retries 5 to 10 postponed 20 seconds later

• retries 10 to 15 postponed 30 seconds later

• all subsequent retries postponed 5 minutes later

Bypass jobs on running Odoo

When you are developing (ie: connector modules) you might want to bypass the queue job and run your code immediately.

To do so you can set TEST_QUEUE_JOB_NO_DELAY=1 in your enviroment.

Bypass jobs in tests

When writing tests on job-related methods is always tricky to deal with delayed recordsets. To make your testing life easier you can set test_queue_job_no_delay=True in the context.

Tip: you can do this at test case level like this

@classmethod
def setUpClass(cls):
    super().setUpClass()
    cls.env = cls.env(context=dict(
        cls.env.context,
        test_queue_job_no_delay=True,  # no jobs thanks
    ))

Then all your tests execute the job methods synchronously without delaying any jobs.

Tips and tricks

• Idempotency (https://www.restapitutorial.com/lessons/idempotency.html): The queue_job should be idempotent so they can be retried several times without impact on the data.

• The job should test at the very beginning its relevance: the moment the job will be executed is unknown by design. So the first task of a job should be to check if the related work is still relevant at the moment of the execution.

Patterns

Through the time, two main patterns emerged:

1. For data exposed to users, a model should store the data and the model should be the creator of the job. The job is kept hidden from the users

2. For technical data, that are not exposed to the users, it is generally alright to create directly jobs with data passed as arguments to the job, without intermediary models.

Known issues / Roadmap

• After creating a new database or installing queue_job on an existing database, Odoo must be restarted for the runner to detect it.

• When Odoo shuts down normally, it waits for running jobs to finish. However, when the Odoo server crashes or is otherwise force-stopped, running jobs are interrupted while the runner has no chance to know they have been aborted. In such situations, jobs may remain in started or enqueued state after the Odoo server is halted. Since the runner has no way to know if they are actually running or not, and does not know for sure if it is safe to restart the jobs, it does not attempt to restart them automatically. Such stale jobs therefore fill the running queue and prevent other jobs to start. You must therefore requeue them manually, either from the Jobs view, or by running the following SQL statement before starting Odoo:

update queue_job set state='pending' where state in ('started', 'enqueued')

Changelog

Next

• [ADD] Run jobrunner as a worker process instead of a thread in the main process (when running with –workers > 0)

• [REF] @job and @related_action deprecated, any method can be delayed, and configured using queue.job.function records

12.0.1.1.0 (2019-11-01)

Important: the license has been changed from AGPL3 to LGPL3.

• [IMP] Dont’ start the Jobrunner if root channel’s capacity is explicitly set to 0

• [ADD] Ability to set several jobs to done using an multi-action (port of #59)

• [REF] Extract a method handling the post of a message when a job is failed, allowing to modify this behavior from addons

• [ADD] Allow Jobrunner configuration from server_environment (details on #124)

• [ADD] Environment variable TEST_QUEUE_JOB_NO_DELAY=1 for test and debug (details on #114)

• [FIX] race condition under pressure, when starting a job takes more than 1 second (details on #131)

• [FIX] retry_postone on a job could be rollbacked on errors (details on #130)

• [FIX] Autovacuum cron job misconfiguration (details on #163)

12.0.1.0.0 (2018-10-02)

• [MIGRATION] from 11.0 branched at rev. b0945be

Bug Tracker

Bugs are tracked on GitHub Issues. In case of trouble, please check there if your issue has already been reported. If you spotted it first, help us smashing it by providing a detailed and welcomed feedback.

Do not contact contributors directly about support or help with technical issues.

Credits

Authors

• Camptocamp

• ACSONE SA/NV

Contributors

• Guewen Baconnier <[e-mail]>

• Stéphane Bidoul <[e-mail]>

• Matthieu Dietrich <[e-mail]>

• Jos De Graeve <[e-mail]>

• David Lefever <[e-mail]>

• Laurent Mignon <[e-mail]>

• Laetitia Gangloff <[e-mail]>

• Cédric Pigeon <[e-mail]>

• Souheil Bejaoui <[e-mail]>

Maintainers

This module is maintained by the OCA.

OCA, or the Odoo Community Association, is a nonprofit organization whose mission is to support the collaborative development of Odoo features and promote its widespread use.

Current maintainer:

This module is part of the OCA/queue project on GitHub.

You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.