Skip to main content

Interface for TaskQueue

Project description

Google Taskqueue client
=======================

Support for Python client for `Task Queue Rest API`_ using `gcloud`_

.. _Task Queue Rest API: https://cloud.google.com/appengine/docs/python/taskqueue/rest/
.. _gcloud: https://googlecloudplatform.github.io/gcloud-python/


Required Dependencies
---------------------

The following third-party Python modules are required:
- gcloud

The easiest way to install the dependencies is to run::

$ pip install -r requirements.txt

Why an interface to the Task Queue API?
---------------------------------------
The Google Cloud Client Library for Python (`gcloud`_) is great, but it's missing
an interface to the Task Queue Rest API. This is understandable, because the API
is experimental and still in alpha, and not fully accesble from outside the App Egine.
And gcloud does support Pubsub.

However, Task Queues offer a few functions that aren't available in Pubsub. And
although Push Queues are restricted to use within App Engine, Pull Queues can be
used outside the App Engine enviroment.

With Pull Queues a worker can lease tasks for a certain period. During this period,
the tasks aren't available to other workers. This prevents processing a single task
multiple times by different workers. This makes Task Queues very useful for background
processing, for example in combination with object change notifications, to process
objects in a Storage Bucket the moment they are changed or created.

.. https://cloud.google.com/compute/docs/tutorials/batch-processing-with-autoscaler
.. https://cloud.google.com/appengine/docs/python/taskqueue/rest


Example
-------

Example::

>>> from gcloud_taskqueue import Taskqueue, Client

For object change notifications, you need to create a service account. To pull tasks
created with this account, you need to use it's credentials::

>>> json_credentials_path = "/path/to/my-service-credential.json"
>>> client = Client.from_service_account_json(json_credentials_path, project="my-project")


Get taskqueue::

>>> tq = Taskqueue(client=client, id="my-taskqueue")


Get tasks in taskqueue::

>>> for task in tq.list_tasks(client=client):
>>> print("{}\t{}\t{}".format(task.id, task.leaseTimestamp, task.retry_count))


Lease 10 tasks for 60 seconds::

>>> for task in tq.lease(lease_time=60, num_tasks=10, client=client):
>>> print("{}\t{}\t{}".format(task.id, task.leaseTimestamp, task.retry_count))


Lease 10 tasks with specific tag for 60 seconds::

>>> for task in tq.lease(lease_time=60, num_tasks=10, tag='my-tag', client=client):
>>> print("{}\t{}\t{}".format(task.id, task.leaseTimestamp, task.retry_count))
v0.1.0, -- Initial release.
v0.1.1, 2016.04.07 -- bugfix, Taskqueue().exists() returns false if taskqueue does not exist (https://github.com/sebastiancodes/gcloud_taskqueue/issues/1)
v0.1.2, 2016.04.07 -- fix indentation, raise DoesNotExist exception if Taskqueue does not exist https://github.com/sebastiancodes/gcloud_taskqueue/issues/1

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gcloud_taskqueue, version 0.1.2
Filename, size File type Python version Upload date Hashes
Filename, size gcloud_taskqueue-0.1.2-py2.py3-none-any.whl (13.3 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size gcloud_taskqueue-0.1.2.tar.gz (7.8 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page