Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Appengine related Python Packages from Lovely Systems

Project Description

lovely.gae.async

This package executes jobs asynchronously, it uses the appengine taskqueue to exectue the jobs.

>>> from lovely.gae.async import defer, get_tasks

The defer function executes a handler asynchronously as a job. We create 3 jobs that have the same signature.

>>> import time
>>> for i in range(3):
...     print defer(time.sleep, [0.3])
<google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
None
None

Let us have a look on what jobs are there. Note that there is only one because all the 3 jobs we added were the same.

>>> len(get_tasks())
1

If we change the signature of the job, a new one will be added.

>>> added = defer(time.sleep, [0.4])
>>> len(get_tasks())
2

Normally jobs are automatically executed by the taskqueueapi, we have a test method which executes the jobs and returns the number of jobs done.

>>> run_tasks()
2

Now we can add the same signature again.

>>> added = defer(time.sleep, [0.4])
>>> run_tasks()
1

We can also set only_once to false to execute a worker many times with the same signature.

>>> from pprint import pprint
>>> defer(pprint, ['hello'], once_only=False)
<google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
>>> defer(pprint, ['hello'], once_only=False)
<google.appengine.api.labs.taskqueue.taskqueue.Task object at ...>
>>> run_tasks()
'hello'
'hello'
2

DB Custom Property Classes

Typed Lists

This property converts model instances to keys and ensures a length.

>>> from lovely.gae.db.property import TypedListProperty
>>> from google.appengine.ext import db

Let us create a model

>>> class Yummy(db.Model): pass
>>> class Bummy(db.Model): pass

We can now reference Yummy instances with our property. Note that we can also use the kind name as an argument of the kind.

>>> class Dummy(db.Model):
...     yummies = TypedListProperty(Yummy)
...     bummies = TypedListProperty('Bummy', length=3)

The kind arguement needs to be a subclass kind name or a db.Model.

>>> TypedListProperty(object)
Traceback (most recent call last):
...
ValueError: Kind needs to be a subclass of db.Model
>>> dummy = Dummy()
>>> dummy_key = dummy.put()
>>> yummy1 = Yummy()
>>> yummy1_key = yummy1.put()
>>> dummy.yummies = [yummy1]

We cannot set any other type.

>>> bummy1 = Bummy()
>>> bummy1_key = bummy1.put()
>>> dummy.yummies = [bummy1]
Traceback (most recent call last):
...
BadValueError: Wrong kind u'Bummy'

The length needs to match if defined (see above).

>>> dummy.bummies = [bummy1]
Traceback (most recent call last):
...
BadValueError: Wrong length need 3 got 1
>>> dummy.bummies = [bummy1, bummy1, bummy1]
>>> dummy_key == dummy.put()
True

Case-Insensitive String Property

This property allows searching for the lowercase prefix in a case-insensitive manner. This is usefull for autocomplete implementations where we do not want to have a seperate property just for searching.

>>> from lovely.gae.db.property import IStringProperty
>>> class Foo(db.Model):
...     title = IStringProperty()
>>> f1 = Foo(title='Foo 1')
>>> kf1 = f1.put()
>>> f2 = Foo(title='Foo 2')
>>> kf2 = f2.put()
>>> f3 = Foo(title='foo 3')
>>> kf3 = f3.put()
>>> f4 = Foo(title=None)
>>> kf4 = f4.put()

The property does not allow the special seperator character which is just one below the highest unicode character,

>>> f3 = Foo(title='Foo 3' + IStringProperty.SEPERATOR)
Traceback (most recent call last):
...
BadValueError: Not all characters in property title

Note that if we want to do an exact search, we have to use a special filter that can be created by the property instance.

>>> [f.title for f in Foo.all().filter('title =', 'foo 1')]
[]

The “equal” filter arguments can be computed with a special method on the property.

>>> ef = Foo.title.equals_filter('Foo 1')
>>> ef
('title =', u'foo 1\xef\xbf\xbcFoo 1')
>>> [f.title for f in Foo.all().filter(*ef)]
[u'Foo 1']

Let us try with inequallity, e.g. prefix search. Prefix search is normally done with two filters using the highest unicode character.

Search for all that starts with “fo” case-insensitive.

>>> q = Foo.all()
>>> q = q.filter('title >=', 'fo')
>>> q = q.filter('title <', 'fo' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'Foo 1', u'Foo 2', u'foo 3']

Search for all that starts with ‘foo 1’

>>> q = Foo.all()
>>> q = q.filter('title >=', 'foo 1')
>>> q = q.filter('title <', 'foo 1' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'Foo 1']
>>> q = Foo.all()
>>> q = q.filter('title >=', 'foo 2')
>>> q = q.filter('title <=', 'foo 2' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'Foo 2']
>>> q = Foo.all()
>>> q = q.filter('title >=', 'foo 3')
>>> q = q.filter('title <=', 'foo 3' + u'\xEF\xBF\xBD')
>>> [f.title for f in q]
[u'foo 3']

Pickle Property

A pickle property can hold any pickleable python object.

>>> from lovely.gae.db.property import PickleProperty
>>> class Pickie(db.Model):
...     data = PickleProperty()
>>> pickie = Pickie(data={})
>>> pickie.data
{}
>>> kp = pickie.put()
>>> pickie.data
{}
>>> pickie = db.get(kp)
>>> pickie.data
{}
>>> pickie.data = {'key':501*"x"}
>>> kp = pickie.put()
>>> pickie.data
{'key': 'xx...xx'}

If the value is not pickleable we get a validation error.

>>> pickie = Pickie(data=dict(y=lambda x:x))
Traceback (most recent call last):
BadValueError: Property 'data' must be pickleable:
(Can't pickle <function <lambda> at ...>:
it's not found as __main__.<lambda>)

Safe ReferenceProperty

>>> from lovely.gae.db.property import SafeReferenceProperty

We use a new model with a gae reference and our safe reference.

>>> class Refie(db.Model):
...     ref   = db.ReferenceProperty(Yummy, collection_name='ref_ref')
...     sfref = SafeReferenceProperty(Yummy, collection_name='sfref_ref')
>>> refie = Refie()
>>> refie.sfref is None
True
>>> refie.ref is None
True

An object to be referenced.

>>> refyummy1 = Yummy()
>>> ignore = refyummy1.put()

Set the references to our yummy object.

>>> refie.sfref = refyummy1
>>> refie.sfref
<Yummy object at ...>
>>> refie.ref = refyummy1
>>> refie.ref
<Yummy object at ...>
>>> refieKey = refie.put()

Now we delete the referenced object.

>>> refyummy1.delete()

And reload our referencing object.

>>> refie = db.get(refieKey)

The gae reference raises an exception.

>>> refie.ref
Traceback (most recent call last):
Error: ReferenceProperty failed to be resolved

We catch the logs here.

>>> import logging
>>> from StringIO import StringIO
>>> log = StringIO()
>>> handler = logging.StreamHandler(log)
>>> logger = logging.getLogger('lovely.gae.db')
>>> logger.setLevel(logging.INFO)
>>> logger.addHandler(handler)

Our safe reference returns None.

>>> pos = log.tell()
>>> refie.sfref is None
True

Let’s see what the log contains.

>>> log.seek(pos)
>>> print log.read()
Unresolved Reference for "Refie._sfref" set to None

Accessing the stale property once again we will see it was reset to None:

>>> pos = log.tell()
>>> refie.sfref is None
True

>>> log.seek(pos)
>>> print log.read() == ''
True

The property get set to None if the reference points to a dead object but only if the property is not required:

>>> class Requy(db.Model):
...     sfref = SafeReferenceProperty(Yummy, collection_name='req_sfref_ref',
...                                   required=True)

>>> refyummy1 = Yummy()
>>> ignore = refyummy1.put()

>>> requy = Requy(sfref = refyummy1)
>>> requyKey = requy.put()

>>> requy.sfref
<Yummy object at ...>

>>> refyummy1.delete()

>>> requy = db.get(requyKey)

>>> pos = log.tell()
>>> requy.sfref is None
True

>>> log.seek(pos)
>>> print log.read()
Unresolved Reference for "Requy._sfref" will remain because it is required

Batch marker creation

This packages provides the possibility to create markers for every N objects of a given query. This is useful to create batched html pages or to generate jobs for every N objects.

A list of attribute values are created that represent the end of a batch at any given position in a given query. The result is stored in memcache and the key is provided to a callback function.

>>> from lovely.gae import batch

Let us create some test objects.

>>> from google.appengine.ext import db
>>> class Stub(db.Model):
...     c_time = db.DateTimeProperty(auto_now_add=True, required=True)
...     name = db.StringProperty()
...     age = db.IntegerProperty()
...     state = db.IntegerProperty()
...     def __repr__(self):
...         return '<Stub %s>' % self.name
>>> for i in range(1,13):
...     s = Stub(name=str(i), age=i, state=i%2)
...     sk = s.put()
>>> Stub.all().count()
12

First we make sure that we have no tasks in the queue for testing.

>>> from lovely.gae.async import get_tasks
>>> len(get_tasks())
0

So for example if we want to know any 100th key of a given kind we could calculate it like shown below. Note that we provide the pprint function as a callback, so we get the memcache key in the output.

The calculate_markers function returns the memcache key that will be used to store the result when the calculation is completed.

>>> from pprint import pprint
>>> mc_key = batch.create_markers('Stub', callback=pprint)
>>> mc_key
'create_markers:...-...-...'

A task gets created.

>>> tasks = get_tasks()
>>> len(tasks)
1

Let us run the task.

>>> run_tasks(1)
1

We now have another task left for the callback, which is actually the pprint function.

>>> run_tasks()
'create_markers:...-...-...'
1

We should now have a result. The result shows that we need no batches for the given batch size (because we only have 12 objects).

>>> from google.appengine.api import memcache
>>> memcache.get(mc_key)
[]

Let us use another batch size. This time without callback.

>>> mc_key = batch.create_markers('Stub', batchsize=1)
>>> run_tasks()
1

We now have exatly 12 keys, because the batch size was 1.

>>> len(memcache.get(mc_key))
12

The default attributes returned are the keys.

>>> memcache.get(mc_key)
[datastore_types.Key.fro...

We can also use other attributes. Let us get items batched by c_time descending. Note that it is not checked if values are not unique, so if a non-unique attribute is used it might be the case that batch ranges contains objects twice.

>>> mc_key = batch.create_markers('Stub',
...                               attribute='c_time',
...                               order='desc',
...                               batchsize=3)
>>> run_tasks()
1
>>> markers = memcache.get(mc_key)
>>> markers
[datetime.datetime(...
>>> len(markers)
4
>>> sorted(markers, reverse=True) == markers
True
>>> mc_key = batch.create_markers('Stub',
...                               attribute='c_time',
...                               order='asc',
...                               batchsize=3)
>>> run_tasks()
1
>>> markers = memcache.get(mc_key)
>>> markers
[datetime.datetime(...
>>> len(markers)
4
>>> sorted(markers) == markers
True

We can also pass filters to be applied to the query for the batch like this:

>>> mc_key = batch.create_markers('Stub',
...                               filters=[('state', 0)],
...                               attribute='c_time',
...                               order='asc',
...                               batchsize=3)
>>> run_tasks()
1
>>> markers = memcache.get(mc_key)
>>> len(markers)
2
Release History

Release History

This version
History Node

1.0.0a2

History Node

1.0.0a1

History Node

0.5.0a8

History Node

0.5.0a6

History Node

0.5.0a5

History Node

0.5.0a4

History Node

0.5.0a3

History Node

0.5.0a2

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
lovely.gae-1.0.0a2.tar.gz (23.3 kB) Copy SHA256 Checksum SHA256 Source Jun 11, 2010

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting