Memcached based locking factory functions to provide shared locking (e.g. bet. zeo-clients)
unimr.memcachedlock implements a distributed “soft” locking mechanism using memcached. It provides factory functions and decorators for a primitive locking, a reentrant locking and a special locking for zeo-clients.
The native locking methods of python’s threading module supports thread safe locking and therefore, provides only full locking support for single zope installations. However, zeo-clients have no locking mechanism beetween each other for concurrent write operations on identical objects (e.g. Catalog) and are unnecessarily stressed to resolve ConflictErrors on heavy load. The reason for this problem is the optimistic concurrency control of the ZODB which primarly prepares the changes of an object (in many cases expensive calculations) and thereafter checks the validity of the object for the write process. The higher the number of writes on the same object the higher the risk that a concurrent zeo-client has already invalidated the object while another zeo-client has still this object in use. The client with the invalidated object is constrained to roll back its changes and to recalculate the changes based on the refreshed object. At worst, this state goes in circles and results in a ConflictError. The optimistic concurrency control therefore perfectly fits only concurrent write operations on distinct objects.
Memcache locking overcomes this problem because it extends the regular concurrency mechanism by a shared locking beetween all involved zeo-clients by serializing the concurrent write operations before a ConflictError is provoked. This mechanism is also known as pessimistic concurrency control.
There is no risk of loosing data within a zope environment because memcachedlock will always fall back to zope’s transaction control.
unimr.memcachedlock easily serializes concurrent writes
>>> from unimr.memcachedlock import memcachedlock ... ... # define an invariant unique key of an instance ... def get_key(fun,*args,**kargs): ... fun, instance = args[0:2] ... return '/'.join(instance.getPhysicalPath())
>>> # lock decorator ... @memcachedlock.lock(get_key) ... def concurrent_write(self, *args, **kw): ... """ method which produces many conflict errors (bottle neck)""" ... # ...
or for recursive function calls
>>> # rlock decorator ... @memcachedlock.rlock(get_key) ... def concurrent_write(self, *args, **kw): ... """ method which produces many conflict errors (bottle neck)""" ... # ...
or for function calls of zeo-clients providing a special ConflictError handling to interact properly with the optimistic concurrency control
>>> # zlock decorator ... @memcachedlock.zlock(get_key) ... def concurrent_write(self, *args, **kw): ... """ method which produces many conflict errors (bottle neck)""" ... # ...
The decorators @memcachedlock.lock, @memcachedlock.rlock or @memcachedlock.zlock take exactly one argument
- function which returns an invariant unique key of an instance known by all zeo-clients (required)
and two optional keywords:
- livetime in seconds of the lock (default: 30)
- retrial interval of the lock check in seconds (default: 0.05)
unimr.memcachedlock already includes a patch for zope’s Catalog to enable locking for the catalog_object method. Uncomment corresponding lines in configure.zcml of this package.
Configuring memcached servers
You can configure one ore mor memcached servers by adding the environment variable MEMCACHEDLOCK_SERVERS to the buildout.cfg as follows (default server: 127.0.0.1:11211):
[instance] ... zope-conf-additional = <environment> MEMCACHEDLOCK_SERVERS <ip/hostname of host1>:<port>,<ip/hostname of host2>:<port> </environment>
- Tests …
- Pessimistic concurrency control implementation by means of native zeo server protocol.
2010-09-23: 0.1 - RC2
- ConflictError handling for zeo clients enhanced
2009-05-08: 0.1 - RC1
- Initial release
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size & hash SHA256 hash help||File type||Python version||Upload date|
|unimr.memcachedlock-0.1rc2_r144828-py2.4.egg (15.2 kB) Copy SHA256 hash SHA256||Egg||2.4||Sep 23, 2010|
|unimr.memcachedlock-0.1rc2-r144828.tar.gz (7.2 kB) Copy SHA256 hash SHA256||Source||None||Sep 23, 2010|