cornerstone.soup 2.3.1

Container with queryable Records for Zope 2

Overview

cornerstone.soup provides a container for persistent records which are queryable. It is a genric storage for mass-data in an isolated container. Light-weight records are stored in an IOBTree. A Zope-Tool-Kit catalog in used to index values of interest. cornerstone.soup is no out-of-the-box package. Its addressed to developers needing to solve the problem of storing tiny entities of mass-data, where heavy weight archetypes or dexterity are too much effort and are to slow. I.e if you need a container for non-CMSish content, like votes, data from a poll, orders in a webshop, measuring data, or alike.

Updating

In earlier days of this package we thought it’s a good idea to persist the soup data in persistent local components. That was quite a mistake, at least in Plone context, because GenericSetup purges local components when applying base profiles - what you’re normally not doing, but experience shows that shit happens ;). So we changed the storage location to annotations on an acquireable, ISoupAnnotatable providing context.

Further the soup API was designed as utility, which was basically a good idea, but caused toubles when looking up SoupData after the storage change. We used getSiteManager to access the Acquisition context, and encountered inconsistencies for accessing the Acquisition context from different site managers in Plone.

The second problem forced us more or less to abandon the utility pattern, the soup object itself now acts as adapter for context and is looked up via getSoup instead of a utility lookup. After updating, you’ll get NoLongerSupported errors when trying to access a soup which is provided and looked up as utility. You’ll have to change your code to use getSoup, and remove the soup local component registration from your GS Profile(s).

>>> from cornerstone.soup import getSoup
>>> soup = getSoup(context, 'mysoup')

The new package ships with data migration. After updating call soup-controlpanel (in Plone) and run storage migration and remove persistent local component for each soup. Prior to running the storage migration the existing soup data is inaccessible.

Due to the fact that the soup was originally persisted to the ZOBD, it still inherits from SimpleItem. This will be changed with the release 3.0. Keep in mind that 3.0 will break installations with non-cleaned-up installations.

Usage

SoupData objects are stored as annotation to an object providing the ISoupAnnotatable interface.

For use inside Plone, provide ISoupAnnotatable via five.implements on the plone site object usind ZCML.

<five:implements
    class="Products.CMFPlone.Portal.PloneSite"
    interface="cornerstone.soup.interfaces.ISoupAnnotatable" />

SoupData is looked up by id for a given context. This context acquires it’s parent until ISoupAnnotatable is found, on which the SoupData is annotated by id. Use getSoup function for this.

>>> from cornerstone.soup import getSoup
>>> soup = getSoup(context, 'my_soup_id')
>>> soup
<Soup at my_soup_id>

If no SoupData is found for given id, a new one is created and annotated to ISoupAnnotatable.

We must provide an ICatalogFactory implementation for each soup, registered as utility under the same id as SoupData is annotated.

Make sure that Catalog is re-created each time catalog factory gets called. this is needed for correct record reindexing.

>>> from zope.interface import implements
>>> from zope.catalog.catalog import Catalog
>>> from zope.catalog.field import FieldIndex
>>> from cornerstone.soup.interfaces import ICatalogFactory
>>> class MyCatalogFactory(object):
...     implements(ICatalogFactory)
...
...     def __call__(self):
...         catalog = Catalog()
...         catalog[u'name'] = FieldIndex(field_name='name',
...                                       field_callable=False)
...         return catalog

ZCML.

<utility
    name="my_soup_id"
    factory=".mymodule.MyCatalogFactory"
    provides="cornerstone.soup.interfaces.ICatalogFactory" />

A Soup can only contain Records. A Record is a simple persistent object which accepts any keyword arguments on __init__ time. This arguments are used as Record properties.

Create a Record and add it to soup.

>>> from cornerstone.soup import Record
>>> record = Record(user='user1')
>>> id = soup.add(record)

Check querying.

>>> [r for r in soup.query(user='user1')]
[<Record at ...>]

>>> [r for r in soup.query(user='nonexist')]
[]

Add some more Records.

>>> id = soup.add(Record(user='user1'))
>>> id = soup.add(Record(user='user2'))
>>> u1records = [r for r in soup.query(user='user1')]
>>> u1records
[<Record at ...>,
<Record at ...>]

Change user attribute of one record.

>>> u1records[0].data['user'] = 'user2'

The query still returns the old result. The Record must be reindexed.

>>> [r for r in soup.query(user='user1')]
[<Record at ...>,
<Record at ...>]

>>> soup.reindex([u1records[0]])

>>> u1 = [r for r in soup.query(user='user1')]
>>> u1
[<Record at ...>]

>>> u2 = [r for r in soup.query(user='user2')]
>>> u2
[<Record at ...>,
<Record at ...>]

You can reindex all records in soup at once.

>>> all = [r for r in soup.data.values()]
>>> all = sorted(all, key=lambda x: x.user)
>>> all
[<Record at ...>,
<Record at ...>,
<Record at ...>]

>>> all[-1].data['user'] = 'user3'
>>> soup.reindex()
>>> [r for r in soup.query(user='user3')]
[<Record at ...>]

You can also rebuild the catalog. In this case the catalog factory is called again and the new catalog is used. Lets modify catalog of catalog factory.

>>> from zope.catalog.field import FieldIndex
>>> catalogfactory = getUtility(ICatalogFactory, name='mysoup')
>>> catalogfactory.catalog[u'name'] = FieldIndex(field_name='name',
...                                   field_callable=False)
>>> catalogfactory()[u'name']
<zope.catalog.field.FieldIndex object at ...>

Set name attribute on some record data, rebuild soup and check results.

>>> all[0].data['name'] = 'name'
>>> all[1].data['name'] = 'name'
>>> all[2].data['name'] = 'name'
>>> soup.rebuild()
>>> [r for r in soup.query(name='name')]
[<Record at ...>,
<Record at ...>,
<Record at ...>]

Delete records.

>>> del soup[all[0]]
>>> [r for r in soup.query(name='name')]
[<Record at ...>,
<Record at ...>]

For huge expected results we can query LazyRecords. They return the real record on call.

>>> lazy = [l for l in soup.lazy(name='name')]
>>> lazy
[<cornerstone.soup.soup.LazyRecord object at ...>,
<cornerstone.soup.soup.LazyRecord object at ...>]

>>> lazy[0]()
<Record at ...>

Text Index NG 3 support

This package provides a zope3 index wrapper for textindexng3. It is located at cornerstone.soup.ting.TingIndex.

You can use textindexng3 to index multiple fields of record at once, and make complex queries to this index. See Products.TextIndexNG3 for more information.

I you want to use textindexng3 with cornerstone.soup, make sure package zopyx.txng3.core is installed and it’s ZCML is loaded. zopyx.txng3.core is NO hard dependency of cornerstone.soup.

A TingIndex just expects field names as space separated string, or as iterable. A catalog factory using TingIndex looks like this.

>>> class TingCatalogFactory(object):
...     implements(ICatalogFactory)
...
...     def __call__(self):
...         catalog = Catalog()
...         catalog[u'ting'] = TingIndex(field_name=('foo', 'bar', 'baz'),
...                                      field_callable=False)
...         return catalog

Register this catalog factory as utility, we use tingsoup in this example.

Query textindexng3 using soup.

>>> soup = getSoup(site, 'tingsoup')
>>> soup
<Soup at tingsoup>

Index some records.

>>> id = soup.add(Record(foo='foo', bar='bar', baz='baz'))
>>> id = soup.add(Record(foo='foobar', bar='barbaz', baz='bazfoo'))
>>> id = soup.add(Record(foo='aaa', bar='barrrr', baz='ccc'))

and query them.

>>> query = {
...     'query': u'bar::and(bar*)',
...     'search_all_fields': True,
... }
>>> [r.bar for r in soup.query(ting=query)]
['bar', 'barbaz', 'barrrr']

Todo

• No more index query special handling for FieldIndex in cornerstone.soup.Soup._query (planned for 2.4)

Contributors

• Robert Niederreiter <[e-mail]>

• Jens Klein <[e-mail]>

Changes

2.3.1

• do not import ting interfaces to make cornerstone.soup work without textindexng3 - 2010-12-16 - rnix

2.3

• fixed move code if empty soup is moved - 2010-12-16 - rnix

• implemented clear function. 2010-12-16 - rnix

• add text index NG3 support. 2010-12-16 - rnix

2.2

• added mechanism to point soups to a path with an existing soup, so mount points can be used from several sites. Also overhauled UI of control panel and added more information what happens on mount or move. 2010-11-16 - jensens

2.1

• added working move mechanism 2010-09-09 - jensens, rnix

• added deepcopy support 2010-09-09 - jensens

• added length support for SoupData using BTree’s Length. 2010-09-08 - jensens

• overhauled control panel UI 2010-09-08 - jensens

• fix query bug. only convert single value to 2-tuple when search key refers to FieldIndex instance 2010-07-20 - rnix

• consider sort_index, limit and reverse as query kwargs 2010-07-20 - rnix

2.0

• change API 2010-06-18 - rnix

• change storage strategy 2010-06-18 - rnix

1.0

• complete tests 2009-12-02 - rnix

• remove IIntIds and INameChooser dependencies 2009-12-02 - rnix, jensens

1.0b2

• change namechoosing of records. Use uuid.uuid4() 2009-10-01 - rnix

• add lazy function for querying huge results 2009-10-01 - rnix

• yield query result instead of collect in a list 2009-09-29 - rnix

1.0b1

• initial work - rnix, jensens

License

Copyright (c) 2008-2010, BlueDynamics Alliance, Austria All rights reserved.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

• Neither the name of the BlueDynamics Alliance nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY BlueDynamics Alliance AS IS AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL BlueDynamics Alliance BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.