collective.recipe.filestorage 0.1

This recipe aids the creation and management of multiple Zope 2 filestorages.

Code repository: http://svn.plone.org/collective/buildout/collective.recipe.filestorage/trunk

Change history
**************

0.1 (2008-04-22)
================

- Happy Earth Day!
[davisagli]

- Created recipe with ZopeSkel. Initial implementation.
[davisagli]

Detailed Documentation
**********************

Introduction
============

This recipe adds additional filestorage and zodb_db stanzas to the zope.conf and zeo.conf files generated by the
plone.recipe.zope2instance and plone.recipe.zope2zeoserver recipes. It also creates the directories in which
the extra filestorage data files will be created. This makes it easy to add additional filestorages and mountpoints
to a Zope 2 instance.


Supported options
=================

parts
A list of filestorage sub-parts to be generated, one per line. (This is different from buildout parts.)
zeo
The name of a plone.recipe.zope2zeoserver part to which we want to add the extra filestorage. Defaults to the
first plone.recipe.zope2zeoserver part in the buildout, if any.
zopes
A list of names of plone.recipe.zope2instance parts to which we want to add the extra filestorage. Defaults to
all plone.recipe.zope2instance parts connected to the associated plone.recipe.zope2zeoserver part, if any, or all
plone.recipe.zope2instance parts, if no ZEO is found.

The following options affect the generated zope.conf and zeo.conf. Each may be specified for all filestorage subparts
in the collective.recipe.filestorage buildout part, or for one particular filestorage subpart by placing the option in
a new buildout part called 'filestorage_subpart', where subpart is the name of the subpart listed in the 'parts' option
of this recipe. The name of the subpart may be interpolated by using '%(fs_part_name)s' in the option.

location
The location of the Data.fs file, relative to the buildout root directory
Defaults to var/filestorage/%(fs_part_name)s/Data.fs
zodb-name
The name of the ZODB. Defaults to '%(fs_part_name)s'.
zodb-cache-size
Set the ZODB cache size, i.e. the number of objects which the ZODB cache will try to hold. Inherits from the
associated zope part. Defaults to 5000.
zodb-mountpoint
Set the path to the mountpoint. Defaults to '/%(fs_part_name)s'.
zeo-address
Set the port of the associated ZEO server. Inherits from the associated Zope and ZEO parts. Defaults to 8100.
zeo-client-cache-size
Set the size of the ZEO client cache. Inherits from the associated Zope part. Defaults to '30MB'.
zeo-storage
Set the id of the ZEO storage. Defaults to '%(fs_part_name)s'.
zeo-client-name
Set the name of the ZEO client. Defaults to '%(fs_part_name)s_zeostorage'.
zeo-client-client
Set the persistent cache name that is used to construct the cache filenames. Persistent cache files are
disabled by default.


Example usage
=============


Let's create and run a minimal buildout that adds an extra filestorage::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')

Our zope.conf should get the extra filestorage stanza automatically injected into it::

>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
instancehome ...instance
...
<BLANKLINE>
<zodb_db my-fs>
cache-size 5000
<filestorage >
path .../var/filestorage/my-fs/my-fs.fs
</filestorage>
mount-point /my-fs
</zodb_db>
<BLANKLINE>

The recipe will also create a directory for the new filestorage::

>>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
True

Let's make sure that the conf files will be regenerated whenever we make a change to a filestorage part,
even if the direct configuration for the zope/zeo parts hasn't changed.

>>> open('buildout.cfg', 'a').write(" my-fs-2\n")
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs-2' in open('parts/instance/etc/zope.conf').read()
True

Let's make sure that the filestorage directory is not clobbered even if the filestorage part is removed
from the buildout::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
True

We can override the defaults for a number of settings::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... location = var/filestorage/%(fs_part_name)s/Data.fs
... zodb-name = %(fs_part_name)s_db
... zodb-cache-size = 1000
... zodb-mountpoint = /%(fs_part_name)s_mountpoint
... parts =
... my-fs
... '''.replace('%(zope2_location)s', zope2_location))
>>> print system(join('bin', 'buildout') + ' -q')
>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
instancehome ...instance
...
<BLANKLINE>
<zodb_db my-fs_db>
cache-size 1000
<filestorage >
path .../var/filestorage/my-fs/Data.fs
</filestorage>
mount-point /my-fs_mountpoint
</zodb_db>
<BLANKLINE>

A setting can also be modified just for one particular filestorage, by creating a new part with
the 'filestorage_' prefix, like so::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
... my-fs
...
... [filestorage_my-fs]
... zodb-cache-size = 1000
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
instancehome ...instance
...
<BLANKLINE>
<zodb_db my-fs>
cache-size 1000
<filestorage >
path .../var/filestorage/my-fs/my-fs.fs
</filestorage>
mount-point /my-fs
</zodb_db>
<BLANKLINE>


By default, the recipe adds the extra filestorages to each plone.recipe.zope2instance part in the buildout,
but you can tell it to only add it to certain parts:

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... instance1
... instance2
...
... [instance1]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [instance2]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zopes = instance1
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in open('parts/instance1/etc/zope.conf').read()
True
>>> 'my-fs' in open('parts/instance2/etc/zope.conf').read()
False

Example Usage with ZEO
======================

Here is a minimal buildout including a ZEO server and two ZODB clients::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... zeoserver
... primary
... secondary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')

This should result in the appropriate additions to zeo.conf and both zope.conf's::

>>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
>>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
%define INSTANCE /sample-buildout/parts/zeoserver
...
<BLANKLINE>
<filestorage my-fs>
path /sample-buildout/var/filestorage/my-fs/my-fs.fs
</filestorage>
<BLANKLINE>

>>> primary = os.path.join(sample_buildout, 'parts', 'primary')
>>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
instancehome /sample-buildout/parts/primary
...
<BLANKLINE>
<zodb_db my-fs>
cache-size 5000
<zeoclient>
server 8100
storage my-fs
name my-fs_zeostorage
var /sample-buildout/parts/primary/var
cache-size 30MB
<BLANKLINE>
</zeoclient>
mount-point /my-fs
</zodb_db>
<BLANKLINE>

>>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
>>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
instancehome /sample-buildout/parts/secondary
...
<BLANKLINE>
<zodb_db my-fs>
cache-size 5000
<zeoclient>
server 8100
storage my-fs
name my-fs_zeostorage
var /sample-buildout/parts/secondary/var
cache-size 30MB
<BLANKLINE>
</zeoclient>
mount-point /my-fs
</zodb_db>
<BLANKLINE>

As above, we can override a number of the default parameters::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... zeoserver
... primary
... secondary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... location = var/filestorage/%(fs_part_name)s/Data.fs
... zodb-cache-size = 1000
... zodb-name = %(fs_part_name)s_db
... zodb-mountpoint = /%(fs_part_name)s_mountpoint
... zeo-address = 8101
... zeo-client-cache-size = 50MB
... zeo-storage = %(fs_part_name)s_storage
... zeo-client-name = %(fs_part_name)s_zeostorage_name
... parts =
... my-fs
... '''.replace('%(zope2_location)s', zope2_location))
>>> print system(join('bin', 'buildout') + ' -q')
>>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
>>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
%define INSTANCE /sample-buildout/parts/zeoserver
...
<BLANKLINE>
<filestorage my-fs_storage>
path /sample-buildout/var/filestorage/my-fs/Data.fs
</filestorage>
<BLANKLINE>
>>> primary = os.path.join(sample_buildout, 'parts', 'primary')
>>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
instancehome /sample-buildout/parts/primary
...
<BLANKLINE>
<zodb_db my-fs_db>
cache-size 1000
<zeoclient>
server 8101
storage my-fs_storage
name my-fs_zeostorage_name
var /sample-buildout/parts/primary/var
cache-size 50MB
<BLANKLINE>
</zeoclient>
mount-point /my-fs_mountpoint
</zodb_db>
<BLANKLINE>
>>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
>>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
instancehome /sample-buildout/parts/secondary
...
<BLANKLINE>
<zodb_db my-fs_db>
cache-size 1000
<zeoclient>
server 8101
storage my-fs_storage
name my-fs_zeostorage_name
var /sample-buildout/parts/secondary/var
cache-size 50MB
<BLANKLINE>
</zeoclient>
mount-point /my-fs_mountpoint
</zodb_db>
<BLANKLINE>

By default, the recipe adds the extra filestorages to the first plone.recipe.zope2zeoserver part in the buildout,
and will throw an error if there is more than one part using this recipe. However, you can override this behavior
by specifying a particular ZEO part. In this case, the filestorages will only be added to the Zopes using that ZEO, by default::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... zeoserver1
... zeoserver2
... primary
... secondary
...
... [zeoserver1]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
... zeo-address = 8100
...
... [zeoserver2]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
... zeo-address = 8101
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8101
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8101
...
... [other-zope]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8100
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zeo = zeoserver2
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in open('parts/zeoserver2/etc/zeo.conf').read()
True
>>> 'my-fs' in open('parts/zeoserver1/etc/zeo.conf').read()
False
>>> 'my-fs' in open('parts/primary/etc/zope.conf').read()
True
>>> 'my-fs' in open('parts/other-zope/etc/zope.conf').read()
False


Error conditions
================

Important note: You must place all parts using the collective.recipe.filestorage recipe before
the part for the instances and zeoservers that you are adding the filestorage to. Otherwise you'll
get an error::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... instance
... filestorage
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] The "filestorage" part must be listed before the following parts in ${buildout:parts}: instance
<BLANKLINE>


A buildout with the filestorage recipe must include at least one part using the plone.recipe.zope2instance recipe::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part requires at least one plone.recipe.zope2instance part.


Buildouts with multiple plone.recipe.zope2zeoserver parts will result in an error if the desired ZEO to associate
with is not explicitly specified::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... zeoserver1
... zeoserver2
... primary
... secondary
...
... [zeoserver1]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [zeoserver2]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part found multiple plone.recipe.zope2zeoserver parts; please specify which one to use with the "zeo" option.

Specifying a nonexistent zeo should result in an error::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... zeoserver
... primary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zeo = foobar
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part specifies nonexistant zeo part "foobar".

So should specifying a nonexistent zope part::

>>> write('buildout.cfg',
... '''
... [buildout]
... index = http://pypi.python.org/simple
... parts =
... filestorage
... zeoserver
... primary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zopes = foobar
... parts =
... my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] The "filestorage" part expected but failed to find the following parts in ${buildout:parts}: foobar


Running the tests
=================

The subversion checkout of collective.recipe.filestorage includes a buildout
which installs a script for running the tests.

Just run::

python2.4 bootstrap.py
bin/buildout
bin/test

Known issue: The tests run buildout in a separate process, so it's currently
impossible to put a pdb breakpoint in the recipe and debug during the test.
If you need to do this, set up another buildout which uses collective.recipe.filestorage
as a development egg.


Reporting bugs or asking questions
==================================

There is a shared bugtracker and help desk on Launchpad:
https://bugs.launchpad.net/collective.buildout/

Contributors
************

- David Glick [davisagli], Author

Download
********