collective.recipe.filestorage 0.4

This recipe aids the creation and management of multiple Zope 2 filestorages.

Detailed Documentation

Introduction

This recipe adds additional filestorage and zodb_db stanzas to the zope.conf and zeo.conf files generated by the plone.recipe.zope2instance and plone.recipe.zope2zeoserver recipes. It also creates the directories in which the extra filestorage data files will be created. This makes it easy to add additional filestorages and mountpoints to a Zope 2 instance.

Supported options

parts

A list of filestorage sub-parts to be generated, one per line. (This is different from buildout parts.)

zeo

The name of a plone.recipe.zope2zeoserver part to which we want to add the extra filestorage. Defaults to the first plone.recipe.zope2zeoserver part in the buildout, if any.

zopes

A list of names of plone.recipe.zope2instance parts to which we want to add the extra filestorage. Defaults to all plone.recipe.zope2instance parts connected to the associated plone.recipe.zope2zeoserver part, if any, or all plone.recipe.zope2instance parts, if no ZEO is found.

The following options affect the generated zope.conf and zeo.conf. Each may be specified for all filestorage subparts in the collective.recipe.filestorage buildout part, or for one particular filestorage subpart by placing the option in a new buildout part called ‘filestorage_subpart’, where subpart is the name of the subpart listed in the ‘parts’ option of this recipe. The name of the subpart may be interpolated by using ‘%(fs_part_name)s’ in the option.

location

The location of the Data.fs file, relative to the buildout root directory. Defaults to var/filestorage/%(fs_part_name)s/Data.fs

zodb-name

The name of the ZODB. Defaults to ‘%(fs_part_name)s’.

zodb-cache-size

Set the ZODB cache size, i.e. the number of objects which the ZODB cache will try to hold. Inherits from the associated zope part. Defaults to 5000.

zodb-mountpoint

Set the path to the mountpoint. Defaults to ‘/%(fs_part_name)s’.

zodb-container-class

Set the class of the object being mounted. Defaults to not being set.

zeo-address

Set the port of the associated ZEO server. Inherits from the associated Zope and ZEO parts. Defaults to 8100.

zeo-client-cache-size

Set the size of the ZEO client cache. Inherits from the associated Zope part. Defaults to ‘30MB’.

zeo-storage

Set the id of the ZEO storage. Defaults to ‘%(fs_part_name)s’.

zeo-client-name

Set the name of the ZEO client. Defaults to ‘%(fs_part_name)s_zeostorage’.

zeo-client-client

Set the persistent cache name that is used to construct the cache filenames. Persistent cache files are disabled by default.

blob-storage

Set the directory to be used to store blobs for a standalone Zope instance or a ZEO server. Optional. Defaults to ‘var/blobstorage-%(fs_part_name)s’.

zeo-blob-storage

Set the directory to be used to store blobs for a ZEO client. Defaults to using the same value as blob-storage.

zeo-shared-blob-dir

Boolean that should be ‘on’ if the blob dir is being shared by the ZEO server and client. Defaults to ‘on’.

Example usage

Let’s create and run a minimal buildout that adds an extra filestorage:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')

Our zope.conf should get the extra filestorage stanza automatically injected into it:

>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
%define INSTANCEHOME...instance
...
<BLANKLINE>
<zodb_db my-fs>
    cache-size 5000
    <filestorage >
      path .../var/filestorage/my-fs/my-fs.fs
    </filestorage>
    mount-point /my-fs
</zodb_db>
<BLANKLINE>

The recipe will also create a directory for the new filestorage:

>>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
True

Let’s make sure that the conf files will be regenerated whenever we make a change to a filestorage part, even if the direct configuration for the zope/zeo parts hasn’t changed:

>>> open('buildout.cfg', 'a').write("    my-fs-2\n")
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs-2' in open('parts/instance/etc/zope.conf').read()
True

Let’s make sure that the filestorage directory is not clobbered even if the filestorage part is removed from the buildout:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in os.listdir(os.path.join(sample_buildout, 'var', 'filestorage'))
True

We can override the defaults for a number of settings:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... location = var/filestorage/%(fs_part_name)s/Data.fs
... blob-storage = var/blobstorage-%(fs_part_name)s
... zodb-name = %(fs_part_name)s_db
... zodb-cache-size = 1000
... zodb-mountpoint = /%(fs_part_name)s_mountpoint
... zodb-container-class = Products.ATContentTypes.content.folder.ATFolder
... parts =
...     my-fs
... '''.replace('%(zope2_location)s', zope2_location))
>>> print system(join('bin', 'buildout') + ' -q')
>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
%define INSTANCEHOME...instance
...
<BLANKLINE>
<zodb_db my-fs_db>
    cache-size 1000
    <blobstorage >
      blob-dir .../var/blobstorage-my-fs
      <filestorage >
        path .../var/filestorage/my-fs/Data.fs
      </filestorage>
    </blobstorage>
    mount-point /my-fs_mountpoint
    container-class Products.ATContentTypes.content.folder.ATFolder
</zodb_db>
<BLANKLINE>

A setting can also be modified just for one particular filestorage, by creating a new part with the filestorage_ prefix, like so:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
...
... [filestorage_my-fs]
... zodb-cache-size = 1000
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> instance = os.path.join(sample_buildout, 'parts', 'instance')
>>> print open(os.path.join(instance, 'etc', 'zope.conf')).read()
%define INSTANCEHOME...instance
...
<BLANKLINE>
<zodb_db my-fs>
    cache-size 1000
    <filestorage >
      path .../var/filestorage/my-fs/my-fs.fs
    </filestorage>
    mount-point /my-fs
</zodb_db>
<BLANKLINE>

By default, the recipe adds the extra filestorages to each plone.recipe.zope2instance part in the buildout, but you can tell it to only add it to certain parts:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance1
...     instance2
...
... [instance1]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [instance2]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zopes = instance1
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in open('parts/instance1/etc/zope.conf').read()
True
>>> 'my-fs' in open('parts/instance2/etc/zope.conf').read()
False

Example Usage with ZEO

Here is a minimal buildout including a ZEO server and two ZODB clients:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...     secondary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')

This should result in the appropriate additions to zeo.conf and both zope.conf’s:

>>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
>>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
%define INSTANCE /sample-buildout/parts/zeoserver
...
<BLANKLINE>
    <filestorage my-fs>
      path /sample-buildout/var/filestorage/my-fs/my-fs.fs
    </filestorage>
<BLANKLINE>

>>> primary = os.path.join(sample_buildout, 'parts', 'primary')
>>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/primary
...
<BLANKLINE>
<zodb_db my-fs>
 cache-size 5000
 <zeoclient>
   server 8100
   storage my-fs
   name my-fs_zeostorage
   var /sample-buildout/parts/primary/var
   cache-size 30MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs
</zodb_db>
<BLANKLINE>

>>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
>>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/secondary
...
<BLANKLINE>
<zodb_db my-fs>
 cache-size 5000
 <zeoclient>
   server 8100
   storage my-fs
   name my-fs_zeostorage
   var /sample-buildout/parts/secondary/var
   cache-size 30MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs
</zodb_db>
<BLANKLINE>

As above, we can override a number of the default parameters:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...     secondary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... location = var/filestorage/%(fs_part_name)s/Data.fs
... blob-storage = var/blobstorage-%(fs_part_name)s
... zodb-cache-size = 1000
... zodb-name = %(fs_part_name)s_db
... zodb-mountpoint = /%(fs_part_name)s_mountpoint
... zeo-address = 8101
... zeo-client-cache-size = 50MB
... zeo-storage = %(fs_part_name)s_storage
... zeo-client-name = %(fs_part_name)s_zeostorage_name
... parts =
...     my-fs
... '''.replace('%(zope2_location)s', zope2_location))
>>> print system(join('bin', 'buildout') + ' -q')
>>> zeoserver = os.path.join(sample_buildout, 'parts', 'zeoserver')
>>> print open(os.path.join(zeoserver, 'etc', 'zeo.conf')).read()
%define INSTANCE /sample-buildout/parts/zeoserver
...
<BLANKLINE>
    <blobstorage my-fs_storage>
      blob-dir /sample-buildout/var/blobstorage-my-fs
      <filestorage my-fs_storage>
        path /sample-buildout/var/filestorage/my-fs/Data.fs
      </filestorage>
    </blobstorage>
<BLANKLINE>
>>> primary = os.path.join(sample_buildout, 'parts', 'primary')
>>> print open(os.path.join(primary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/primary
...
<BLANKLINE>
<zodb_db my-fs_db>
 cache-size 1000
 <zeoclient>
   blob-dir /sample-buildout/var/blobstorage-my-fs
   shared-blob-dir on
   server 8101
   storage my-fs_storage
   name my-fs_zeostorage_name
   var /sample-buildout/parts/primary/var
   cache-size 50MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs_mountpoint
</zodb_db>
<BLANKLINE>
>>> secondary = os.path.join(sample_buildout, 'parts', 'secondary')
>>> print open(os.path.join(secondary, 'etc', 'zope.conf')).read()
%define INSTANCEHOME /sample-buildout/parts/secondary
...
<BLANKLINE>
<zodb_db my-fs_db>
 cache-size 1000
 <zeoclient>
   blob-dir /sample-buildout/var/blobstorage-my-fs
   shared-blob-dir on
   server 8101
   storage my-fs_storage
   name my-fs_zeostorage_name
   var /sample-buildout/parts/secondary/var
   cache-size 50MB
<BLANKLINE>
 </zeoclient>
 mount-point /my-fs_mountpoint
</zodb_db>
<BLANKLINE>

By default, the recipe adds the extra filestorages to the first plone.recipe.zope2zeoserver part in the buildout, and will throw an error if there is more than one part using this recipe. However, you can override this behavior by specifying a particular ZEO part. In this case, the filestorages will only be added to the Zopes using that ZEO, by default:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver1
...     zeoserver2
...     primary
...     secondary
...     other-zope
...
... [zeoserver1]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
... zeo-address = 8100
...
... [zeoserver2]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
... zeo-address = 8101
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8101
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8101
...
... [other-zope]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
... zeo-address = 8100
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zeo = zeoserver2
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'my-fs' in open('parts/zeoserver2/etc/zeo.conf').read()
True
>>> 'my-fs' in open('parts/zeoserver1/etc/zeo.conf').read()
False
>>> 'my-fs' in open('parts/primary/etc/zope.conf').read()
True
>>> 'my-fs' in open('parts/other-zope/etc/zope.conf').read()
False

Error conditions

Important note: You must place all parts using the collective.recipe.filestorage recipe before the part for the instances and zeoservers that you are adding the filestorage to. Otherwise you’ll get an error:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     instance
...     filestorage
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] The "filestorage" part must be listed before the following parts in ${buildout:parts}: instance
<BLANKLINE>

A buildout with the filestorage recipe must include at least one part using the plone.recipe.zope2instance recipe:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part requires at least one plone.recipe.zope2instance part.

Buildouts with multiple plone.recipe.zope2zeoserver parts will result in an error if the desired ZEO to associate with is not explicitly specified:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver1
...     zeoserver2
...     primary
...     secondary
...
... [zeoserver1]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [zeoserver2]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [secondary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part found multiple plone.recipe.zope2zeoserver parts; please specify which one to use with the "zeo" option.

Specifying a nonexistent zeo should result in an error:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zeo = foobar
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] "filestorage" part specifies nonexistant zeo part "foobar".

So should specifying a nonexistent zope part:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     zeoserver
...     primary
...
... [zeoserver]
... recipe = plone.recipe.zope2zeoserver
... zope2-location = %(zope2_location)s
...
... [primary]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
... zeo-client = 1
...
... [filestorage]
... recipe = collective.recipe.filestorage
... zopes = foobar
... parts =
...     my-fs
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
While:
...
Error: [collective.recipe.filestorage] The "filestorage" part expected but failed to find the following parts in ${buildout:parts}: foobar

If the Zope/ZEO parts are being automatically identified, let’s make sure that we don’t accidentally “wake up” parts that would not otherwise be included in the buildout:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     my-fs
...
... [foobar]
... recipe = plone.recipe.distros
... urls =
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q')
>>> 'foobar' in os.listdir(os.path.join(sample_buildout, 'parts'))
False

Make sure that instance parts are found correctly in buildouts using ‘extends’ and the += or -= options:

>>> write('buildout.cfg',
... '''
... [buildout]
... extends = base.cfg
... parts =
...     filestorage
...     instance
...
... [instance]
... recipe = plone.recipe.zope2instance
... zope2-location = %(zope2_location)s
... user = me
...
... [filestorage]
... recipe = collective.recipe.filestorage
... parts =
...     extendstest
... ''' % globals())
>>> write('prod.cfg',
... '''
... [buildout]
... extends = buildout.cfg
... parts +=
...     foobar
...
... [foobar]
... recipe = plone.recipe.distros
... urls =
... ''' % globals())
>>> print system(join('bin', 'buildout') + ' -q -c prod.cfg')
>>> 'extendstest' in open(os.path.join(instance, 'etc', 'zope.conf')).read()
True

Running the tests

The subversion checkout of collective.recipe.filestorage includes a buildout which installs a script for running the tests.

Just run:

python2.4 bootstrap.py
bin/buildout
bin/test

Known issue: The tests run buildout in a separate process, so it’s currently impossible to put a pdb breakpoint in the recipe and debug during the test. If you need to do this, set up another buildout which uses collective.recipe.filestorage as a development egg.

Reporting bugs or asking questions

There is a shared bugtracker and help desk on Launchpad: https://bugs.launchpad.net/collective.buildout/

Change history

0.4 (2009-12-30)

• Add support for creating blob storages associated with the created filestorage parts. [davisagli]

• Fixed message indicating which parts the filestorage part must be installed before. This fixes https://bugs.launchpad.net/bugs/433877 (Thanks, Jean Jordaan) [davisagli]

0.3 (2009-03-19)

• Avoid causing the installation of all parts, even those not listed in the [buildout] parts variable. Thanks to Dylan Jay for the diagnosis and fix. [davisagli]

• Fixed restructured text of long description (README.txt). [maurits]

0.2 (2008-04-30)

• Added zodb_container_class option. [davisagli]

0.1 (2008-04-22)

• Happy Earth Day! [davisagli]

• Created recipe with ZopeSkel. Initial implementation. [davisagli]

Contributors

• David Glick [davisagli], Author

Thanks to:

• Dylan Jay