iw.fss 2.7.6

Archetypes storage for storing fields raw values on the file system.

By Ingeniweb.

---

Table of contents

• 1 About FileSystemStorage

• 2 Copyright and license

• 3 Requirements

• 4 Upgrading

  • 4.1 From FSS 2.5.x and older

  • 4.2 From iw.fss 2.7.1

• 5 Migrating between strategies

• 6 Installation

  • 6.1 With buildout

  • 6.2 From SVN reposo

  • 6.3 Configuring your Plone sites

• 7 Storage strategies

  • 7.1 Strategies overview

  • 7.2 Flat storage strategy

  • 7.3 Directory storage strategy

  • 7.4 Site storage strategy 1

  • 7.5 Site storage strategy 2

• 8 FSS main configuration

• 9 Configuration panel

  • 9.1 FSS and RDF

  • 9.2 FSS maintenance

• 10 Zope backups

• 11 Caveats and pittfalls

• 12 Developer’s corner

  • 12.1 Using FSS in your content types

  • 12.2 Monkey patching a third party content type

    • 12.2.1 The Python way

    • 12.2.2 The ZCML way

  • 12.3 Storing Plone standard content types with FSS

  • 12.4 Customize RDF info set

    • 12.4.1 RDF script interface

    • 12.4.2 Simple example

    • 12.4.3 More complex example

    • 12.4.4 Recommanded practices on custom RDF

• 13 Testing

• 14 Other documentation

• 15 Downloads

• 16 Subversion repository

• 17 Support and feedback

• 18 Credits

---

1 About FileSystemStorage

FileSystemStorage (FSS) is an Archetypes storage for storing fields raw values on the file system. This storage is used to avoid unnecessary growth of the ZODB’s FileStorage (Data.fs) when using a lot of large files.

Please note that FSS is a Plone component for content types developers. Do not expect anything more in your Plone site if you don’t use Plone components that may use FSS, such as AttachmentField.

2 Copyright and license

Copyright (c) 2005 - 2008 Ingeniweb SAS

This software is subject to the provisions of the GNU General Public License, Version 2.0 (GPL). A copy of the GPL should accompany this distribution. THIS SOFTWARE IS PROVIDED “AS IS” AND ANY AND ALL EXPRESS OR IMPLIED WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE

See the LICENSE file that comes with this product.

3 Requirements

• Plone 3.0+ bundle

4 Upgrading

4.1 From FSS 2.5.x and older

The way we configure the storage and backup paths as well as the storage strategies HAVE CHANGED IN DEPTH. There’s no automatic migration from this version. Please create a configuration file that clones your old configlet preferences as indicated in FSS main configuration.

4.2 From iw.fss 2.7.1

From this version and later version, iw.fss registers upgrade steps in GenericSetup. When upgrading to a newer version of iw.fss, just have a look at the “Upgrade” tab of the portal_setup tool in your Plone site.

Warning, due to the removal of the portal_fss tool upgrading from 2.7.1 does not retain the RDF options (activated and specific script). If you used RDF with 2.7.1, you need to redo the same RDF setting in the FileSystemStorage config panel.

5 Migrating between strategies

Use the bin/strategymigrator.py shell utility that ships with FSS. Get more info on this utility using the –help option.

Note that only flat -> directory and directory -> flat are available today. Sponsorship is welcome - as indicated in Support and feedback - if you need other strategy migrations.

6 Installation

6.1 With buildout

This example speaks of itself:

[buildout]
...
eggs =
  ...
  iw.fss
  ...

[instance]
...
zcml =
  ...
  iw.fss
  iw.fss-meta
  ...

You may also use the iw.recipe.fss recipe egg to configure fully an instance.

6.2 From SVN reposo

• Inflate iw.fss package into your zope instance lib/python directory. I’m sure it’s already done.

• Add $INSTANCE_HOME/etc/package-includes/iw.fss-meta.zcml file with this line:

  <include package="iw.fss" file="meta.zcml"/>

• Add $INSTANCE_HOME/etc/package-includes/iw.fss-configure.zcml file with this line:

  <include package="iw.fss" />

6.3 Configuring your Plone sites

• Read carefully Storage strategies and FSS main configuration below and configure your instance in plone-filesystemstorage.conf accordingly. You may use the iw.recipe.fss recipe in your buildout.cfg to do this.

• Start your Zope instance

• In ZMI, go into your plone site and install the product using portal_quickinstaller

• Optionally, tweak some - self explained - preferences in the FileSystemStorage configuration panel of your Plone site.

7 Storage strategies

A storage strategy defines how field values will be stored in your files system.

7.1 Strategies overview

FSS comes with 4 storage strategies. Each strategy requires two directories:

• The storage directory stores the values of relevant fields in files according to the selected strategy.

• The backup directory that keeps the values of the fields of deleted contents for “Undo” purpose.

• Only one storage strategy can be selected on a given Plone site, but different Plone site in the same Zope instance may have distinct storage strategies.

• You need the best performances: choose the Flat storage strategy or Directory storage strategy.

• You need a storage that looks like your Plone site (in example to publish the storage directory in a read only samba or NFS share): choose the Site storage strategy 1 or the Site storage strategy 2.

• If you care about CMFEdition support (means that one of the content types using FSS is versioned with CMFEditions), you should not use the Site storage strategy 1 or the Site storage strategy 2.

See FSS main configuration to configure the best suited strategy on your Zope instance and Plone sites.

7.2 Flat storage strategy

All field values are stored in a flat structure. This strategy is the default one.

• Filename of these stored values: <uid of content>_<field name>

• Filename of rdf file: <uid of content>_<field name>.rdf

• Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated.

Example of storage:

fssDirectory
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image
|
|- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file
|
|- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

7.3 Directory storage strategy

All field values are stored in a directory structure. Sub directories are defined on two level. First level of directory uses the 2 characters of content uid. Second level of directory uses the 4 characters of content uid. Backup files are stored in a flat structure.

• Filename of these stored values: <uid of content>_<field name>

• Filename of rdf file: <uid of content>_<field name>.rdf

• Filename of backup values: <uid of content>_<field name>.bak

RDF files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- f42
   |
   |- f42ad
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_image.rdf
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file
      |
      |- f42ad00adb7d4580f929d734bf1ed3ab_file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

Depending in one hand on your file system performances facing a huge amount of files in the same directory, and in the other hand on the number of contents relying on FSS, you might choose this strategy or the Flat storage strategy.

7.4 Site storage strategy 1

All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.

• Filename of these stored values: Filename of field value or field name if not defined

• Filename of rdf file: <field name>.rdf

• Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- image
         |  |
         |  |- moutain.jpg
         |  |
         |  |- image.rdf
         |
         |- file
            |
            |- diary.odt
            |
            |- file.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

7.5 Site storage strategy 2

All field values are stored in a directory structure mirroring structure of PloneSite. Backup files are stored in a flat structure.

• Filename of these stored values: Filename of field value or field name if not defined

• Filename of rdf file: <field filename>.rdf

• Filename of backup values: <uid of content>_<field name>.bak

Rdf files are not backed up. They are automatically generated

Example of storage:

fssDirectory
|
|- members
   |
   |- john
      |
      |- dummy-document
         |
         |- fss.cfg
         |
         |- moutain.jpg
         |
         |- mountain.jpg.rdf
         |
         |- diary.odt
         |
         |- diary.odt.rdf

fssBackupDirectory
|
|- 9efeb7638fb35f5f9b9955da3e2dbfec_file.bak

8 FSS main configuration

FSS is mainly configured the ZConfig way. At startup, the configuration file will be searched (in that order) in:

• $INSTANCE_HOME/etc/plone-filesystemstorage.conf

• /path/to/iw/fss/etc/plone-filesystemstorage.conf

• /path/to/iw/fss/etc/plone-filesystemstorage.conf.in

A sample working configuration in provided in this last file. It assumes you have $INSTANCE_HOME/var/fss_storage and $INSTANCE_HOME/var/fss_backup directories, both being read/write enabled to the user that runs the Zope process, unless Zope will raise an error at startup.

All configuration doc you need is in plone-filesystemstorage.conf.in comments.

Inconsistent configuration features raise explicit error messages at Zope startup.

Note that we didn’t include this in zope.conf in order to keep Zope 2.7 and Zope 2.8 compatibility.

9 Configuration panel

9.1 FSS and RDF

As mentioned above in Storage strategies, FSS can optionally store RDF files in the back-end storage. These RDF files conform the DCMES-XML standard on XML expression of DublinCore elements.

If you select the Flat storage strategy, the RDF files may be used to build a files tree as close as possible of the Plone tree structure from the storage back-end, with the build_fs_tree.py utility:

$ cd /path/to/iw/fss/bin
$ python build_fs_tree.py --help

9.2 FSS maintenance

The “Maintenance” tab of FSS configuration panel shows some statistics about the amount of files managed by FSS.

In addition 2 additional buttons are provided:

• Update FSS files : Cleans up the back-end storage directories. Files not referenced from a content object are removed.

• Update RDF files : Builds all RDF files, in the event you chose to select the generetion of RDF files after having inserted relevant contents.

10 Zope backups

Of course, you need to synch your Data.fs backups with the storage paths backups, unless you may restore corrupted/incomplete sites.

11 Caveats and pittfalls

Note that whatever’s the selected strategies, DO NEVER CHANGE ANY FILE IN THE STORAGE PATHS UNLESS YOU KNOW WHAT YOU DO, otherwise you’ll loose your contents, your job, your money, your friends, your wife/husband and children.

DO NEVER CHANGE THE STRATEGY FOR A PLONE SITE ONCE THIS SITE HAS FIELDS STORED IN ITS FSS STORAGE PATH unless you ran the appropriate strategy migrator - see Migrating between strategies otherwise you’re loose your contents, your job, …

Thought it is possible to share the same storage path and backup path within various Plone sites, since you don’t mind of your data in development sites, you should really configure FSS such each Plone site has its own private storage path and backup path. In cas you ignore this warning, DO NEVER CLICK ANY BUTTON IN THE MAINTENANCE TAB OF THE FSS CONFIGURATION PANEL otherwise you’ll loose your contents, your job, …

Zexp exports don’t embed the associated FSS storage. So don’t move your Plone site within Zope instances using zexp exports unless you move the storage and backup directories along with the zexp file.

After changing your configuration (see FSS main configuration), always restart your Zope instance in foreground since configuration errors are not reported when Zope is started as daemon or Windows service.

12 Developer’s corner

12.1 Using FSS in your content types

We assume that creating AT based content types is a familiar activity.

When creating the schema of a content type, you usually don’t mention the storage of your fields since AT provides a default ZODB storage. You can choose another storage, either another one that ships in the Archetypes bundle, or a third party storage like FSS.

Here is a small example that doesn’t need much comments:

# Usual Zope/CMF/Plone/Archetypes imports
...
from iw.fss.FileSystemStorage import FileSystemStorage
...
my_schema = Schema((
    FileField('file',
              ...
              storage=FileSystemStorage(),
              widget=FileWidget(...)
              ),
    ...
    ))
 ...

You may have a look at examples/FSSItem.py to see a demo content type using FSS. If you want to play with this content type:

$ $INSTANCE_HOME/bin/zopectl stop
$ export FSS_INSTALL_EXAMPLE_TYPES=1
$ $INSTANCE_HOME/bin/zopectl start

12.2 Monkey patching a third party content type

12.2.1 The Python way

This works with any Zope version listed in the Requirements. If you run a Zope 2.9 or later instance, you should prefer The ZCML way.

This simple example shows how to plug the ATFile standard type from any custom product. Let’s say we’re in the __init__.py of your custom product:

...
from iw.fss import zcml as fss_utils
from Products.ATContentTypes import atct
...
fss_utils.patchATType(atct.ATFile, ['file'])
...

12.2.2 The ZCML way

This works only with Zope 2.9 and up, and is the recommanded way to wire your AT based content types with FSS since this way hides the API details that may be deprecated in the future.

You just need to add the “fss” namespace support and some directives in your “configure.zcml” like this to have the same effect as in the example of The Python way above:

<configure
  ...
  xmlns:zcml="http://namespaces.zope.org/zcml"
  xmlns:fss="http://namespaces.ingeniweb.com/filesystemstorage"
  ...>
  ...
  <fss:typeWithFSS
    zcml:condition="installed iw.fss"
    class="Products.ATContentTypes.atct.ATFile"
    fields="file" />
  ...
</configure>

The (required) attributes of this directive are:

• class: the dotted name of the AT based content type class.

• fields: one or more (space separated) field names to wire with FSS.

12.3 Storing Plone standard content types with FSS

Include this ZCML element either in your instance package-includes directory or in any personal configure.zcml in your instance packages:

<include package="iw.fss" file="atct.zcml" />

This ZCML file wires FSS with the relevant content types that ship with Plone bundle:

• File

• Image

• News Item

12.4 Customize RDF info set

You can add data to the RDF info set via a FSS dedicated - and optional - hook.

Open the FSS configuration panel and give the name of your personal RDF data script, let’s call it fss_custom_rdf for this example. This script will add or change the RDF default data set.

12.4.1 RDF script interface

The script may be in any Plone layer like custom or any other in the skins path.

Such script is expected to have 4 parameters:

• name: the name of the field

• instance: the content object being processed

• properties: FSS field properties mapping, depending on the type of field associated to FSS storage.

• default_rdf: a mapping structure of the RDF data, including the namespaces, and the elements.

Such script is expected to return an updated default_rdf mapping structure.

Please read the source of classes FSSInfo, FSSFileInfo and FSSImageInfo from module FileSystemStorage.py, RDFWriter from module rdf.py for detailed information on formats of expected data.

12.4.2 Simple example

We need to add the date of RDF file generation in an element like this:

<fss:rdf_date>2007-05-13 13:21:00</fss:rdf_date>

Here’s the script:

## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
"""We add the date of RDF file generation"""

rdf_args = default_rdf.copy()

# Add the date of RDF generation
new_props = []
new_props.extend(rdf_args['field_props'])
new_props.append({'id': 'fss:rdf_date', 'value': DateTime().ISO()})
rdf_args['field_props'] = new_props

return rdf_args

12.4.3 More complex example

In this more complex (and stupid - don’t do this ;) example, we remove the elements from fss namespace and simulate the standard behaviour for other elements (dc and rdf elements).

Here’s the code:

## Script (Python) "fss_custom_rdf"
##bind container=container
##bind context=context
##bind namespace=
##bind script=script
##bind subpath=traverse_subpath
##parameters=name, instance, properties, default_rdf
##title=
##
from Products.CMFCore.utils import getToolByName

rdf_args = {}

# Set RDF charset
ptool = getToolByName(instance, 'portal_properties')
rdf_args['charset'] = ptool.site_properties.default_charset

# Set RDF namespaces
rdf_args['namespaces'] = (
    {'id': 'xmlns:rdf', 'value': 'http://www.w3.org/1999/02/22-rdf-syntax-ns#'},
    {'id': 'xmlns:dc', 'value': 'http://purl.org/dc/elements/1.1/'},
    {'id': 'xmlns:fss', 'value': 'http://namespace.ingeniweb.com/fss'},)

# Set field url
utool = getToolByName(instance, 'portal_url')
portal_path = utool.getPortalObject().getPhysicalPath()
portal_path_len = len(portal_path)
rel_path = '/'.join(instance.getPhysicalPath()[portal_path_len:])
fss_path = '%s/%s' % (rel_path, name)
rdf_args['field_url'] = fss_path

# Set RDF properties
props = (
   {'id': 'dc:title', 'value': instance.title_or_id()},
   {'id': 'dc:description', 'value': instance.Description()},
   {'id': 'dc:language', 'value': instance.Language()},
   {'id': 'dc:creator', 'value': instance.Creator()},
   {'id': 'dc:date', 'value': instance.modified()},
   {'id': 'dc:format', 'value': properties.get('mimetype', 'text/plain')},
   )
rdf_args['field_props'] = props

# Returns the RDF structure
return rdf_args

12.4.4 Recommanded practices on custom RDF

Do not change the existing elements under fss namespace unless the bin/build_fs_tree.py utility won’t work. In other terms, do not use the script of the More complex example.

Adding or changing elements to the dc and rdf namespaces may screw up external utilities based on canonical DCMES-XML standard.

Consider adding your own namespace for your custom extra elements.

13 Testing

Please read ./tests/README.txt.

14 Other documentation

See ./doc/* and ./locales/README.txt

15 Downloads

You may find newer stable versions of FSS and pointers to related informations (tracker, doc, …) from http://pypi.python.org/pypi/iw.fss

16 Subversion repository

Stay in tune with the freshest (maybe unstable) versions or participate to the FileSystemStorage evolutions:

https://svn.plone.org/svn/collective/iw.fss

17 Support and feedback

Please read all the documentation that comes with this product before asking for support, unless you might get a RTFM reply ;)

Localisation issues - other than french - should be reported to the relevant translators (see Credits below).

Report bugs using the tracker (the Tracker link from http://plone.org/iw.fss). Please provide in your bug report:

• Your configuration (Operating system+Zope+Plone+Products/versions).

• The storage strategy in use.

• The full traceback if available.

• One or more scenario that triggers the bug.

Note that we do not support bug reports on Subversion trunk or branches checkouts.

Mail to Ingeniweb support in English or French to ask for specific support.

Donations are welcome for new features requests

18 Credits

• Main developer: Cyrille Lebeaupin

• ZConfig and ZCML support: Gilles Lenfant

• HTTP “Range” support: Youenn Boussard

• Paris 2008 sprint additional contributors: - Mehdi Benammar - Romain Griffiths

See locales/README.txt about localization credits.

---