Products.csvreplicata 1.1.10

CSV import/export for Archetypes or other contents (via plugins for the later). Created by Makina Corpus.

Introduction
============

Allows to export and import AT objects contents and hierarchy.

It supports also other contents or objects via plugins (ZCA).
This part mainly targeted for a developper audiance, so see interfaces.py / adapters.py.


We have plans to integrate the export/import code with transmogrifier_, [ also_ ] but we have no ETA For now.

.. _transmogrifier: http://pypi.python.org/pypi/collective.transmogrifier/
.. _also: http://pypi.python.org/pypi/plone.app.transmogrifier/


Repository: git_


.. _git: https://github.com/collective/Products.csvreplicata.git

A Makina Corpus add-on
-------------------------
|makinacom|_

* `Planet Makina Corpus <http://www.makina-corpus.org>`_
* `Contact us <mailto:[e-mail]>`_

.. |makinacom| image:: http://depot.makina-corpus.org/public/logo.gif
.. _makinacom: http://www.makina-corpus.com


Products.csvreplicata Installation
======================================

To install Products.csvreplicata into the global Python environment (or a workingenv),
using a traditional Zope 2 instance, you can do this:

* When you're reading this you have probably already run
``easy_install Products.csvreplicata``. Find out how to install setuptools
(and EasyInstall) here:
http://peak.telecommunity.com/DevCenter/EasyInstall

* If you are using Zope 2.9 (not 2.10), get `pythonproducts`_ and install it
via::

python setup.py install --home /path/to/instance

into your Zope instance.

* Create a file called ``Products.csvreplicata-configure.zcml`` in the
``/path/to/instance/etc/package-includes`` directory. The file
should only contain this::

<include package="Products.csvreplicata" />

.. _pythonproducts: http://plone.org/products/pythonproducts


Alternatively, if you are using zc.buildout and the plone.recipe.zope2instance
recipe to manage your project, you can do this:

* Add ``Products.csvreplicata`` to the list of eggs to install, e.g.

::

[buildout]
...
eggs =
...
Products.csvreplicata

* Tell the plone.recipe.zope2instance recipe to install a ZCML slug

::

[instance]
recipe = plone.recipe.zope2instance
...
zcml =
Products.csvreplicata

* Re-run buildout, e.g. with:

$ ./bin/buildout

You can skip the ZCML slug if you are going to explicitly include the package
from another package's configure.zcml file.


Because its top level Python namespace package is called ``Products``, this
package can also be installed in Zope 2 as an old style **Zope 2 Product**.

For that, move (or symlink) the ``csvreplicata`` folder of this project
(``Products.csvreplicata/Products/csvreplicata``) into the ``Products`` directory of
the Zope instance it has to be installed for, and restart the server.

You can also skip the ZCML slug if you install this package the **Zope 2
Product** way.

Detailled documentation
=======================


Interfaces
-------------
import interfaces and classes ::

>>> from zope.interface.verify import verifyClass
>>> from zope.interface import implements
>>> from Products.csvreplicata.handlers.base import CSVdefault
>>> from Products.csvreplicata.handlers.file import CSVFile
>>> from Products.csvreplicata.interfaces import ICSVDefault, ICSVFile

Verify implementation ::

>>> verifyClass(ICSVDefault, CSVdefault)
True
>>> verifyClass(ICSVFile, CSVFile)
True



Export / Import in plain format
------------------------------------

Export
+++++++++
here we export folders and documents
::

>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document' , id='doc1' , title="Document 1")
>>> id=self.folder.invokeFactory('Document' , id='doc2' , title="Document 2")
>>> id=self.folder.invokeFactory('News Item' , id='news1' , title="news 'super' 3")
>>> id=self.folder.invokeFactory('Document' , id='doc4' , title="Document 4")
>>> id=self.folder.invokeFactory('Folder' , id='sub1' , title="mytest")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc11' , title="Document 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('News Item', id='news21' , title="news 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc31' , title="Document 2 du dossier 1")
>>> self.portal.portal_csvreplicatatool.getPlainFormat()
False

>>> self.portal.portal_csvreplicatatool.getEncoding()
'UTF-8'
>>> self.portal.portal_csvreplicatatool.getDelimiter()
';'
>>> self.portal.portal_csvreplicatatool.getStringdelimiter()
'"'
>>> self.portal.portal_csvreplicatatool.replicabletypes = {'Document': ['default'], 'Folder': ['default'], 'News': ['Default']}
>>> from Products.csvreplicata.interfaces import Icsvreplicata
>>> repl = Icsvreplicata(self.folder)
>>> print repl.csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue()
"/plone/Members/test_user_1_";"..."
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc1";"Document";"Document 1";"";"";"simple_publication_workflow";"private";"";""
"";"doc2";"Document";"Document 2";"";"";"simple_publication_workflow";"private";"";""
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"news1";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc4";"Document";"Document 4";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"sub1";"Folder";"mytest";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"news21";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"";"simple_publication_workflow";"private"
<BLANKLINE>

This output is not that good to deal with CSV apis, we will try to export it to a flat structure.
::

>>> from Products.csvreplicata import replicator
>>> self.portal.portal_csvreplicatatool.setPlainFormat(True)
>>> self.portal.portal_csvreplicatatool.getPlainFormat()
True
>>> repl = Icsvreplicata(self.folder)
>>> print Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue() # doctest: +ELLIPSIS
"startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";...
"Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";...
"/plone/Members/test_user_1_";"...";"";"doc1";"Document";"Document 1";...
"/plone/Members/test_user_1_";"...";"";"doc2";"Document";"Document 2";...
"/plone/Members/test_user_1_";"...";"";"news1";"News Item";...
"/plone/Members/test_user_1_";"...";"";"doc4";"Document";"Document 4";...
"/plone/Members/test_user_1_";"...";"";"sub1";"Folder";"mytest";...
"/plone/Members/test_user_1_";"...";"sub1";"doc11";"Document";"Document 1 du dossier 1";...
"/plone/Members/test_user_1_";"...";"sub1";"news21";"News Item";...
"/plone/Members/test_user_1_";"...";"sub1";"doc31";"Document";"Document 2 du dossier 1";...
<BLANKLINE>

Redo the export but with a plugin that find the title on the object.
::

>>> from Products.csvreplicata import adapters
>>> class CustomExporter(adapters.CSVReplicataExportImportPluginAbstract):
... def __init__(self, *args, **kwargs):
... adapters.CSVReplicataExportImportPluginAbstract.__init__(self, *args, **kwargs)
... self.ids.append('title')
... def fill_values(self, row, row_ids):
... """."""
... for id in row_ids:
... if id.replace(self.prefix, '') in self.ids:
... index = row_ids.index(id)
... if index < len(row):
... row[index] = self.context.Title()
... def set_values(self, row, row_ids):
... """."""
... print "plugin.setValue called with %s <-> %s" % (row, row_ids)
...
>>> provideAdapter(CustomExporter, (interfaces.Icsvreplicata, zope.interface.Interface), interfaces.ICSVReplicataExportImportPlugin, name ='fooplugin' )
>>> from csv import DictReader
>>> content = Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=None).getvalue()
>>> items = [item for item in DictReader(StringIO(content), delimiter=";", quotechar='"')]
>>> keys = items[0].keys();keys.sort()

As we cant predict order of the keys, doing some magic to order them before testing.
::

>>> pprint([[(key, item[key]) for key in keys if not 'date' in key]for item in items], width=130) # doctest: +REPORT_NDIFF
[[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain',
'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state',
'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title',
'ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title'),
('description', 'label_...'),
('id', 'Identifier'),
('parent', 'Parent folder'),
('startpoint', 'Start point'),
('text', 'label_body_text'),
('title', 'Title'),
('type', 'Content type')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1'),
('description', ''),
('id', 'doc1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 1'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2'),
('description', ''),
('id', 'doc2'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 2'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', "news 'super' 3"),
('description', ''),
('id', 'news1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', ''),
('type', 'News Item')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 4'),
('description', ''),
('id', 'doc4'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 4'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'mytest'),
('description', ''),
('id', 'sub1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'mytest'),
('type', 'Folder')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1 du dossier 1'),
('description', ''),
('id', 'doc11'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 1 du dossier 1'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'news 1 du dossier 1'),
('description', ''),
('id', 'news21'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', ''),
('type', 'News Item')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2 du dossier 1'),
('description', ''),
('id', 'doc31'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 2 du dossier 1'),
('type', 'Document')]]




Import
+++++++++

Then, now that we got a working export, what about importing it
::

>>> id = self.folder.invokeFactory('Folder' , id='fa' , title="tests import")
>>> id = self.folder.invokeFactory('Folder' , id='fb' , title="tests import")
>>> fa = self.folder.fa; fb = self.folder.fb

Import in CSVReplicata plain format, an entry per line without contextual type hinting
::

>>> CSV = StringIO("""\
... "startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"text"
... "Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";"label_...";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"label_body_text"
... "/plone/Members/test_user_1_";20090101010101;"";"doc1";"Document";"Document 1";"";"Document 1";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc2";"Document";"Document 2";"";"Document 2";""
... "/plone/Members/test_user_1_";20090101010101;"";"news1";"News Item";"";"";"news 'super' 3";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc4";"Document";"Document 4";"";"Document 4";""
... "/plone/Members/test_user_1_";20090101010101;"";"sub1";"Folder";"mytest";"";"mytest";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"Document 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"news21";"News Item";"";"";"news 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"Document 2 du dossier 1";""
... """)
>>> print Icsvreplicata(self.folder.fa).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=True)
p...
(8, 0, ..., [])
>>> self.folder.fa.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fa.sub1.objectIds()
['doc11', 'news21', 'doc31']

Import in CSVReplicata orignal format, an entry per line with contextual type hinting
::

>>> CSV = StringIO("""\
... "/plone/Members/test_user_1_";20100101010101
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc1";"Document";"Document 1";"";""
... "";"doc2";"Document";"Document 2";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "";"news1";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc4";"Document";"Document 4";"";""
... "parent";"id";"type";"title";"description"
... "Parent folder";"Identifier";"Content type";"Title";"label_..."
... "";"sub1";"Folder";"mytest";""
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc11";"Document";"Document 1 du dossier 1";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "sub1";"news21";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc31";"Document";"Document 2 du dossier 1";"";""
... """)
>>> print Icsvreplicata(self.folder.fb).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=False)
p...
(8, 0, ..., [])
>>> self.folder.fb.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fb.sub1.objectIds()
['doc11', 'news21', 'doc31']



Default handlers
----------------

Verify the default handlers provided for Archetypes fields::

>>> handlersDict = self.portal.portal_csvreplicatatool.getHandlers()
>>> handlers = handlersDict.keys();handlers.sort()
>>> pprint(handlers)
['Products.Archetypes.Field.BooleanField',
'Products.Archetypes.Field.DateTimeField',
'Products.Archetypes.Field.FileField',
'Products.Archetypes.Field.FloatField',
'Products.Archetypes.Field.ImageField',
'Products.Archetypes.Field.IntegerField',
'Products.Archetypes.Field.LinesField',
'Products.Archetypes.Field.ReferenceField',
'Products.Archetypes.Field.StringField',
'Products.Archetypes.Field.TextField',
'Products.AttachmentField.AttachmentField.AttachmentField',
'default_handler',
...]
>>> pprint([(h, handlersDict[h]['handler_class']) for h in handlers], width=130)
[('Products.Archetypes.Field.BooleanField', <Products.csvreplicata.handlers.base.CSVBoolean object at ...>),
('Products.Archetypes.Field.DateTimeField', <Products.csvreplicata.handlers.base.CSVDateTime object at ...>),
('Products.Archetypes.Field.FileField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('Products.Archetypes.Field.FloatField', <Products.csvreplicata.handlers.base.CSVFloat object at ...>),
('Products.Archetypes.Field.ImageField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('Products.Archetypes.Field.IntegerField', <Products.csvreplicata.handlers.base.CSVInteger object at ...>),
('Products.Archetypes.Field.LinesField', <Products.csvreplicata.handlers.base.CSVLines object at ...>),
('Products.Archetypes.Field.ReferenceField', <Products.csvreplicata.handlers.reference.CSVReference object at ...>),
('Products.Archetypes.Field.StringField', <Products.csvreplicata.handlers.base.CSVString object at ...>),
('Products.Archetypes.Field.TextField', <Products.csvreplicata.handlers.base.CSVText object at ...>),
('Products.AttachmentField.AttachmentField.AttachmentField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('default_handler', <Products.csvreplicata.handlers.base.CSVdefault object at ...>),
(...


What happened with csvreplicata during import/export if MyField is not in tool's
handlers. replicator.py apllies default_handler on it::

{'default_handler':
{'handler_class': base.CSVdefault(),'file': False}}





The replicator exporter/downloader
------------------------------------

Export in normal mode
+++++++++++++++++++++++++
here we export folders and documents ::

>>> from Products.csvreplicata.browser import manager
>>> self.portal.portal_csvreplicatatool.replicabletypes = \
... {'Document':['default'], 'Folder':['default'],
... 'News Item': ['default'], 'File': ['default'] }
>>> import re
>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document' , id='doc1' , title="Document 1")
>>> id=self.folder.invokeFactory('Document' , id='doc2' , title="Document 2")
>>> params = {"datetimeformat": '%d/%m/%Y %H:%M:%S',
... "vocabularyvalue": "No",
... "encoding": "UTF-8",
... "delimiter": ";",
... "stringdelimiter": '"',
... "exportable_content_types": ["News Items", "Document", "Folder"],
... }
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from Products.csvreplicata import replicator
>>> print ''.join([a for a in repl.doExport()])
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"text";...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";"";...
"";"doc2";"Document";"Document 2";"";...
<BLANKLINE>

>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.csv']),
('content-length', ['...']),
('content-type', ['text/csv;charset=UTF-8'])]


Export as zip when there are files out there and we want them
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Now adding and exporting a file::

>>> params['exportfiles'] = 'Yes'
>>> params["exportable_content_types"].append('File')
>>> id=self.folder.invokeFactory('File', id='f1' , title="File 1")
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> f1 = self.folder.f1
>>> f1.getFile().data = 'foo'
>>> f1.setFilename('bar')
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from StringIO import StringIO
>>> import zipfile
>>> content = StringIO([a for a in repl.doExport()][0])
>>> zip = zipfile.ZipFile(content)
>>> print zip.read('export.csv')
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";...
"";"doc2";"Document";"Document 2";...
"parent";"id";"type";"title";"description";...
"Parent folder";"Identifier";"Content ...
"";"f1";"File";"File 1";"";...
<BLANKLINE>


>>> zip.printdir()
File Name...
bar...
export.csv...
>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.zip']),
('content-length', ['...']),
('content-type', ['application/zip'])]


The File Stream Iterator
+++++++++++++++++++++++++

This object returns a generator to read our big files!::

>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fpath = tempfile.mkstemp('foo')[1]
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> len(FileStreamIterator(fpath))
3

We can play with chunks to divide rendering into small parts ::

>>> [[a for a in FileStreamIterator(fpath, chunk=chunk)] for chunk in [1,2, 3,4]]
[['f', 'o', 'o'], ['fo', 'o'], ['foo'], ['foo']]

>>> os.unlink(fpath)


The Ephemeral Stream Iterator
+++++++++++++++++++++++++++++++++


This object returns a generator to read our big files but delete them when they are read!::

>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fdir = tempfile.mkdtemp(); fdir2 = tempfile.mkdtemp(dir=fdir);fpath = os.path.join(fdir2, 'foo')
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()

Files are there, we can ask to not delete parents (default)::

>>> [os.path.exists(p) for p in fdir,fdir2, fpath]
[True, True, True]
>>> len(EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False))
3
>>> [a for a in EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False)]
['foo']
>>> [os.path.exists(p) for p in fdir,fdir2, fpath]
[True, True, False]

We have read it, the file and the parent are deleted::

>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> [a for a in EphemeralStreamIterator(fpath, delete_parent=True, delete_grand_parent=True)]
['foo']
>>> [os.path.exists(p) for p in fdir2, fdir, fpath]
[False, False, False]



Changelog
===========


1.1.10 (2012-12-21)
-------------------

- Better non exportable types[kiorky]
- "better references restore [kiorky]
* fixed error (invalid literal for int) [potar]
* fixed compatibility with SmartFolder [potar]


1.1.8 (2012-05-28)
------------------
* fix blob files export/import [kiorky]
* handle years < 999 [kiorky]
* let commits be more incremental [kiorky]
* fix export order [kiorky]
* improve retry policy [kiorky]


1.1.7 (2010-08-20)
---------------------
* Adding comments exporter/searcher [kiorky]


1.1.6 (2010-07-30)
----------------------
* Fix importObject (obj referenced before assignment) [danjacka]

1.1.5 (2010-07-29)
-------------------
* Fix unittest for plone4 msgid changes [kiorky]
* Implement import plugins [kiorky]
* Add a meta.zcml with some basic plugins, like for wf state export/import [kiorky]
* Rename objects for consistency purpose [kiorky]
* Fix some Acquisition bugs [kiorky]
* Support blobstorage [kiorky]

1.1.4 (2010-03-05)
-------------------

* fix some encoding issues with file handler [kiorky]
* better plain format import [kiorky]
* Fix setSubject [kiorky]

1.1.3 (2010-01-27)
--------------------

* Fixed missing import of interface.implements in browser.manager [fRiSi]

1.1.2 (2010-01-27)
---------------------


* Fix silly empty list bug in replicator.csvimport prototype. [kiorky]

* fixed missing import of interface.implements in browser.manager [fRiSi]

* Fix unicode error with reference fields [kiorky]

* Add support for temporary export path [kiorky]

* Add support for flattened CSV Files [kiorky]

* Add tests, and tests infrastructure [kiorky]

* Add Stream (Files) Iterators not to overhead the RAM [kiorky]

1.1.1 - Unreleased
-------------------

* remove five:traversable directive (deprecated in plone3) [toutpt]

* Add default config for importcsvStep [toutpt]

* importcsvStep now depend on plone-final step [toutpt]

* option ``ignore_content_errors`` added to import step config (set it to true) and to the replicator methods. It allows to log errors when setting fields instead of raising exceptions and stop. [kiorky]



1.1 (2009-10-17)
-----------------

* A new import step to use csvreplicata to import contents [toutpt]

* Fix creationflag issue [toutpt]

* plugins system. Export other things than AT. take a look at interfaces and adpaters [kiorky]

* some encoding bugs fixed [kiorky]

* datetime format settings in the tool [kiorky]

* images/files created with folder structure in export [kiorky]

* Plone 2.5 compatibility back [kiorky]

1.0.7 (2009-07-15)
--------------------

* Prevent fixTools from running outside of the context of Products.csvreplicata [pigeonflight]

* Fix export when data value is not ascii [yboussard]

1.0.6
-------

* support vocabulary defined as object method [Jim BAACK]

1.0.5
-----

* fix bug crc32 with large media files
* when you set a CSVHandledTypesSchematas default is automatically preselected for user convenience [Jean-Philippe CAMGUILHEM]

1.0.4
-----

* Uninstall problem fix

IMPORTANT NOTE: when migrating from previous versions, you need to uninstall csvreplicata and then reinstall csvreplicata. [Jean-Philippe CAMGUILHEM]

1.0.3
-----

* Custom handlers can be now implemented outside the product, and dynamically declared to the csvreplicata tool. [Jean-Philippe CAMGUILHEM]

1.0.2
-----

* Initial release.[Eric BREHAULT / Christophe SAUTHIER]

1.0
---

* Unreleased