Skip to main content

CSV import/export for Archetypes or other contents (via plugins for the later). Created by Makina Corpus.

Project description

Introduction
============

Allows to export and import AT objects contents and hierarchy.

It supports also other contents or objects via plugins (ZCA).
This part mainly targeted for a developper audiance, so see interfaces.py / adapters.py.


We have plans to integrate the export/import code with transmogrifier_, [ also_ ] but we have no ETA For now.

.. _transmogrifier: http://pypi.python.org/pypi/collective.transmogrifier/
.. _also: http://pypi.python.org/pypi/plone.app.transmogrifier/


Repository: git_


.. _git: https://github.com/collective/Products.csvreplicata.git

A Makina Corpus add-on
-------------------------
|makinacom|_

* `Planet Makina Corpus <http://www.makina-corpus.org>`_
* `Contact us <mailto:python@makina-corpus.org>`_

.. |makinacom| image:: http://depot.makina-corpus.org/public/logo.gif
.. _makinacom: http://www.makina-corpus.com


Products.csvreplicata Installation
======================================

To install Products.csvreplicata into the global Python environment (or a workingenv),
using a traditional Zope 2 instance, you can do this:

* When you're reading this you have probably already run
``easy_install Products.csvreplicata``. Find out how to install setuptools
(and EasyInstall) here:
http://peak.telecommunity.com/DevCenter/EasyInstall

* If you are using Zope 2.9 (not 2.10), get `pythonproducts`_ and install it
via::

python setup.py install --home /path/to/instance

into your Zope instance.

* Create a file called ``Products.csvreplicata-configure.zcml`` in the
``/path/to/instance/etc/package-includes`` directory. The file
should only contain this::

<include package="Products.csvreplicata" />

.. _pythonproducts: http://plone.org/products/pythonproducts


Alternatively, if you are using zc.buildout and the plone.recipe.zope2instance
recipe to manage your project, you can do this:

* Add ``Products.csvreplicata`` to the list of eggs to install, e.g.

::

[buildout]
...
eggs =
...
Products.csvreplicata

* Tell the plone.recipe.zope2instance recipe to install a ZCML slug

::

[instance]
recipe = plone.recipe.zope2instance
...
zcml =
Products.csvreplicata

* Re-run buildout, e.g. with:

$ ./bin/buildout

You can skip the ZCML slug if you are going to explicitly include the package
from another package's configure.zcml file.


Because its top level Python namespace package is called ``Products``, this
package can also be installed in Zope 2 as an old style **Zope 2 Product**.

For that, move (or symlink) the ``csvreplicata`` folder of this project
(``Products.csvreplicata/Products/csvreplicata``) into the ``Products`` directory of
the Zope instance it has to be installed for, and restart the server.

You can also skip the ZCML slug if you install this package the **Zope 2
Product** way.

Detailled documentation
=======================


Interfaces
-------------
import interfaces and classes ::

>>> from zope.interface.verify import verifyClass
>>> from zope.interface import implements
>>> from Products.csvreplicata.handlers.base import CSVdefault
>>> from Products.csvreplicata.handlers.file import CSVFile
>>> from Products.csvreplicata.interfaces import ICSVDefault, ICSVFile

Verify implementation ::

>>> verifyClass(ICSVDefault, CSVdefault)
True
>>> verifyClass(ICSVFile, CSVFile)
True



Export / Import in plain format
------------------------------------

Export
+++++++++
here we export folders and documents
::

>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document' , id='doc1' , title="Document 1")
>>> id=self.folder.invokeFactory('Document' , id='doc2' , title="Document 2")
>>> id=self.folder.invokeFactory('News Item' , id='news1' , title="news 'super' 3")
>>> id=self.folder.invokeFactory('Document' , id='doc4' , title="Document 4")
>>> id=self.folder.invokeFactory('Folder' , id='sub1' , title="mytest")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc11' , title="Document 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('News Item', id='news21' , title="news 1 du dossier 1")
>>> id=self.folder.sub1.invokeFactory('Document' , id='doc31' , title="Document 2 du dossier 1")
>>> self.portal.portal_csvreplicatatool.getPlainFormat()
False

>>> self.portal.portal_csvreplicatatool.getEncoding()
'UTF-8'
>>> self.portal.portal_csvreplicatatool.getDelimiter()
';'
>>> self.portal.portal_csvreplicatatool.getStringdelimiter()
'"'
>>> self.portal.portal_csvreplicatatool.replicabletypes = {'Document': ['default'], 'Folder': ['default'], 'News': ['Default']}
>>> from Products.csvreplicata.interfaces import Icsvreplicata
>>> repl = Icsvreplicata(self.folder)
>>> print repl.csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue()
"/plone/Members/test_user_1_";"..."
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc1";"Document";"Document 1";"";"";"simple_publication_workflow";"private";"";""
"";"doc2";"Document";"Document 2";"";"";"simple_publication_workflow";"private";"";""
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"news1";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"doc4";"Document";"Document 4";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_description";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"";"sub1";"Folder";"mytest";"";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"";"simple_publication_workflow";"private"
"parent";"id";"type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"news21";"News Item";"simple_publication_workflow";"private"
"parent";"id";"type";"title";"description";"text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"Parent folder";"Identifier";"Content type";"Title";"label_summary";"label_body_text";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain";"ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state"
"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"";"simple_publication_workflow";"private"
<BLANKLINE>

This output is not that good to deal with CSV apis, we will try to export it to a flat structure.
::

>>> from Products.csvreplicata import replicator
>>> self.portal.portal_csvreplicatatool.setPlainFormat(True)
>>> self.portal.portal_csvreplicatatool.getPlainFormat()
True
>>> repl = Icsvreplicata(self.folder)
>>> print Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=0).getvalue() # doctest: +ELLIPSIS
"startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";...
"Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";...
"/plone/Members/test_user_1_";"...";"";"doc1";"Document";"Document 1";...
"/plone/Members/test_user_1_";"...";"";"doc2";"Document";"Document 2";...
"/plone/Members/test_user_1_";"...";"";"news1";"News Item";...
"/plone/Members/test_user_1_";"...";"";"doc4";"Document";"Document 4";...
"/plone/Members/test_user_1_";"...";"";"sub1";"Folder";"mytest";...
"/plone/Members/test_user_1_";"...";"sub1";"doc11";"Document";"Document 1 du dossier 1";...
"/plone/Members/test_user_1_";"...";"sub1";"news21";"News Item";...
"/plone/Members/test_user_1_";"...";"sub1";"doc31";"Document";"Document 2 du dossier 1";...
<BLANKLINE>

Redo the export but with a plugin that find the title on the object.
::

>>> from Products.csvreplicata import adapters
>>> class CustomExporter(adapters.CSVReplicataExportImportPluginAbstract):
... def __init__(self, *args, **kwargs):
... adapters.CSVReplicataExportImportPluginAbstract.__init__(self, *args, **kwargs)
... self.ids.append('title')
... def fill_values(self, row, row_ids):
... """."""
... for id in row_ids:
... if id.replace(self.prefix, '') in self.ids:
... index = row_ids.index(id)
... if index < len(row):
... row[index] = self.context.Title()
... def set_values(self, row, row_ids):
... """."""
... print "plugin.setValue called with %s <-> %s" % (row, row_ids)
...
>>> provideAdapter(CustomExporter, (interfaces.Icsvreplicata, zope.interface.Interface), interfaces.ICSVReplicataExportImportPlugin, name ='fooplugin' )
>>> from csv import DictReader
>>> content = Icsvreplicata(self.folder).csvexport(exportable_content_types=['Document', 'Folder', 'News Item'], depth=None).getvalue()
>>> items = [item for item in DictReader(StringIO(content), delimiter=";", quotechar='"')]
>>> keys = items[0].keys();keys.sort()

As we cant predict order of the keys, doing some magic to order them before testing.
::

>>> pprint([[(key, item[key]) for key in keys if not 'date' in key]for item in items], width=130) # doctest: +REPORT_NDIFF
[[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain',
'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state',
'ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title',
'ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title'),
('description', 'label_...'),
('id', 'Identifier'),
('parent', 'Parent folder'),
('startpoint', 'Start point'),
('text', 'label_body_text'),
('title', 'Title'),
('type', 'Content type')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1'),
('description', ''),
('id', 'doc1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 1'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2'),
('description', ''),
('id', 'doc2'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 2'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', "news 'super' 3"),
('description', ''),
('id', 'news1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', ''),
('type', 'News Item')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 4'),
('description', ''),
('id', 'doc4'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 4'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'mytest'),
('description', ''),
('id', 'sub1'),
('parent', ''),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'mytest'),
('type', 'Folder')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 1 du dossier 1'),
('description', ''),
('id', 'doc11'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 1 du dossier 1'),
('type', 'Document')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'news 1 du dossier 1'),
('description', ''),
('id', 'news21'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', ''),
('type', 'News Item')],
[('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_chain', 'simple_publication_workflow'),
('ReplicataPlugin_Products_csvreplicata_exportimport_plugins_WorkflowExportImporter_wf_state', 'private'),
('ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title', 'Document 2 du dossier 1'),
('description', ''),
('id', 'doc31'),
('parent', 'sub1'),
('startpoint', '/plone/Members/test_user_1_'),
('text', ''),
('title', 'Document 2 du dossier 1'),
('type', 'Document')]]




Import
+++++++++

Then, now that we got a working export, what about importing it
::

>>> id = self.folder.invokeFactory('Folder' , id='fa' , title="tests import")
>>> id = self.folder.invokeFactory('Folder' , id='fb' , title="tests import")
>>> fa = self.folder.fa; fb = self.folder.fb

Import in CSVReplicata plain format, an entry per line without contextual type hinting
::

>>> CSV = StringIO("""\
... "startpoint";"replicata_export_date";"parent";"id";"type";"title";"description";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"text"
... "Start point";"Export Date";"Parent folder";"Identifier";"Content type";"Title";"label_...";"ReplicataPlugin_Products_csvreplicata_tests_test_doctests_CustomExporter_title";"label_body_text"
... "/plone/Members/test_user_1_";20090101010101;"";"doc1";"Document";"Document 1";"";"Document 1";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc2";"Document";"Document 2";"";"Document 2";""
... "/plone/Members/test_user_1_";20090101010101;"";"news1";"News Item";"";"";"news 'super' 3";""
... "/plone/Members/test_user_1_";20090101010101;"";"doc4";"Document";"Document 4";"";"Document 4";""
... "/plone/Members/test_user_1_";20090101010101;"";"sub1";"Folder";"mytest";"";"mytest";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc11";"Document";"Document 1 du dossier 1";"";"Document 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"news21";"News Item";"";"";"news 1 du dossier 1";""
... "/plone/Members/test_user_1_";20090101010101;"sub1";"doc31";"Document";"Document 2 du dossier 1";"";"Document 2 du dossier 1";""
... """)
>>> print Icsvreplicata(self.folder.fa).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=True)
p...
(8, 0, ..., [])
>>> self.folder.fa.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fa.sub1.objectIds()
['doc11', 'news21', 'doc31']

Import in CSVReplicata orignal format, an entry per line with contextual type hinting
::

>>> CSV = StringIO("""\
... "/plone/Members/test_user_1_";20100101010101
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc1";"Document";"Document 1";"";""
... "";"doc2";"Document";"Document 2";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "";"news1";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "";"doc4";"Document";"Document 4";"";""
... "parent";"id";"type";"title";"description"
... "Parent folder";"Identifier";"Content type";"Title";"label_..."
... "";"sub1";"Folder";"mytest";""
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc11";"Document";"Document 1 du dossier 1";"";""
... "parent";"id";"type"
... "Parent folder";"Identifier";"Content type"
... "sub1";"news21";"News Item"
... "parent";"id";"type";"title";"description";"text"
... "Parent folder";"Identifier";"Content type";"Title";"label_...";"label_body_text"
... "sub1";"doc31";"Document";"Document 2 du dossier 1";"";""
... """)
>>> print Icsvreplicata(self.folder.fb).csvimport(CSV, datetimeformat='%d%m%Y', delimiter=";", stringdelimiter='"', plain_format=False)
p...
(8, 0, ..., [])
>>> self.folder.fb.objectIds()
['doc1', 'doc2', 'news1', 'doc4', 'sub1']
>>> self.folder.fb.sub1.objectIds()
['doc11', 'news21', 'doc31']



Default handlers
----------------

Verify the default handlers provided for Archetypes fields::

>>> handlersDict = self.portal.portal_csvreplicatatool.getHandlers()
>>> handlers = handlersDict.keys();handlers.sort()
>>> pprint(handlers)
['Products.Archetypes.Field.BooleanField',
'Products.Archetypes.Field.DateTimeField',
'Products.Archetypes.Field.FileField',
'Products.Archetypes.Field.FloatField',
'Products.Archetypes.Field.ImageField',
'Products.Archetypes.Field.IntegerField',
'Products.Archetypes.Field.LinesField',
'Products.Archetypes.Field.ReferenceField',
'Products.Archetypes.Field.StringField',
'Products.Archetypes.Field.TextField',
'Products.AttachmentField.AttachmentField.AttachmentField',
'default_handler',
...]
>>> pprint([(h, handlersDict[h]['handler_class']) for h in handlers], width=130)
[('Products.Archetypes.Field.BooleanField', <Products.csvreplicata.handlers.base.CSVBoolean object at ...>),
('Products.Archetypes.Field.DateTimeField', <Products.csvreplicata.handlers.base.CSVDateTime object at ...>),
('Products.Archetypes.Field.FileField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('Products.Archetypes.Field.FloatField', <Products.csvreplicata.handlers.base.CSVFloat object at ...>),
('Products.Archetypes.Field.ImageField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('Products.Archetypes.Field.IntegerField', <Products.csvreplicata.handlers.base.CSVInteger object at ...>),
('Products.Archetypes.Field.LinesField', <Products.csvreplicata.handlers.base.CSVLines object at ...>),
('Products.Archetypes.Field.ReferenceField', <Products.csvreplicata.handlers.reference.CSVReference object at ...>),
('Products.Archetypes.Field.StringField', <Products.csvreplicata.handlers.base.CSVString object at ...>),
('Products.Archetypes.Field.TextField', <Products.csvreplicata.handlers.base.CSVText object at ...>),
('Products.AttachmentField.AttachmentField.AttachmentField', <Products.csvreplicata.handlers.file.CSVFile object at ...>),
('default_handler', <Products.csvreplicata.handlers.base.CSVdefault object at ...>),
(...


What happened with csvreplicata during import/export if MyField is not in tool's
handlers. replicator.py apllies default_handler on it::

{'default_handler':
{'handler_class': base.CSVdefault(),'file': False}}





The replicator exporter/downloader
------------------------------------

Export in normal mode
+++++++++++++++++++++++++
here we export folders and documents ::

>>> from Products.csvreplicata.browser import manager
>>> self.portal.portal_csvreplicatatool.replicabletypes = \
... {'Document':['default'], 'Folder':['default'],
... 'News Item': ['default'], 'File': ['default'] }
>>> import re
>>> self.setRoles(['Manager'])
>>> id=self.folder.invokeFactory('Document' , id='doc1' , title="Document 1")
>>> id=self.folder.invokeFactory('Document' , id='doc2' , title="Document 2")
>>> params = {"datetimeformat": '%d/%m/%Y %H:%M:%S',
... "vocabularyvalue": "No",
... "encoding": "UTF-8",
... "delimiter": ";",
... "stringdelimiter": '"',
... "exportable_content_types": ["News Items", "Document", "Folder"],
... }
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from Products.csvreplicata import replicator
>>> print ''.join([a for a in repl.doExport()])
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"text";...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";"";...
"";"doc2";"Document";"Document 2";"";...
<BLANKLINE>

>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.csv']),
('content-length', ['...']),
('content-type', ['text/csv;charset=UTF-8'])]


Export as zip when there are files out there and we want them
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Now adding and exporting a file::

>>> params['exportfiles'] = 'Yes'
>>> params["exportable_content_types"].append('File')
>>> id=self.folder.invokeFactory('File', id='f1' , title="File 1")
>>> req = make_request('/'.join(self.folder.getPhysicalPath())+'@@csvreplicata', **params)
>>> f1 = self.folder.f1
>>> f1.getFile().data = 'foo'
>>> f1.setFilename('bar')
>>> repl = manager.ReplicationManager(self.folder, req)
>>> from StringIO import StringIO
>>> import zipfile
>>> content = StringIO([a for a in repl.doExport()][0])
>>> zip = zipfile.ZipFile(content)
>>> print zip.read('export.csv')
"/plone/Members/test_user_1_";...
"parent";"id";"type";"title";"description";"...
"Parent folder";"Identifier";"Content type";"Title";"label_summary";...
"";"doc1";"Document";"Document 1";...
"";"doc2";"Document";"Document 2";...
"parent";"id";"type";"title";"description";...
"Parent folder";"Identifier";"Content ...
"";"f1";"File";"File 1";"";...
<BLANKLINE>


>>> zip.printdir()
File Name...
bar...
export.csv...
>>> items = list(req.response._headers.iteritems());items.sort();pprint(items)
[('content-disposition', ['attachment; filename=export.zip']),
('content-length', ['...']),
('content-type', ['application/zip'])]


The File Stream Iterator
+++++++++++++++++++++++++

This object returns a generator to read our big files!::

>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fpath = tempfile.mkstemp('foo')[1]
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> len(FileStreamIterator(fpath))
3

We can play with chunks to divide rendering into small parts ::

>>> [[a for a in FileStreamIterator(fpath, chunk=chunk)] for chunk in [1,2, 3,4]]
[['f', 'o', 'o'], ['fo', 'o'], ['foo'], ['foo']]

>>> os.unlink(fpath)


The Ephemeral Stream Iterator
+++++++++++++++++++++++++++++++++


This object returns a generator to read our big files but delete them when they are read!::

>>> from Products.csvreplicata.browser.manager import FileStreamIterator, EphemeralStreamIterator
>>> import tempfile
>>> fdir = tempfile.mkdtemp(); fdir2 = tempfile.mkdtemp(dir=fdir);fpath = os.path.join(fdir2, 'foo')
>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()

Files are there, we can ask to not delete parents (default)::

>>> [os.path.exists(p) for p in fdir,fdir2, fpath]
[True, True, True]
>>> len(EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False))
3
>>> [a for a in EphemeralStreamIterator(fpath, delete_parent=False, delete_grand_parent=False)]
['foo']
>>> [os.path.exists(p) for p in fdir,fdir2, fpath]
[True, True, False]

We have read it, the file and the parent are deleted::

>>> fobj = open(fpath, 'w');fobj.write('foo');fobj.close()
>>> [a for a in EphemeralStreamIterator(fpath, delete_parent=True, delete_grand_parent=True)]
['foo']
>>> [os.path.exists(p) for p in fdir2, fdir, fpath]
[False, False, False]



Changelog
===========


1.1.8 (2012-05-28)
------------------
* fix blob files export/import [kiorky]
* handle years < 999 [kiorky]
* let commits be more incremental [kiorky]
* fix export order [kiorky]
* improve retry policy [kiorky]


1.1.7 (2010-08-20)
---------------------
* Adding comments exporter/searcher [kiorky]


1.1.6 (2010-07-30)
----------------------
* Fix importObject (obj referenced before assignment) [danjacka]

1.1.5 (2010-07-29)
-------------------
* Fix unittest for plone4 msgid changes [kiorky]
* Implement import plugins [kiorky]
* Add a meta.zcml with some basic plugins, like for wf state export/import [kiorky]
* Rename objects for consistency purpose [kiorky]
* Fix some Acquisition bugs [kiorky]
* Support blobstorage [kiorky]

1.1.4 (2010-03-05)
-------------------

* fix some encoding issues with file handler [kiorky]
* better plain format import [kiorky]
* Fix setSubject [kiorky]

1.1.3 (2010-01-27)
------------------

* Fixed missing import of interface.implements in browser.manager [fRiSi]

1.1.2 (2010-01-27)
---------------------


* Fix silly empty list bug in replicator.csvimport prototype. [kiorky]

* fixed missing import of interface.implements in browser.manager [fRiSi]

* Fix unicode error with reference fields [kiorky]

* Add support for temporary export path [kiorky]

* Add support for flattened CSV Files [kiorky]

* Add tests, and tests infrastructure [kiorky]

* Add Stream (Files) Iterators not to overhead the RAM [kiorky]

1.1.1 - Unreleased
-------------------

* remove five:traversable directive (deprecated in plone3) [toutpt]

* Add default config for importcsvStep [toutpt]

* importcsvStep now depend on plone-final step [toutpt]

* option ``ignore_content_errors`` added to import step config (set it to true) and to the replicator methods. It allows to log errors when setting fields instead of raising exceptions and stop. [kiorky]



1.1 (2009-10-17)
-----------------

* A new import step to use csvreplicata to import contents [toutpt]

* Fix creationflag issue [toutpt]

* plugins system. Export other things than AT. take a look at interfaces and adpaters [kiorky]

* some encoding bugs fixed [kiorky]

* datetime format settings in the tool [kiorky]

* images/files created with folder structure in export [kiorky]

* Plone 2.5 compatibility back [kiorky]

1.0.7 (2009-07-15)
--------------------

* Prevent fixTools from running outside of the context of Products.csvreplicata [pigeonflight]

* Fix export when data value is not ascii [yboussard]

1.0.6
-------

* support vocabulary defined as object method [Jim BAACK]

1.0.5
-----

* fix bug crc32 with large media files
* when you set a CSVHandledTypesSchematas default is automatically preselected for user convenience [Jean-Philippe CAMGUILHEM]

1.0.4
-----

* Uninstall problem fix

IMPORTANT NOTE: when migrating from previous versions, you need to uninstall csvreplicata and then reinstall csvreplicata. [Jean-Philippe CAMGUILHEM]

1.0.3
-----

* Custom handlers can be now implemented outside the product, and dynamically declared to the csvreplicata tool. [Jean-Philippe CAMGUILHEM]

1.0.2
-----

* Initial release.[Eric BREHAULT / Christophe SAUTHIER]

1.0
---

* Unreleased

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Products.csvreplicata-1.1.8.zip (90.9 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page