ftw.publisher.sender 2.14.6

Staging and publishing addon for Plone contents.

Introduction

The ftw.publisher packages provide tools for publishing plone contents from one instance to another.

By providing a simple mechanism to invoke the publishing, it’s possible to set up the publisher in a variety of ways, such as workflow bound, manually invoked or automated publication.

The ftw.publisher package library also provides a variety of surveillance and analysis tools for making maintenance confortable.

Staging

The ftw.publisher is meant to be used in a environment where there are two seperate plone-sites which do not share their database. The editors work on a editorial site and the contents are published to a public site when they are ready. Setting up such an environment with ftw.publisher let you have a powerful staging solution with completly isolated instances.

Network security

Using an environment with two isolated installations makes it possible to protect the editorial site with firewalls or to put it even in a private company network. This way the editorial site is completly protected from the internet, which is in some use cases mandatory for protecting other - unpublished - contents (for example when publishing the internet contents from the intranet).

Component support

• Archetypes objects

• Standard Archetypes field types

• Topics: criterias are published automatically when topic is published

• Backreferences - references are added automatically as soon both objects are published

• Additional interfaces added on /manage_interfaces or by other products

• Contextual portlets

• Properties

With the publisher adapter structure it is as easy as creating another adapter to support other components. For instance annotations are not supported by design, because you may not wan’t to publish all annotations but only certain ones. Therefore it is easyer to implement custom adapters for those annotations which need to be published.

Packages

Core packages

ftw.publisher.sender

The sender package provides a configuration panel and is responsible for sending contents to the target instance. It’s usually installed on a editorial site. ftw.publisher.sender source | ftw.publisher.sender pypi

ftw.publisher.receiver

The receiver package is installed on the public site and is the target of the publishing process. It has tools for receiving a request and creating, updating or deleting objects which should be published or retrieved. ftw.publisher.receiver source | ftw.publisher.receiver pypi

ftw.publisher.core

The core package is installed on both, sender and receiver instances. It provides adapters for serializing and unserializing components of plone contents (such as portlets). ftw.publisher.core source | ftw.publisher.core pypi

Addon packages

ftw.publisher.example

This example package provides a publisher-bound workflow and shows how to write an integration package for this use case. ftw.publisher.example source | ftw.publisher.example pypi

ftw.publisher.monitor

Sends alert mails when the publisher queue is blocked and publishing does no longer work. ftw.publisher.monitor source | ftw.publisher.monitor pypi

ftw.publisher.mailreport

Sends scheduled reports about the publishing state (executed jobs, failed jobs, etc.). ftw.publisher.mailreport source | ftw.publisher.mailreport pypi

ftw.publisher.controlling

Views for comparing the editorial and the public site. Detects inconsistencies and problems by comparing the catalogs of the two sites. Useful in workflow based publishing environments. ftw.publisher.controlling source | ftw.publisher.controlling pypi

Installation & Usage

Take a look at the ftw.pubisher.example package and the example buildout. See also the wiki.

Override realm configuration with ZCML

The realms are by default configured in the database through the control panel. When copying the database from a production environment to a staging environment the realm configuration is copied too, which could result in publishing from the staging editorial site to the production public site, which is very bad.

For solving this issue it is possible to override the realm configuration with ZCML, so that it can be configured also using the zcml-additional option of the buildout.

<configure xmlns:publisher="http://namespaces.zope.org/ftw.publisher">

    <include package="ftw.publisher.sender" file="meta.zcml" />

    <publisher:override-realm
        url="http://localhost:9090/site"
        username="publisher-user"
        password="publisher-password" />

</configure>

Use asynchronous task queue for extraction

When creating a lot of publishing jobs in one single request, the extracting of the content (JSON serializing) may take a lot of time. In order to reduce the feedback time the user has an optional taskqueue based on collective.taskqueue may be configured. By default the extraction is done blocking.

Example buildout configuration:

zope-conf-additional +=

%import collective.taskqueue <taskqueue /> <taskqueue-server />

environment-vars +=

PUBLISHER_TASKQUEUE true

Configure workflows to publish

The ftw.publisher can be used with workflows. For using it with workflows you need to configure your workflow to use publisher actions and you need to provide a configuration for your workflow, telling the publisher what each state and transition means.

Defining a publisher configuration

A publisher configuration is a simple IWorkflowConfiguration adapter, which could look like this:

from ftw.publisher.sender.workflows import interfaces
from zope.component import adapts
from zope.interface import Interface
from zope.interface import implements


class MyWorkflowConfiguration(object):
    implements(interfaces.IWorkflowConfiguration)
    adapts(Interface)

    def __init__(self, request):
        self.request = request

    def states(self):
        return {
            'private': None,
            'pending': None,
            'published': interfaces.PUBLISHED,
            'revision': interfaces.REVISION}

    def transitions(self):
        return {
            'submit': interfaces.SUBMIT,
            'publish': interfaces.PUBLISH,
            'reject': interfaces.RETRACT,
            'retract': interfaces.RETRACT,
            'revise': None}

The named-adapter is then registered with some ZCML, where the name of the adapter is the ID of the workflow in portal_workflow.

<adapter factory=".config.MyWorkflowConfiguration"
         name="my-workflow" />

Lawgiver-Workflows

ftw.lawgiver is a tool for writing workflows. If you are using the lawgiver, you can use LawgiverWorkflowConfiguration as a base class, which allows you to define the states and transitions by name / statement instead of ID:

from ftw.publisher.sender.workflows import config
from ftw.publisher.sender.workflows import interfaces


class ExampleWorkflowConfiguration(config.LawgiverWorkflowConfiguration):
    workflow_id = 'publisher-example-workflow'

    def lawgiver_states(self):
        return {
            'Internal': None,
            'Pending': None,
            'Published': interfaces.PUBLISHED,
            'Revision': interfaces.REVISION}

    def lawgiver_transitions(self):
        return {
            'submit (Internal => Pending)': interfaces.SUBMIT,
            'publish (Internal => Published)': interfaces.PUBLISH,
            'reject (Pending => Internal)': None,
            'publish (Pending => Published)': interfaces.PUBLISH,
            'retract (Published => Internal)': interfaces.RETRACT,
            'revise (Published => Revision)': None,
            'publish (Revision => Published)': interfaces.PUBLISH,
            }

Transition validation (constraints)

When a user publishes a content and its container is not yet published it will fail on the remote system, because the container is missing.

The publisher provides workflow constraints for prohibiting bad transitions and for warning when something should be done (e.g. references should also be published).

You should enable those constraints for your workflow by changing the transition action URL (“Display in actions box” -> “URL (formatted)”) to the format %(content_url)s/publisher-modify-status?transition=TRANSITION (replace TRANSITION) with the transition ID. The default Plone URL is %(content_url)s/content_status_modify?workflow_action=TRANSITION.

The constraints are adapters registered for each workflows. This allows to change the constraints per workflow easily. Take a look at the publisher example workflow constraints.

You might either subclass the example workflow constraint and extend it, write your own constraint definitions from scratch or directly use the example workflow constraints for your workflow.

Reusing the example workflow constraints is as simple as registering a named adapter (your workflow ID in portal_workflow is the name of the adapter):

<adapter factory="ftw.publisher.sender.workflows.example.ExampleWorkflowConstraintDefinition"
         name="my-workflow" />

Testing workflows

For automatically testing whether your worlfow configuration is correct you can reuse the publisher example workflow configuration tests <https://github.com/4teamwork/ftw.publisher.sender/blob/master/ftw/publisher/sender/tests/test_example_workflow_config.py>:

from ftw.publisher.sender.tests import test_example_workflow_config
from my.package.testing import MY_INTEGRATION_TESTING

class TestMyWorkflowConfig(test_example_workflow_config.TestWorkflowConfig):
    layer = MY_INTEGRATION_TESTING
    workflow_id = 'my-workflow'

If you write custom constraints you should also take at the example constraints tests.

Disable creating publisher jobs

from ftw.publisher.sender.nojobs import publisher_jobs_disabled

with publisher_jobs_disabled():
    pass  # no publisher jobs created here.

FormGen / EasyForm integration

The FormGen integration allows it to download the data entered on a published site. If the content is published it will get the data from from the public site. If it isn’t published it will use the local data. The reason for this distinction is that we support internal forms, which are never published at all. In this case the internally collected data should be downloaded.

Further hint for EasyForm: Easyform supports Excel as export format from saved data. In order to support this openpyxl is required. openpyxl will compine the two excel files (local and remote) and return it

Links

The main project package is ftw.publisher.sender since it contains all the configuration panels and the most tools - but without the other mandatory packages it will not work. Here are some additional links:

• Publisher packages on pypi: http://pypi.python.org/pypi?%3Aaction=search&term=ftw.publisher&submit=search

• Github: https://github.com/4teamwork/ftw.publisher.sender

• Issues: https://github.com/4teamwork/ftw.publisher.sender/issues

• Wiki: https://github.com/4teamwork/ftw.publisher.sender/wiki

• Continuous integration: https://jenkins.4teamwork.ch/search?q=ftw.publisher.sender

Copyright

This package is copyright by 4teamwork.

ftw.publisher.sender is licensed under GNU General Public License, version 2.

Changelog

2.14.6 (2022-02-16)

• Fix easyform xlsx download integration. Do not combine xlsx if remote does not exist. [mathias.leimgruber]

2.14.5 (2021-09-30)

• Fix easyform download integration. Since v3 delimiter is part of the download method signature. [mathias.leimgruber]

2.14.4 (2021-07-19)

• Create a move job upon folder_contents rename view (Plone 5 only). [mathias.leimgruber]

2.14.3 (2021-07-19)

• Create a move job upon object_rename view (Plone 5 only). [mathias.leimgruber]

2.14.2 (2021-07-16)

• Support save data adapter which returns excel format. [mathias.leimgruber]

2.14.1 (2020-07-30)

• Fix tests after constrained types adapter added to core. [djowett-ftw]

• Fix IDisableCSRFProtection import error for older versions of plone.protect. [busykoala]

2.14.0 (2020-01-31)

• Add collective.easyform support: download remote data of safe data adapter. [jone]

• Fix references on new page onto itself are matched as unpublished [Nachtalb]

2.13.0 (2019-11-27)

• Drop ftw.contentpage / simplelayout.base support. [jone]

• Compatibility with ftw.publisher.core = 2.13.0. [jone]

2.10.1 (2019-11-11)

• Support CSRF protection and fix publishing on plone 5 with it [Nachtalb]

• Fix realm editing form [Nachtalb]

2.10.0 (2019-10-18)

• Improve ftw.trash compatibility by using contentValues instead of objectValues. [jone]

• Refactor synchronizing simplelayout state so that it works for non-content-pages. [jone]

• Introduce new “BeforePublishEvent”. [jone]

• Replace simplelayout publishing recursion with a generic mechanism in order to support PloneFormGen. [jone]

• Replace use of “is_sl_contentish” function with “belongs_to_parent”. This requires an update of ftw.publisher.core to the compatible version. [jone]

2.9.0 (2019-08-29)

• Concatenate csv entries from self and realm in monkeypatched Products.PloneFormGen download method. [busykoala]

2.8.2 (2018-10-02)

• Do not warn unpublished references which will now be published. [jone]

2.8.1 (2018-09-17)

• Extraction: increase attempts to 10 and pause 0.5s in between. [jone]

2.8.0 (2018-04-25)

• Improve extraction worker. [mbaechtold]

• Add support for Plone 5.1. [mbaechtold]

2.7.13 (2018-02-23)

• Define a two minute timeout when sending the data to the receiver. [mbaechtold]

2.7.12 (2018-02-06)

• Execute move jobs synchronously, instead of asynchronously. [mbaechtold]

2.7.11 (2017-10-03)

• Fix changing the ID of an object (bug only occurred in combination with Redis). [mbaechtold]

• Make renaming objects on the Plone Site work again. [mbaechtold]

2.7.10 (2017-07-27)

• Make the WorkflowConfigTest base class importable again without installing the tests extras. [jone]

2.7.9 (2017-07-25)

• Fix retrying extraction of move jobs. [mathias.leimgruber]

• Rewrite tests using “ftw.testbrowser”, drop dependency on “ftw.testing[splinter]”. [mbaechtold]

2.7.8 (2017-06-12)

• Fix bug when retrying extraction, introduced in 2.7.7. [jone]

2.7.7 (2017-06-09)

• Fix extraction bug on move when redis is active. [jone]

2.7.6 (2017-06-08)

• Fix extraction token access error on first publishing. [jone]

2.7.5 (2017-05-16)

• Get the FormGen data from public site if Form Folder is published. [tschanzt]

2.7.4 (2017-04-27)

• No longer move simplelayout blocks instantly. [jone]

2.7.3 (2017-03-20)

• Fix a bug where an old version was published. [jone]

• Close connection after publishing. [jone]

2.7.2 (2016-09-06)

• Make moving to worker queue more robust. [jone]

2.7.1 (2016-08-30)

• Fix edge-case, if the published object is removed before the async worker collects the data from the object. [mathias.leimgruber]

2.7.0 (2016-08-16)

• Add IPreventPublishing marker interface. [jone]

• Fix removeJob method, which always tried to remove the job from the worker queue. [mathias.leimgruber]

2.6.0 (2016-07-29)

• Add second worker queue, which is used by the worker to perform the actual job. Also remove the batch feature, was disabled for ages. [mathias.leimgruber]

• Do not add delete jobs to the task queue. It’s quite certain that object will no longer exists if the extractor tries to get some data of the delete obj. [mathias.leimgruber]

2.5.0 (2016-07-27)

• Deferred extraction using collective.taskqueue. [jone]

• Migrate queue storage to zc.queue. [jone]

• Speedup enqueuing by logging less. [jone]

• Support dexterity relations. [mbaechtold]

• Add context manager for disabling job creation. [jone]

2.4.0 (2016-05-20)

• Workflow context state: skip parents without workflow when looking for parent state. This adds support for nested containers without workflows. [jone]

2.3.1 (2015-12-23)

• Use base class WorkflowConfigTest for TestWorkflowConfig test. If we don’t, we’ll get builder registrations conflict errors after importing the test class into another project. There are also problems while imoprting the test class directly on a project without ftw.simplelayout but ftw.contentpage. [elioschmutz]

• Blocks of ftw.contentpage will no longer published directly after deleting. Backport for ftw.contentpage of functionality provided for ftw.simplelayout: https://github.com/4teamwork/ftw.publisher.sender/pull/20 [elioschmutz]

2.3.0 (2015-09-30)

• Drop Plone 4.2 support. [jone]

• Add ftw.simplelayout support. [jone]

2.2.2 (2015-03-19)

• Sort data collectors so that we have an expected order. This allows to affect the order with data collector names. [jone]

2.2.1 (2014-12-31)

• Get unpublished and published references in a save way - the may be removed. [mathias.leimgruber]

2.2.0 (2013-09-02)

• Add simplelayout support: publish blocks within simplelayout containers automatically when they have no workflow. [jone]

• Provide workflow integration with an example publisher workflow. [jone]

• Use publisher permission for protecting publisher control panel action. Required permission: “ftw.publisher.sender: Change Publisher Configuration” [jone]

2.1.0 (2013-06-13)

• Add separate permission for accessing the publisher settings in the control panel. By default, the permission is granted to Manager and Site Administrator. The permission is “ftw.publisher.sender: Change Publisher Configuration” [jone]

2.0.1 (2013-05-24)

• Execute queue view: add batch size argument. [jone]

2.0 (2013-05-24)

• Make status messages transaction aware, so that it works with integrity check rollbacks. [jone]

• Implement an override-realm ZCML directive for overriding database settings with ZCML. [jone]

• Make ugprade step 1.1 -> 1.2 save: do not migrate when there is no data. [jone]

• Plone 4.3 compatibility. [jone]

• Add “disable-publisher-for-testing” environment variable for disabling event handler. [jone]

• Fixed error if data-adapter (JSON-data) doesnt contain field_data_adapter. [jinfanger]

• UnicodeError in log. LogStream should not be mixed utf8 and unicode. [jinfanger]

• Ignored fields can be configurated, which will be ignored by the publisher. [jinfanger]

• Drop Plone 3 support. [jone]

1.5 (2011-07-05)

• Fixed authentication problem (base64) (issue #1) [jone]

• Fix PloneFormGen publish support. Recursively publish all items from a FormFolder. [mathias.leimgruber]

1.4 (2011-04-06)

• Cleanup, move to github (https://github.com/4teamwork/ftw.publisher.sender) and prepare for release. [jone]

1.3 (2011-01-19)

• Added some german translations and added plain english po-file [mathias.leimgruber]

• Merged lock-config-branch into trunk Bei Direktem Publishing ohne Queu macht das Locking keinen Sinn. Daher kann man nun das Locking einschalten/ausschalten. [mathias.leimgruber]

• Added new IBeforeQueueExecutionEvent, which allows third party packages to do dome stuff. [jone]

1.2 (2010-09-27)

• Added missing message factory import in publishploneformgen.py [jone]

• Added QueueExecutedEvent [jone]

• Added viewlet manager on configlet page. [jone]

• Do not catch BadStatusLine exceptions since the should be moved to the executed jobs list like when regular exceptions occur. [jone]

• Added batching to executed jobs view [jone]

• Added upgrade step for migrating the executed jobs storage to IOBTree [jone]

• Added some storage tests and changed the executed jobs storage to a IOBTree storage for better performance. [jone]

UPGRADES:

• portal_setup: run the ftw.publisher.sender upgrade step for migrating storage

1.1 (2010-07-21)

• Use the translated communication state names [jone]

• Show json cache path in configuration [jone]

• Executed jobs list: made more robust ; reversed listing [jone]

1.0c4 (2010-06-24)

• Fixed bad adapter usage [jone]

1.0c3 (2010-06-24)

• Added string encoding / decoding methods which work with json [jone]

1.0c2 (2010-06-22)

• Fixed fixed namespace_packages in setup.py [jone]

1.0c1 (2010-06-18)

• Locking while executing queue [jone]

• Added possiblity to disable automatic publishing in control panel [jone]

• Added possibility to execute / delete a single job in the queue [jone]

• Added executed jobs list [jone]

• Added publisher control panel icon [jone]

• Implemented blacklist support [jone]

• Added modification date to medata: it will be set by the receiver [jone]

• Fixed encoding problem: the dumped data should always contain unicode strings [jone]

• Client-Objects are now IAttributeAnnotable [jone]

• Using z3c.autoinclude for including zcml-dependencies [jone]

• implement rename action [mle]