eea.similarity 2.1

A package that suggests similar titles to one being added

Introduction

EEA Similarity is a Plone add-on which provides content suggestions based on similarity scores to a search string. It uses NLP algorithms like TF-IDF (frequency–inverse document frequency) and LSI (Latent Semantic Indexing).

Initial use case is to provide a list of possible duplicates when adding content (based on the entered title).

Contents

• Introduction

• Installation

• Getting started

• Dependencies

  • Possible issues with scipy and numpy:

• Source code

• Copyright and license

• Funding

• Changelog

  • 2.1 - (2018-08-27)

  • 2.0 - (2018-07-16)

  • 1.9 - (2017-12-12)

  • 1.8 - (2017-09-26)

  • 1.7 - (2017-09-15)

  • 1.6 - (2017-08-16)

  • 1.5 - (2017-05-22)

  • 1.4 - (2017-04-24)

  • 1.3 - (2017-04-19)

  • 1.2 - (2017-04-14)

  • 1.1 - (2017-04-13)

  • 1.0 - (2017-04-12)

Installation

zc.buildout

If you are using zc.buildout and the plone.recipe.zope2instance recipe to manage your project, you can do this:

• Update your buildout.cfg file:

  • Add eea.similarity to the list of eggs to install

  • Tell the plone.recipe.zope2instance recipe to install a ZCML slug

  [instance]
  ...
  eggs =
    ...
    eea.similarity
  
  zcml =
    ...
    eea.similarity

• Re-run buildout, e.g. with:

  $ ./bin/buildout

You can skip the ZCML slug if you are going to explicitly include the package from another package’s configure.zcml file.

Getting started

1. Go to Site Setup > Add-ons and install EEA Similarity

2. Create IDF index by calling @@create_idf_index Browser view on site root. If you have plone.app.async installed, it will add a daily job to async queue, otherwise you’ll have to setup an external cron job to call this periodically.

3. Customize settings via Site Setup > EEA Similarity Settings

Dependencies

EEA Similarity has the following dependencies:

• stemming

• gensim (which in turn depends on numpy and scipy)

On CentOS you need to install:

• blas-devel

• lapack-devel

• gcc-fortran

On Debian/Ubuntu you need to install:

• libblas-dev

• liblapack

• gfortran

Possible issues with scipy and numpy:

Due to a bug in scipy, it is possible that the packages will not install when running buildout. There are two solutions/options:

1. Install them with easy_install pip before running the buildout.

2. Update zc.buildout to version 2.9.0 and activate the wheel support via buildout.wheel and the dependencies will be installed normally.

Source code

• EEA on Github

Copyright and license

The Initial Owner of the Original Code is European Environment Agency (EEA). All Rights Reserved.

The EEA Similarity (the Original Code) is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

More details in License.txt

Funding

EEA - European Environment Agency (EU)

Changelog

2.1 - (2018-08-27)

• Change: Get base-url from body data base-url [avoinea refs #97843]

2.0 - (2018-07-16)

• Change: Use logger.warn for expected exceptions [avoinea refs #96835]

• Change: updated URLs pointing to eea.europa.eu with https:// [alecghica refs #95849]

1.9 - (2017-12-12)

• Change: Replace eeacms/zptlint with eeacms/plone-test:4 zptlint [avoinea refs #90415]

1.8 - (2017-09-26)

• bugfix in javascript [valipod refs #88020]

1.7 - (2017-09-15)

• bugfix in javascript [valipod refs #88020]

1.6 - (2017-08-16)

• Upgrade step: Within “Plone > Site setup > Add-ons” click on upgrade button available for EEA Similarity [valipod refs #84915]

• the similarity check is now available on edit pages [valipod refs #85448]

• the similarity threshold can be passed as parameter [valipod refs #85448]

• added possibility to turn on check for all content types [valipod refs #85448]

• button to request the similarity check manually [valipod refs #85448]

• suggest popup appears even if there are now suggestions (different message) [valipod refs #85448]

• integrated the suggest popup also for indicators aggregated edit [valipod refs #85448]

1.5 - (2017-05-22)

• Upgrade step: Within “Plone > Site setup > Add-ons” click on upgrade button available for EEA Similarity [avoinea refs #84915]

• Change: Add dedicated async quota for similarity jobs [avoinea refs #84915]

• Change: fixed PyLint warnings and errors [valipod refs #84949]

1.4 - (2017-04-24)

• Feature: add control panel option to show suggestions only after a certain title length [valipod refs #84480]

• Change: updated package information [eduard-fironda]

1.3 - (2017-04-19)

• Bug fix: fixed issue with the equivalent sets [valipod refs #84294]

• Feature: added metadata info to the suggestion dialog [valipod refs #84294]

1.2 - (2017-04-14)

• Bug fix: Remove hard dependency on plone.app.async and eea.versions [avoinea refs #83163]

• Bug fix: removed hard coded path in the dialog’s javascript [valipod refs #83163]

• Bug fix: added some defaults to the control panel settings [valipod refs #83163]

• Bug fix: removed mandatory status for some control panel settings [valipod refs #83163]

1.1 - (2017-04-13)

• Bug fix: fixed README markup [alecghica refs #82296]

• Upgrade step: trigger index generation once (/www/create_idf_index) Feature: added index update frequency to the registry settings [valipod refs #83163]

1.0 - (2017-04-12)

• Feature: related to index and score:

  • moved the index creation to the async instance

  • add similarity score next to object link in dialog

  • [valipod refs #83163]

• Feature: added to registry settings:

  • equivalent content types sets, use of stopwords, number of shown suggestions

  • max. score difference between suggestions, dialog title and text

  • [valipod refs #83163]

• Pre-upgrade step: run etc/scripts/prereq.sh on each backend [alecghica refs #82296]

• Pre-upgrade step: run deploy.sh [alecghica refs #82296]

• Upgrade step: from control panel instal eea.similarity [valipod refs #82296] [alecghica refs #82296]

• Feature: Initial release [valipod refs #82296]