ftw.solr 1.1.2

Solr integration for Plone using collective.solr

Introduction

ftw.solr provides various customizations and enhancements on top of collective.solr which integrates the Solr search engine with Plone.

Features

Highlighting (aka Snippets)

When displaying search results, Plone by default displays the title and the description of an item. Solr, like Google and other search engines, can return a snippet of the text containing the words searched for. ftw.solr enables this feature in Plone.

Live search grouping

Search results in Plone’s live search can be grouped by portal_type. This is the way search results are shown in Spotlight on Mac OS X.

Facet queries

In addition to facet fields support provided by collective.solr, ftw.solr adds support for facet queries. This type of faceting offers a lot of flexibility. Instead of choosing a specific field to facet its values, multiple Solr queries can be specified, that themselve become facets.

Word Cloud

Assuming there is a correctly configured index ‘wordCloudTerms’, a Word Cloud showing the most common terms across documents can be displayed.

The Word Cloud is implemented in a browser view that can either be displayed stand-alone by traversing to /@@wordcloud or rendered in a portlet.

Ajax-ified search form

The search form is fully ajax-ified which leads to faster search results when changing search criteria.

Solr connection configuration in ZCML

The connections settings for Solr can be configured in ZCML and thus in buildout. This makes it easier when copying databases between multiple Zope instances with different Solr servers. Example:

zcml-additional =
    <configure xmlns:solr="http://namespaces.plone.org/solr">
        <solr:connection host="localhost" port="8983" base="/solr"/>
   </configure>

Solr Configuration

Search Handlers

ftw.solr requires two custom search handlers that must be configured on the Solr server.

The livesearch request handler is used for live search and should limit the returned fields to a minimum for maximum speed. Example:

<requestHandler name="livesearch" class="solr.SearchHandler">
    <lst name="defaults">
        <str name="echoParams">explicit</str>
        <int name="rows">1000</int>
    </lst>
    <lst name="invariants">
        <str name="fl">Title Description portal_type path_string getIcon</str>
    </lst>
</requestHandler>

The hlsearch request handler should contain the configuration for higlighting. Example:

<requestHandler name="hlsearch" class="solr.SearchHandler">
    <lst name="defaults">
        <str name="echoParams">explicit</str>
        <int name="rows">10</int>
        <bool name="hl">true</bool>
        <bool name="hl.useFastVectorHighlighter">true</bool>
        <str name="hl.fl">snippetText</str>
        <int name="hl.fragsize">200</int>
        <str name="hl.alternateField">Description</str>
        <int name="hl.maxAlternateFieldLength">200</int>
        <int name="hl.snippets">3</int>
    </lst>
</requestHandler>

Field types and indexes

Highlighting

Highlighting requires an index named snippetText with it’s own field type which does not too much text analysis. Example:

<fieldType name="text_snippets" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
  <analyzer type="query">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.LowerCaseFilterFactory"/>
  </analyzer>
</fieldType>

<field name="snippetText" type="text_snippets" indexed="true"
       stored="true" required="false" multiValued="false"
       termVectors="true" termPositions="true"
       termOffsets="true"/>

Word Cloud

The Word Cloud feature requires an index named wordCloudTerms with it’s own field type. It’s basically a copy of SearchableText but with less analysis and filtering (no lowercasing, no character normalization, etc…).

Field type example:

<fieldType name="cloud_terms" class="solr.TextField" positionIncrementGap="100">
  <analyzer type="index">
      <tokenizer class="solr.WhitespaceTokenizerFactory"/>
      <filter class="solr.StopFilterFactory" ignoreCase="true" words="${buildout:directory}/german_stop.txt" enablePositionIncrements="true"/>
      <filter class="solr.WordDelimiterFilterFactory"
              splitOnCaseChange="1"
              splitOnNumerics="1"
              stemEnglishPossessive="1"
              generateWordParts="0"
              generateNumberParts="0"
              catenateWords="0"
              catenateNumbers="0"
              catenateAll="0"
              preserveOriginal="1"/>
      <!-- Strip punctuation characters from beginning and end of terms -->
      <filter class="solr.PatternReplaceFilterFactory" pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$" replacement="$2"/>
      <!-- Filter everything that does not contain at least 3 regular letters -->
      <filter class="solr.PatternReplaceFilterFactory" pattern="^([^a-zA-Z]*)([a-zA-Z]{0,2})([^a-zA-Z]*)$" replacement=""/>
      <!-- Filter any term shorter than 3 characters (incl. empty string) -->
      <filter class="solr.LengthFilterFactory" min="2" max="50"/>
  </analyzer>
</fieldType>

Index example:

<field name="wordCloudTerms" type="cloud_terms" indexed="true"
       stored="true" required="false" multiValued="false"
       termVectors="true" termPositions="true"
       termOffsets="true"/>

<copyField source="SearchableText" dest="wordCloudTerms"/>

Installation

Install ftw.solr by adding it to the list of eggs in your buildout or by adding it as a dependency of your policy package. Then run buildout and restart your instance.

Go to Site Setup of your Plone site and activate the ftw.solr add-on. Check the Solr control panel provided by collective.solr for Solr-specific configuration options.

Links

• Main github project repository: https://github.com/4teamwork/ftw.solr

• Issue tracker: https://github.com/4teamwork/ftw.solr/issues

• Package on pypi: http://pypi.python.org/pypi/ftw.solr

• Continuous integration: https://jenkins.4teamwork.ch/search?q=ftw.solr

Copyright

This package is copyright by 4teamwork.

ftw.solr is licensed under GNU General Public License, version 2.

Changelog

1.1.2 (2013-07-18)

• Sort facet fields in the order specified in the Solr control panel. [buchi]

• Fixed handling of path filter which was always removed when respect_navroot is set to False. [buchi]

• Handle invalid facet parameters. [buchi]

• Monkey patch reindexObjectSecurity for both CatalogAware and CatalogMultiplex so the relevant security indexes in solr also get updated. [lgraf]

• Only add the default search argument to the query if it’s not None and if Solr has a default search field defined in it’s schema (which is deprecated in Solr). This mainly prevents logging of ‘dropping unknown search attribute’ warnings. [buchi]

• Escape forward slashes in all query values, not only in paths. [buchi]

• Always insert the default ‘select’ search handler into the query parameters if no ‘qt’ parameter is provided. We need this because we have to disable the /select search handler in the Solr configuration to be able to select other search handlers by parameter. [buchi]

1.1.1 (2013-06-01)

• Also use livesearch request handler in livesearch when grouping is disabled. [buchi]

• Fixed “show more” link in live search to point to @@search view. [buchi]

1.1 (2013-05-31)

• Reorganized monkey patches. Everything patch-related now lives in the patches subpackage. [lgraf]

• Make sure @@search view doesn’t fail when called without parameters. [lgraf]

• Only display selected facets list if there actually are selected facets. [lgraf]

• Added spellchecking feature (aka “Did you mean …”). [buchi]

• Made respecting the navroot for searches configurable. Only if ISearchSettings.respect_navroot is set searches will be constrained to the navigation root (defaults to False). [lgraf]

• Added autocomplete support based on Solr’s suggester component. [buchi]

1.0.2 (2013-05-28)

• Fixed querytarget of livesearch for Plone 4.2 and later. Use our @@livesearch_reply view instead of livesearch_reply. [buchi]

• Include description in snippetText. [buchi]

• If there’s a SearchableText indexer, use it for snippetText generation. [buchi]

• Make length of breadcrumbs shown in search results configurable. [buchi]

• Added option to generate breadcrumbs from path rather than calling breadcrumbs_view for each item. [buchi]

• Added support for dexterity content types in snippetText indexer. [buchi]

1.0.1 (2013-05-21)

• Monkey patching c.solr.search.Search.buildQuery in order to escape slahes in paths. [lgraf]

• Overwrite search extender: Add write permissions, fixed translations and allowed content types in textfield. [Julian Infanger]

• Added option to scale Word Cloud by a constant factor. [lgraf]

• Added basic portlet to display Word Cloud. [lgraf]

• Added basic Word Cloud browser view. [lgraf]

1.0a1 (2012-08-22)

• Initial release