ftw.solr 2.11.0

Solr integration for Plone

Introduction

ftw.solr integrates the Solr search engine with Plone.

Important

Since version 2.0 ftw.solr no longer depends on collective.solr. Instead it provides it’s own Solr integration using Solr’s REST API. Version 2.0 is a complete rewrite and is not compatible with ftw.solr 1.x.

ftw.solr does not try to replace the portal catalog of Plone and does not hook into the catalog’s search function. Instead it provides a search utiltity that must be used explicitly. It’s meant to be used in search forms for fulltext searches while the portal catalog is still in use for things like navigation or folder contents. The goal is to get rid off all fulltext indexes (e.g. ZCTextIndex) in the portal catalog.

ftw.solr requires Apache Solr 7.0 or higher.

Installation

Add as dependency

Install ftw.solr by adding it to the list of eggs in your buildout or by adding it as a dependency of your policy package.

[instance]
eggs +=
    ftw.solr

Solr installation

To install a Solr server with buildout you can use the ftw.recipe.solr recipe.

[solr]
recipe = ftw.recipe.solr
cores = mycore

Configure the Solr connection

The connections settings for Solr can be configured in ZCML and thus in buildout. Example:

[instance]
zcml-additional =
    <configure xmlns:solr="http://namespaces.plone.org/solr">
        <solr:connection host="localhost" port="8983" base="/solr/mycore"/>
   </configure>

By default, ftw.solr will do full text extraction by passing the blob’s filesystem path to the Solr Cell extract handler, assuming that Solr runs on the same machine and has access to the blob storage.

For setups where this isn’t desired, the connection option upload_blobs can be set to true in order to make ftw.solr upload the blobs directly to the extract handler via HTTP POST:

[instance]
zcml-additional =
    <configure xmlns:solr="http://namespaces.plone.org/solr">
        <solr:connection host="localhost" port="8983" base="/solr/mycore" upload_blobs="true"/>
   </configure>

Run buildout

After running buildout and restarting your instance you can install the ftw.solr addon in Plone.

Usage

Get the ISolrSearch utility and call the search method to get search results from Solr.

from ftw.solr.interfaces import ISolrSearch
from zope.component import getUtility

solr = getUtility(ISolrSearch)
resp = solr.search(query=u'SearchableText:foo')

You can get a plone.app.contentlisting style result by adapting IContentListing:

from plone.app.contentlisting.interfaces import IContentListing
listing = IContentListing(resp)

Solr Index Maintenance

For indexing Plone content and other maintenance work you can use the solr Zope command. Run bin/instance solr -h for available options.

Clear the Solr index:

bin/instance solr clear

Rebuild the complete Solr index:

bin/instance solr reindex

Reindex specific indexes:

bin/instance solr reindex -i modified created

Synchronize the Solr index with the portal catalog:

bin/instance solr sync

Links

• Github: https://github.com/4teamwork/ftw.solr

• Issues: https://github.com/4teamwork/ftw.solr/issues

• Pypi: http://pypi.python.org/pypi/ftw.solr

• Continuous integration: https://jenkins.4teamwork.ch/search?q=ftw.solr

Copyright

This package is copyright by 4teamwork.

ftw.solr is licensed under GNU General Public License, version 2.

Changelog

2.11.0 (2022-07-04)

• Add support for language specific fields. [buchi]

2.10.0 (2021-09-21)

• Add flag to force clearing solr even when solr is not enabled. [njohner]

2.9.5 (2021-08-17)

• Add unrestricted_search method. [njohner]

2.9.4 (2021-07-29)

• SolrDocument returns None for existing fields with missing value. [njohner]

2.9.3 (2021-04-22)

• Fix indexing of SearchableText only for objects with primary field. [njohner]

2.9.2 (2021-01-13)

• Do not ignore None values when indexing data in solr. [njohner]

2.9.1 (2020-12-15)

• Cleanup extract_commands before sending them to solr. [njohner]

2.9.0 (2020-08-04)

• Treat docs with no created field as out of sync in diff. [deiferni]

2.8.6 (2020-06-02)

• Log maintenance commands to a dedicated logfile. [lgraf]

• Add option to limit items logged in diff, switch default to only diff 5 items. [deiferni]

• Add helper for creating Solr filters from catalog queries. [buchi]

2.8.5 (2020-05-06)

• Make sure maintenance scripts are run as system user. [njohner]

2.8.4 (2020-04-02)

• Fix update of the modified index. [njohner]

2.8.3 (2020-04-02)

• Update the modified index to match new rounding of DateTime. [njohner]

2.8.2 (2020-03-27)

• Fix to_iso8601 prevent rounding errors with DateTime milliseconds above 999500. [deiferni]

• Fix blob extraction for DX items. [mbaechtold]

2.8.1 (2020-02-05)

• Fix to_iso8601 for years before 1900. [deiferni]

2.8.0 (2019-12-06)

• Add support for uploading blobs to extract handler. [lgraf]

• Add support for Plone 5.0 and 5.1. [buchi]

2.7.0 (2019-09-16)

• Add console script for maintenance tasks. [buchi]

• Add registry setting for enabling/disabling indexing. [buchi]

2.6.2 (2019-09-12)

• Avoid sending atomic update commands with null-documents. [lgraf]

2.6.1 (2019-08-26)

• Fix ‘path_depth’ indexing handling, when ‘path_depth’ is not part of the schema. [phgross]

2.6.0 (2019-08-19)

• Use Solr Cell only for text extraction. The SearchableText is now also indexed using the update handler which allows atomic updates. Also fixes #137. [buchi]

• Fix error when trying to delete an object without a unique key. [jone]

2.5.0 (2019-07-02)

• Make sure to always update ‘path_depth’ if ‘path’ is (re)indexed. [lgraf]

• Handle facet_counts in solr response. [njohner]

2.4.0 (2019-06-13)

• Perform soft commits and wait for new searcher by default to allow near real time (NRT) searching. [buchi]

• Additionaly index metadata of files using an update command to make them visible as soon as possible. [buchi]

• Flush optimize command. [buchi]

• Add support for indexing path depth. [buchi]

2.3.2 (2019-04-29)

• Fix partial reindexing (atomic updates) for fields without a value. [buchi]

2.3.1 (2018-11-01)

• Fix reindexObjectSecurity patch to account for special handling of system roles in allowedRolesAndUsers indexer. [lgraf]

2.3.0 (2018-10-18)

• Patch reindexObjectSecurity to optimize indexing of large trees. [lgraf]

• Fix diff for items with a modification date that has no fractional seconds. [buchi]

• Provide an error message when raising an AttributeError in SolrDocument. [buchi]

2.2.0 (2018-09-21)

• Add option to synchronize Solr with catalog in maintenance view. [buchi]

2.1.2 (2018-05-15)

• Set Content-Type ‘application/x-www-form-urlencoded’ for requests to /update/extract endpoint to ensure compatibility with Solr 7.3. [lgraf]

• Maintenance view: Allow reindexing without dooming the transaction. [lgraf]

2.1.1 (2018-02-20)

• Make sure we do not pass invalid dates to Solr. [buchi]

2.1.0 (2018-02-20)

• Declare dependency on plone.namedfile[blobs]. [lgraf]

• Add diff function to maintenance view to compare UIDs in portal catalog with UIDs in Solr. [buchi]

• Fix security filter: Properly quote roles/users with spaces. [buchi]

• Add option to reindex Solr from a catalog query. [buchi]

2.0.0 (2018-02-15)

• Complete rewrite. Not compatible with previous versions. [buchi]