gocept.amqparchive 1.3.0.post1

Archiving, indexing and search for AMQP messages.

Please note: We do no longer actively support this package. If you still need it, please drop us a note, we can either transfer the repository or support you on a paid basis.

This package is an add-on application for gocept.amqprun that provides three features:

• Archive queue messages by writing them to the filesystesm

• Index those messages using ElasticSearch

• Search for messages with a HTML/JavaScript GUI

Contents

• Installation

• Development

• Changelog

Installation

gocept.amqparchive requires an ElasticSearch server. To set up the archive and index portion, add something like this to your gocept.amqprun configuration:

<eventlog>...</eventlog>
<amqp-server>...</amqp-server>
<worker>
  amount 2
  component-configuration site.zcml
</worker>
<settings>
  gocept.amqparchive.elastic_hostname localhost:9200
</settings>

and in site.zcml:

<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:amqp="http://namespaces.gocept.com/amqp">

  <include package="gocept.amqprun" />
  <include package="gocept.amqparchive" />

  <amqp:writefiles
    routing_key="key.one key.two key.three"
    queue_name="archive"
    directory="/path/to/archive"
    pattern="{routing_key}/{date}/{msgid}-{unique}.xml"
    />
</configure>

The HTML/JavaScript GUI expects /elasticsearch to proxy to the ElasticSearch server, and /messages to point to the archive directory (/path/to/archive in our example). Here’s an nginx config snippet:

http {
  upstream elasticsearch {
    server localhost:9200;
  }

  server {
    listen localhost:8080;

    location /search/ {
      alias /path/to/gocept.amqparchive.egg/gocept/amqparchive/browser/;
      index index.html;
    }

    location /elasticsearch/ {
      proxy_pass http://elasticsearch/;
    }

    location /messages/ {
      alias /path/to/archive/;
      autoindex on;
    }
}

Development

The source code is available in the mercurial repository at https://bitbucket.org/gocept/gocept.amqparchive

Please report any bugs you find at https://bitbucket.org/gocept/gocept.amqparchive/issues

Changelog

1.3.0.post1 (2018-11-27)

• Fix PyPI page rendering.

1.3.0 (2018-09-19)

• Adapt the Selenium-1 tests to Webdriver.

• Update to current bootstrap.py.

• Improve forward compatibility with Python 3.

• Force pyes < 0.17 for now.

1.2.7 (2015-01-28)

• Update to current bootstrap.py.

• Move repository to bitbucket.org.

1.2.6 (2014-02-17)

• Update to ElasticSearch 1.0 API.

1.2.5 (2014-02-14)

• Allow configuring multiple elasticsearch hosts.

• Fix bug in reindex_directory: we need to index the path relativ to the base directory, not the full filename.

• Start reindexing in parallel while files are still being collected.

1.2.4 (2013-11-27)

• Add parallel worker mode to reindex_directory.

• Handle invalid XML input (#10864).

1.2.3 (2012-04-18)

• Catch connection errors to ElasticSearch so they don’t break the normal message handling. Messages that have not been index due to this can still be indexed later on via reindex_directory (#9363).

1.2.2 (2012-03-29)

• Make amqp server configurable for tests.

1.2.1 (2012-02-22)

• Switch to plone.testing.

1.2.0 (2011-08-23)

• Transform the XML body into a nested dict so ElasticSearch can index the fields individually.

1.1.0 (2011-08-23)

• Add command-line script reindex_directory.

1.0.0 (2011-08-22)

• first release.