Skip to main content

A webapp to query fedmsg history

Project description

datagrepper
===========

A webapp to retrieve historical information about messages on the `fedmsg
<http://fedmsg.com>`_ bus. It is a JSON api for the `datanommer
<https://github.com/fedora-infra/datanommer/>`_ message store.

Production Instance
-------------------

https://apps.fedoraproject.org/datagrepper/

Hacking on datagrepper
----------------------

Setting up the stack
~~~~~~~~~~~~~~~~~~~~

Use a virtualenv::

$ mkvirtualenv datagrepper
$ workon datagrepper

Install dependencies::

$ pip install -r requirements.txt
$ pip install psycopg2

Set up the fedmsg consumer for the job runner::

$ python setup.py develop

Configuring Postgresql (and getting some data)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In normal operations, the `datanommer
<https://github.com/fedora-infra/datanommer>`_ consumer daemon will be
running somewhere and continuously stuff each new `fedmsg
<http://fedmsg.com>`_ message that it sees into a postgres DB. If you're
just sitting down to hack on datagrepper, that won't be your situation
so you'll need a dump of the database.

.. note:: If you've tried installing postgres before and think you've
messed it up, you'll need to blow away the old databases with
``$ rm -rf /var/lib/pgsql``

Install postgres (and fedmsg, while we're at it)::

$ sudo yum install -y postgresql-server fedmsg
$ sudo postgresql-setup initdb

Make sure postgres is set to allow connections over tcp/ip using password
authentication. Edit the ``/var/lib/pgsql/data/pg_hba.conf``. You might
find a line like this::

host all all 127.0.0.1/32 ident

Instead of that line, you need one that looks like this::

host all all 127.0.0.1/32 md5

----

Become yourself again (not the ``postgres`` user) and start up postgres::

$ sudo systemctl restart postgresql.service

Become the postgres user (again) and run the psql command. Use that psql
shell to setup the DB, the user, and privileges::

$ sudo su - postgres
$ psql
# create database datanommer;
# create user datanommer with password 'bunbunbun';
# grant all privileges on database datanommer to datanommer;
# \q

Back in the bash shell (but still as the `postgres` user), grab a DB dump and
restore it::

$ wget http://ralph.fedorapeople.org/datanommer-2014-01-31.dump.xz
$ xzcat datanommer-2014-01-31.dump.xz | psql datanommer

Last step, run datagrepper
~~~~~~~~~~~~~~~~~~~~~~~~~~

You have to configure your development datagrepper instance to talk to
postgres (by default, it looks for a sqlite database). Edit
``fedmsg.d/example-datagrepper.py`` and give it these contents:

.. code-block:: python

config = {
'datanommer.enabled': False,
'datanommer.sqlalchemy.url': 'postgresql+psycopg2://datanommer:bunbunbun@localhost:5432/datanommer',
'fedmsg.consumers.datagrepper-runner.enabled': True,
}

As your normal old user self, run the development server::

$ workon datagrepper
$ python runserver.py

In a browser, visit http://localhost:5000 to see the docs.

You can quick test that you can get data by running::

$ sudo yum install -y httpie
$ http get localhost:5000/raw/ delta==1000000 rows_per_page==1

Running the job runner
~~~~~~~~~~~~~~~~~~~~~~

Without starting ``fedmsg-hub``, the job runner won't actually run jobs::

$ workon datagrepper
$ fedmsg-hub

Project details


Release history Release notifications

History Node

0.9.4

History Node

0.9.3

History Node

0.9.2

History Node

0.9.1

History Node

0.9.0

History Node

0.8.0

History Node

0.7.1

History Node

0.7.0

History Node

0.6.0

History Node

0.5.1

History Node

0.5.0

History Node

0.4.2

History Node

0.4.1

History Node

0.4.0

History Node

0.3.3

History Node

0.3.2

This version
History Node

0.3.1

History Node

0.3.0

History Node

0.2.1

History Node

0.2.0

History Node

0.1.4

History Node

0.1.3

History Node

0.1.2

History Node

0.1.1

History Node

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
datagrepper-0.3.1.tar.gz (110.3 kB) Copy SHA256 hash SHA256 Source None Feb 9, 2014

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page