graphite_log_feeder 0.7.6

Ingests log files into Graphite where they can be visualized.

Overview

Graphite Log Feeder (GLF) is a generic tool that can parse application log files, extract meaningful data, and forward it to Graphite for visualizing. There is a parser included for ingesting data from The Grinder. This parser can be used as the basis for creating other parsers (for Apache, Scribe, your own application, or whatever.)

There is a similar tool – Logster – which also feeds log data to Graphite. However, Logster only supports realtime log analysis, whereas glf only supports non-realtime analysis, making the two tools complimentary.

Installation

GLF is installed via pip.

pip install graphite_log_feeder

Once pip has installed glf, you will need to generate a config file, and edit it to be suitable for your environment. Glf comes with a command-line option to generate a sample config file

graphite_log_feeder.py -e

This will generate a file named ‘glf.sample.conf’ which you can use as the basis for creating your own configuration.

There must be a running installation of Graphite on your network for GLF to forward data to. See the Graphite web site for details on setting up and configuring Graphite

http://graphite.wikidot.com/

Usage

(after adjusting the values in your sample config file to be appropriate for your environment)

graphite_log_feeder.py  <config_file>

Additional Resources

Java/JMX counters and application-level metrics can be fed to Graphite using the JMXTrans tool:

http://code.google.com/p/jmxtrans/

OS-level metrics (CPU, mem, etc.) can be fed to Graphite via quickstatd or collectd (with graphite plugin)

https://bitbucket.org/travis_bear/quickstatd