logdog 0.2.11

logs watching + webui

python-logdog
-------------

distributed log tail viewer

Why?

- tail log files and forward them to web in runtime
- dynamically parse and (pre)process logs
- aggregate and collect logs
- alerting


Quick start
===========

Running a simple one-host configuration.

.. code-block:: bash

$ pip install logdog

Help output:

.. code-block:: bash

$  logdog --help 
logdog command line interface

Usage:
logdog [<pipe-namespace>...] [options]
logdog (-h | --help)
logdog --version

Arguments:
<pipe-namespace> One or more pipe namespaces to be run

Options:
-h --help Show this screen
--version Show version
-v --verbose Run in verbose mode
-l --log-level=<level> Set internal logging level [default: INFO]
-f --log-format=<format> Set internal logging format [default: quiet]
-c --config=<config> Configuration file (yaml config)
-s --sources=<file:...> Force specify files to be watched
-H --handler=<handler> Force set handler for all sources
(e.g. --handler=viewers.console)
--reset-indices Remove current indices (will reset watching state)


Prepare config file:

.. code-block:: yaml

# config.yml
---
sources:
- /var/log/*.log
- /var/log/*/*.log
- /var/log/syslog

Please, see `default_config.py`_

.. _default_config.py: logdog/default_config.py

Start watching:

.. code-block:: bash

$ logdog --config=config.yml

You can run watching and viewing parts separately:

.. code-block:: bash

$ logdog watch --config=config.yml
# another console
$ logdog view --config=config.yml


Config
======

YAML is used as a main format for writing configuration.

Default config:

.. code-block:: yaml

---
sources:
# <path-to-file>
- /var/log/*.log
- /var/log/*/*.log
- /var/log/syslog

``sources`` is a list of target files/logs. Alternatively, this section can be re-written the following way:

.. code-block:: yaml

---
sources:
- /var/log/*.log: pipes.to-web
- /var/log/*/*.log:
handler: pipes.to-web
# ^ note. 4 spaces
# in case of 2 spaces it will be a key in the list object
# {'/var/log/*/*.log': None,
# 'handler': 'pipes.to-web'}
# but must be {'/var/log/*/*.log': {'handler': 'pipes.to-web'}}
- /var/log/syslog: {handler: pipes.to-web}


Pipe is a sequence of steps to process / parse / forward / collect log messages.
``pipes.to-web`` is a predefined pipe (see `default_config.py`_).


Full ``sources`` format:

.. code-block:: none

---
sources:
- (path | search pattern)
# or (`handler`, `watcher`, `meta` are optional)
- (path | search pattern):
handler: handler-name # default pipes.to-web
watcher: watcher-name # default pollers.file-watcher
meta: a-dictionary-containing-any-meta-info # e.g. {tags: [tag1, tag2]}
# or
- (path | search pattern): handler-name
# or
- (path | search pattern): {handler: pipes.to-web}
# or
- (path | search pattern): {watcher: poller.custom-file-poller}
# or
- (path | search pattern): {meta: {tags: [log]}}


Example 1:

.. code-block:: yaml

---
sources:
- /var/log/syslog: {handler: pipes.to-web, meta: {tags: [syslog]}
# or
- /var/log/syslog2:
handler: pipes.to-web
meta:
tags:
- syslog


Builtins
========

Predefined configs:

``pipes``:

- `pipes.to-web` - defines a simple flow (strip -> zmq localhost:7789 -> zmq *:7789 -> webui)

``viewers``:

- `viewers.null` - does nothing with incoming data
- `viewers.console` - print incoming log messages into stdout
- `viewers.webui` - forward all incoming messages to all connected clients using websockets

``connectors``:

- `connectors.zmq-tunnel` - allows to create any zmq sockets to push/pull data

For more details see `default_config.py`_.

Screenshots
===========

.. image:: http://i.imgur.com/B4JQ57T.png


TODO
====

- cover with tests
- detecting new files
- colorize logs
- add documentation
- zmq connectors
- mongodb collector
- webui storages
- webui filtering / searching
- implement `--validate-config`