This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Project Description

This is yet another real-time log scanner done for the ever reoccurring reason of failure to find an existing suitable tool.

Features

  • simple, yet flexible configuration in the YAML format
  • actions programmable in Python, Tcl or Bash
  • log rotation detection
  • variable leading log context for every event
  • burst mode: collect events within a certain time frame and report them together

Configuration

Lognotify reads a configuration file specified on the command line with the --config or -c option. The configuration specifies what content to search for and what to do if some is found in the log.

Overview

Consider this example:

when:
    - error
    - problem
    - critical
    - fatal
    - bad
    -
        - not
        - ^perl
        - ^/pascal/i
do:
    python: |
        if logfile:
            print(logfile, messages[-1][-1])
    burst: 2
---
when:
    - strange
do: |
    echo $logfile: ${message[-1]}

A configuration can have one ore more sections, denoted by a --- delimiting line. Each section specifies what to look for and what to do if something interesting is found.

Sections

Each section consists of a when clause and a do clause. For every line from a logfile, all sections are checked for matching search expressions in the when clause. When a match is found for a section, the corresponding do clause is executed.

When clause

The when clause contains an itemized list of expressions to searche for in every incoming line from the log. Variations in syntax specify how to search for the item. The expressions are tried in order. As soon as a match is found, the do clause is executed and processing of this particular section is terminated.

The when list forms an OR-expression. Within the list, sublists may specify AND-expressions. Thus

- expr1
-
    - expr2a
    - expr2b

means

expr1 OR (expr2a AND expr2b)

Groups can be infinitely nested. The general rule is that every group within an OR group is an AND group and vice versa.

Search expressions

A search expression can have one of the following forms:

word

Search for word irrespective of case and match only at word boundaries. Thus error matches in the following lines:

Error: invalid syntax

An error occurred.

No error-checking enabled.

but not in:

No errorchecking.

maxerror

If you want to match irrespective of word boundaries, you have to use regular expressions (see below).

/word/[flags]

Search for a regular expression. Some flags for altering the operation are available:

i
match case insensitive. See IGNORECASE.
m
match in multi-line mode. Probably not very useful. See MULTILINE.
l
match according to the current locale. See LOCALE.
s
make ‘.’ match any character, including newline. See DOTALL.
u
match according to the Unicode character properties table. See UNICODE.
x
parse verbose regex with comments and white space. See VERBOSE.

All these expressions can be prefixed with a caret (^) to mean “do not match word”:

^word

^/word/

Note

Since the whole configuration is expressed in YAML, strings containing certain characters must be quoted in order not to interfere with the YAML syntax. These characters are: [ ] { } ! " ' : ? % @ , - # ~ | > * &. Also, certain words have special meaning in YAML and must therefore also be quoted: yes, no, on, off, true, false, and null.

Pitfalls

The search algorithm gives rise to surprises in certain constellations. One common error is to request something like this:

-
    - not
    - ^this
-
    - not
    - ^that

where ^this and ^that cancel each other out. If a line contains ‘not’ it will always match, no matter whether this or that occurs in the line. The proper way would be

-
    - not
    - ^this
    - ^that

The most common pattern is to search for any line containing word1, word2 or word3 but not except1 or except2. You might be inclined to write this as

- word1
- word2
- word3
-
    - ^except1
    - ^except2

But this would not work. The way to do it goes along the follong lines: written as a logical expression, it would be

(word1 OR word2 OR word3) AND (NOT except1 OR NOT except2)

which translates to

(word1 OR word2 OR word3) AND NOT except1 AND NOT except2

which, expressed as list operations, translates to

AND(OR(word1, word2, word3), NOT(except1), NOT(except2))

We have therefore an AND list on top. However, in lognotify we start out in an OR list. We therefore have to put our AND list as the single element into the top OR list. The final result would be

# OR list
-
    # AND list
    -
        # OR list
        - word1
        - word2
        - word3
    - ^except1
    - ^except2

Do clause

The do clause specifies what action to take when one of the expressions in the when clause matches. To run commands on the selected logfile lines, Python, bash or Tcl can be used. Some variables are injected, depending on the language used. Scripts receive one or more events at a time depending on whether context and/or burst mode was requested. If neither context not burst mode is requested, one single line is reported at a time.

Note

Use the pipe character at the end of a line prior to the code block to cause YAML to process the following indented block without interpretation, leaving line endings intact (see the examples below).

Context

Context is a number of lines running up to the actual log event line. It can be requested with the --config/-C flag. Context lines are marked with a True value in Python or Tcl or a value of 1 in bash or sh to distinguish them from log lines. However, if a context line is also a regular log line (appearing because it is part of a burst) it is not marked as such.

Burst mode

In burst mode, log lines arriving within a certain time frame are kept together and appear in the same call. Burst mode can be requested either as a burst specifier in a do clause or with a --burst or --force-burst command line flag. Good values for burst time frames are between 2 and 5 seconds. The --force-burst flag overrides values specified in do clauses while --burst does not.

Python

Python code can be one block or be split into an initialization section and a runtime section. The former is executed once at startup and is intended to contain stuff like import statements, function definitions and the like. The latter is run for every event.

In Python, the following variables are available:

logfile
A string containing the path of the logfile where the event was coming from
messages
A list of tuples. For each event the tuple contains:
  • a bool which is True if the entry is a context line
  • the sequence number
  • a float with a timestamp
  • a string with the message text

Example (assuming Python3 syntax):

do:
    python: |
        for msg in messages:
            print('%s: %s' % (logfile, msg[3]))

Example with initialization (assuming Python3 syntax):

python:
    - |
        # Setup UDP socket
        import sys
        import socket
        sock = socket.socket(type=socket.SOCK_DGRAM)
        sock.connect(('127.0.0.1', 7777))
    - |
        # Write stuff to UDP socket
        for msg in messages:
            sock.send('json:{}\n'.format(msg[3].replace(r'\n', ' ')).encode('u8'))

Tcl

Tcl code can be one block or be split into an initialization section and a runtime section. The former is executed once at startup and is intended to contain stuff like proc statements. The latter is run for every event.

In Tcl, the following variables are available:

logfile
The path of the logfile where the event was coming from
messages
A list of lists. For each event the inner list contains:
  • a bool which is True if the entry is a context line
  • the sequence number
  • an int with a timestamp
  • a string with the message text

Example:

do:
    tcl: |
        foreach m $messages {
            puts "$logfile: [clock format [expr int([lindex $m 2])]] [lindex $m 3]"
        }

Example with initialization:

do:
    tcl:
        - |
            proc output {m} {
                puts $m
            }
        - |
            foreach m $messages {
                output "$logfile: [clock format [expr int([lindex $m 2])]] [lindex $m 3]"
            }

Bash and sh

In bash and sh, the following variables are available:

logfile
The path of the logfile where the event was coming from
iscontext
An array with an int for every line where 1 means it is a context line or 0 otherwise
seqno
An array containing the sequence number for every line
time
An array containing the timestamp in ISO format for every line
message
An array containing the text for every line

Example:

do:
    bash: |
        echo $logfile: ${time[-1]} ${message[-1]}

But since bash is the default language, it can be written as:

do: |
    echo $logfile: ${message[-1]}

The do clause can be omitted altogether in which case a default of

do:
    python: |
        for msg in messages:
            print('%s: %s' % (logfile, msg[3]))

is assumed.

Running

Command synopsis:
lognotify [-h] –config CONFIG [–full]
[–burst BURST | –force-burst FORCE_BURST] [–context CONTEXT] [–debug] [–version] logfile [logfile …]
Positional arguments:
logfile
A log file to scan
Optional arguments:
-h, --help show this help message and exit
--config CONFIG, -c CONFIG
 specify config file
--full, -f scan files from beginning
--burst BURST, -b BURST
 report bursts of BURST seconds together
--force-burst FORCE_BURST, -B FORCE_BURST
 force reporting bursts of BURST seconds together
--context CONTEXT, -C CONTEXT
 specify context size
--debug, -d Print some debug information to stderr
--version, -v display version and exit

At least one path to an existing, readable log file is expected.

The --full or -f option requests reading files from the start. Without the flag, reading begins at the current end of file. Sequence numbering always begins from the point where reading begins.

The --debug or -d option sends information to the standard error file. Repeating the flag increases the amount of information.

Useful scripts

This section is a collection of useful scripts.

Send desktop notification

To be used as root (change 'username' accordingly):

from subprocess import check_call

check_call(
    [
        'su', 'username', '-c',
        'DISPLAY=:0 notify-send "%s" "%s"' % (logfile, '\n'.join('> '[m[0]] + m[3] for m in messages))
    ]
)

Send desktop notification to remote machine

To be used as root (change 'hostname' and username accordingly):

from subprocess import check_call
from platform import node

check_call(
    [
        'ssh', 'hostname',
        r'su username -c "DISPLAY=:0 notify-send \"%s: %s\" \"%s\""' % (
            node(),
            logfile,
            '\n'.join('! '[m[0]] + m[3] for m in messages)
        )
    ]
)

Send e-mail

Change 'mail-user', 'mail-user-password', source-email and destination-email accordingly:

do:
    python: |
        from smtplib import SMTP
        import sys

        client = SMTP('localhost')
        try:
            client.starttls()
        except:
            pass
        client.login('mail-user', 'mail-user-password')
        client.sendmail(
            'source-email',
            'destination-email',
            'From: source-email\n'
            'To: destination-email\n'
            'Subject: Message in %s\n\n'
            '%s\n' % (logfile, '\n'.join('> '[m[0]] + m[3] for m in messages))
        )
Release History

Release History

0.3

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.6

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.5

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.1.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
jw.lognotify-0.3.tar.gz (14.1 kB) Copy SHA256 Checksum SHA256 Source Jan 27, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting