tailchaser 0.2.8

In its simplest form

In its simplest form tailchase works like a combination of cat and tail -f except it will start with the oldest rotated file. So if you were to have

/var/log/opendirectoryd.log
/var/log/opendirectoryd.log.0
/var/log/opendirectoryd.log.1
/var/log/opendirectoryd.log.2
/var/log/opendirectoryd.log.3
/var/log/opendirectoryd.log.4

it would first output the text of /var/log/opendirectoryd.log.4, then /var/log/opendirectoryd.log.3, etc. until it reached the newest file then when it has reached the end of /var/log/opendirectoryd.log revert to a “tail -f” behaviour.

So if you were to do

tailchase 'tests/logs/opendirectoryd.*'

“note: In bash you need to quote the wildcard otherwise it will be expanded into the scripts argv array.”

you would get something like

2015-12-24 15:39:56.733754 EST - AID: 0x0000000000000000 - Registered node with name '/Contacts'
2015-12-24 15:39:56.733933 EST - AID: 0x0000000000000000 - Registered node with name '/LDAPv3' as hidden
2015-12-24 15:39:56.736154 EST - AID: 0x0000000000000000 - Registered node with name '/Local' as hidden
2015-12-24 15:39:56.736868 EST - AID: 0x0000000000000000 - Registered node with name '/NIS' as hidden
2015-12-24 15:39:56.737134 EST - AID: 0x0000000000000000 - Discovered configuration for node name '/Search' at path '       2015-12-24 15:39:56.737151 EST - AID: 0x0000000000000000 - Registered node with name '/Search'
2015-12-24 15:39:56.738794 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules     2015-12-24 15:39:56.740509 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules/

Example 1 - Tailchase to a REST service.

#
# Example 1 - Tail to Elastic
#

import requests

import tailchaser

class TailToElastic(tailchaser.Tailer):
    def handoff(self, file_tailed, checkpoint, record):
        """ Expect a record like:

        20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties]
        20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties]
        20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties]
        """

        date, time, level, source, _, message = record.split(5)
        result = requests.json("http://someelacticserver.com:9200/myindex/log", json={
                        'timestamp': '{}T{}'.format(date, time)
                        'level': level,
                        'source': source,
                        'message': message
                        })
        return result.status_code == requests.codes.ok

Example 2 - Tailchase to Kafka

#
# Example 2 - Tail to Kafka - shows how to add your own arguments and then send messages to kafka.
#

from kafka import KafkaProducer
rom tailchaser.tailer import Tailer


class TailToKafka(Tailer):
    def add_arguments(cls, parser=None):
        parser = super(TailToKafka, cls).add_arguments(parser)

    HOSTS = 'localhost:1234'
    TOPIC = b'log'
    def startup(self):
        self.kafka_producer = KafkaProducer(bootstrap_servers=self.HOSTS,value_serializer=msgpack.dumps)


    def handoff(self, file_tailed, checkpoint, record):
        """ Expect a record like:

        20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties]
        20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties]
        20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties]
        """
        self.kafka_producer.send(self.TOPIC, record).get(timeout=10)
        return True

Example 3 - Saving and Loading last Offset from Zookeeper

#
# Example 3 - Saving and Loading last Offset from Zookeeper
#
import platform
import pickle
import six
from kazoo.client import KazooClient
from tailchaser.tailer import Tailer

class TailToElastic(Tailer):
    def start(self):
        self.zk = KazooClient(hosts=self.config.zookeeper_host)
        self.zk.start()


    def stop():
        self.zk.stop()

    def load_checkpoint(self):
        zk_path = self.config.checkpoint_filename
        try:
            checkpoint = pickle.loads(self.zk.get(zk_path))
            log.debug('loaded: %s %s', zk_path, checkpoint)
            return checkpoint
        except (IOError, EOFError):
            log.debug('failed to load: %s', zk_path)
            return self.INIT_CHECKPOINT

    def save_checkpoint(self, checkpoint):
        log.debug('dumping %s %s', self.config.checkpoint_filename, checkpoint)
        return self.zk.set(self.config.checkpoint_filename, six.b(pickle.dumps(checkpoint)))


    @staticmethod
    def make_checkpoint_filename(source_pattern, path=None):
        zk_path =  '/'.join(['tailchase', platform.node(), slugify(source_pattern)]
        self.zk.ensure_path(zk_path)
        return zk_path

0.2.7 (2016-07-24)

• added –filter-re to filter out lines #15

0.2.1 (2016-04-30)

• Fixed #12 where Handoff is getting the tmp file name…

• removed temp files after handoff #14

0.2.0 (2016-04-30)

• Added pipe code from ilabs code base.

• Added example of storing check points remotely #8

• Added REST configuration #9

• Added INIT_CHECKPOINT class attribute to Tailer #11

0.1.3 (2016-04-12)

• Added break tail tests and fixes drived from it - #6

• fixed app description ! - #7

0.1.2 (2016-04-05)

• Add multi-line record support - #3

• add –start-of-record-re option for multi-line records. - #4

• move argparser definition from cli to Tailer class. - #5

0.1.0 (2016-02-21)

• First release on PyPI.