the ultimate tailer
Project description
The ultimate rotation and windows friendly log tailer plus a lot more…
backfills rotated files
handles gz and bz2 files
handles multi line records
doesn’t break windows rotating log handlers
easy to extend
Free software: BSD license
other than six pure Python stdlib.
Installation
pip install tailchaser
Thsi will install the tailchaser library in your site-packages and it will also add the script tailchase to your Scripts directory.
Usage
$ tailchase /where/my/logs/*.log $ tailchase -h usage: tailchase [-h] [--only-backfill] [--dont-backfill] [--clear-checkpoint] [--read-period READ_PERIOD] [--read-pause READ_PAUSE] [--reading-from {unix,win}] [--temp-dir TEMP_DIR] [--logging {DEBUG,INFO,WARN,ERROR,CRITICAL}] file-pattern positional arguments: file-pattern The file pattern to tail, such as /var/log/access.* optional arguments: -h, --help show this help message and exit --only-backfill dont't tail, default: False --dont-backfill basically only tail, default: False --clear-checkpoint start form the begining, default: False --read-period READ_PERIOD how long you read before you pause. If zero you don't pause, default: 1 --read-pause READ_PAUSE how long you pause between reads, default: 0 --reading-from PLATFORM sets how long you read and then pause can be one of {unix,win}, default: win --temp-dir TEMP_DIR on back fill files are copied to a temp directory.Use this to set this directory, default: None --logging LEVEL logging level it can be one of DEBUG,INFO,WARN,ERROR,CRITICAL, default: ERROR
In its simplest form
In its simplest form tailchase works like a combination of cat and tail -f except it will start with the oldest rotated file. So if you were to have
/var/log/opendirectoryd.log /var/log/opendirectoryd.log.0 /var/log/opendirectoryd.log.1 /var/log/opendirectoryd.log.2 /var/log/opendirectoryd.log.3 /var/log/opendirectoryd.log.4
it would first output the text of /var/log/opendirectoryd.log.4, then /var/log/opendirectoryd.log.3, etc. until it reached the newest file then when it has reached the end of /var/log/opendirectoryd.log revert to a “tail -f” behaviour.
So if you were to do
tailchase 'tests/logs/opendirectoryd.*'
“note: In bash you need to quote the wildcard otherwise it will be expanded into the scripts argv array.”
you would get something like
2015-12-24 15:39:56.733754 EST - AID: 0x0000000000000000 - Registered node with name '/Contacts' 2015-12-24 15:39:56.733933 EST - AID: 0x0000000000000000 - Registered node with name '/LDAPv3' as hidden 2015-12-24 15:39:56.736154 EST - AID: 0x0000000000000000 - Registered node with name '/Local' as hidden 2015-12-24 15:39:56.736868 EST - AID: 0x0000000000000000 - Registered node with name '/NIS' as hidden 2015-12-24 15:39:56.737134 EST - AID: 0x0000000000000000 - Discovered configuration for node name '/Search' at path ' 2015-12-24 15:39:56.737151 EST - AID: 0x0000000000000000 - Registered node with name '/Search' 2015-12-24 15:39:56.738794 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules 2015-12-24 15:39:56.740509 EST - AID: 0x0000000000000000 - Loaded bundle at path '/System/Library/OpenDirectory/Modules/
Coding with tailchaser
Using the tailchaser library in a project is probably best done by example.
Example 1 - Tailchase to a REST service.
# # Example 1 - Tail to Elastic # import requests import tailchaser class TailToElastic(tailchaser.Tailer): def handoff(self, file_tailed, checkpoint, record): """ Expect a record like: 20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties] 20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties] 20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties] """ date, time, level, source, _, message = record.split(5) result = requests.json("http://someelacticserver.com:9200/myindex/log", json={ 'timestamp': '{}T{}'.format(date, time) 'level': level, 'source': source, 'message': message }) return result.status_code == requests.codes.ok
Example 2 - Tailchase to Kafka
# # Example 2 - Tail to Kafka - shows how to add your own arguments and then send messages to kafka. # from kafka import KafkaProducer rom tailchaser.tailer import Tailer class TailToKafka(Tailer): def add_arguments(cls, parser=None): parser = super(TailToKafka, cls).add_arguments(parser) HOSTS = 'localhost:1234' TOPIC = b'log' def startup(self): self.kafka_producer = KafkaProducer(bootstrap_servers=self.HOSTS,value_serializer=msgpack.dumps) def handoff(self, file_tailed, checkpoint, record): """ Expect a record like: 20160204 10:28:15,525 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/lme-market.properties] 20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-database.properties] 20160204 10:28:15,541 INFO PropertiesLoaderSupport - Loading properties file from URL [file:C:/WaterWorks/Broken/BSE//config/default-hibernate.properties] """ self.kafka_producer.send(self.TOPIC, record).get(timeout=10) return True
Example 3 - Saving and Loading last Offset from Zookeeper
# # Example 3 - Saving and Loading last Offset from Zookeeper # import platform import pickle import six from kazoo.client import KazooClient from tailchaser.tailer import Tailer class TailToElastic(Tailer): def start(self): self.zk = KazooClient(hosts=self.config.zookeeper_host) self.zk.start() def stop(): self.zk.stop() def load_checkpoint(self): zk_path = self.config.checkpoint_filename try: checkpoint = pickle.loads(self.zk.get(zk_path)) log.debug('loaded: %s %s', zk_path, checkpoint) return checkpoint except (IOError, EOFError): log.debug('failed to load: %s', zk_path) return self.INIT_CHECKPOINT def save_checkpoint(self, checkpoint): log.debug('dumping %s %s', self.config.checkpoint_filename, checkpoint) return self.zk.set(self.config.checkpoint_filename, six.b(pickle.dumps(checkpoint))) @staticmethod def make_checkpoint_filename(source_pattern, path=None): zk_path = '/'.join(['tailchase', platform.node(), slugify(source_pattern)] self.zk.ensure_path(zk_path) return zk_path
Documentation
Development
To run the all tests run:
tox
Note, to combine the coverage data from all the tox environments run:
Windows |
set PYTEST_ADDOPTS=--cov-append tox |
---|---|
Other |
PYTEST_ADDOPTS=--cov-append tox |
Changelog
0.1.0 (2016-02-21)
First release on PyPI.
0.1.2 (2016-04-05)
Add multi-line record support - #3
add –start-of-record-re option for multi-line records. - #4
move argparser definition from cli to Tailer class. - #5
0.1.3 (2016-04-12)
Added break tail tests and fixes drived from it - #6
fixed app description ! - #7
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for tailchaser-0.2.6-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0fd6b292e55c49547c3d0d6c3ed9c0787e6759fed38e473121f72155af8fcad6 |
|
MD5 | 4fd4b1b87e937c271b23290c5e29d212 |
|
BLAKE2b-256 | 5d344f6823e1b71ee074ec6d385fbdb3687b4f2e9b104a02e740dcd2a3188680 |