Skip to main content
This is a pre-production deployment of Warehouse. Changes made here affect the production instance of PyPI (pypi.python.org).
Help us improve Python packaging - Donate today!

Tail your log, extract the data, and send it to Fluentd

Project Description

Flosculus is a work-in-progress script to tail rotated log file, parse each line, and send it to Fluentd. It is intended to replace a subset of Fluentd in_tail features.

Crash Course

First things first, install Flosculus into your Python path.

$ git clone git://github.com/iromli/flosculus.git
$ cd flosculus
$ python setup.py install

Afterwards you need to create a configuration file to run the flosculusd event loop. The convenient way to do this boring thing is by invoking the flosculusd --init > flosculus.ini command. Here’s an example of flosculus.ini (you can call it whatever you like):

[flosculus]
; the IP address (or host name) of the remote server
remote_host = 127.0.0.1

; the TCP port of the remote server
remote_port = 24224


; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/access.log]

; the label
tag = example.api.access

; format to use, either use 'nginx' or custom regex
format = nginx

By default, the format option is matched against nginx default access log format. You may change the format as long as using a valid Python regex.

[log:/var/log/nginx/access.log]

format = (?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?

And did I already mentioned that you can have multiple log files? Simply copy the whole log section.

; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/access.log]

; the label
tag = example.api.access

; format to use, either use 'nginx' or custom regex
format = nginx

; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/timed-combined.access.log]

; the label
tag = test.api.access

; format to use, either use 'nginx' or custom regex
format = (?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?

; the IP address (or host name) of the remote server
; use another Fluentd remote host
remote_host = 10.0.0.1

; the TCP port of the remote server
; use another Fluentd remote port
remote_port = 24225

Assuming the configuration is written properly, run the event loop to see its magic (well, not really):

$ flosculusd -c flosculus.ini

Credits

Release History

Release History

This version
History Node

0.3.0

History Node

0.2

History Node

0.1

Download Files

Download Files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
flosculus-0.3.0.tar.gz (7.0 kB) Copy SHA256 Checksum SHA256 Source Jun 12, 2014

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting