Skip to main content

Tail your log, extract the data, and send it to Fluentd

Project description

Flosculus is a work-in-progress script to tail rotated log file, parse each line, and send it to Fluentd. It is intended to replace a subset of Fluentd in_tail features.

Crash Course

First things first, install Flosculus into your Python path.

$ git clone git://github.com/iromli/flosculus.git
$ cd flosculus
$ python setup.py install

Afterwards you need to create a configuration file to run the flosculusd event loop. The convenient way to do this boring thing is by invoking the flosculusd --init > flosculus.ini command. Here’s an example of flosculus.ini (you can call it whatever you like):

[flosculus]
; the IP address (or host name) of the remote server
remote_host = 127.0.0.1

; the TCP port of the remote server
remote_port = 24224


; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/access.log]

; the label
tag = example.api.access

; format to use, either use 'nginx' or custom regex
format = nginx

By default, the format option is matched against nginx default access log format. You may change the format as long as using a valid Python regex.

[log:/var/log/nginx/access.log]

format = (?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?

And did I already mentioned that you can have multiple log files? Simply copy the whole log section.

; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/access.log]

; the label
tag = example.api.access

; format to use, either use 'nginx' or custom regex
format = nginx

; Each section with `path:/path/to/log` is a valid config
[log:/var/log/nginx/timed-combined.access.log]

; the label
tag = test.api.access

; format to use, either use 'nginx' or custom regex
format = (?P<remote>[^ ]*) (?P<host>[^ ]*) (?P<user>[^ ]*) \[(?P<time>[^\]]*)\] "(?P<method>\S+)(?: +(?P<path>[^\"]*) +\S*)?" (?P<code>[^ ]*) (?P<size>[^ ]*)(?: "(?P<referer>[^\"]*)" "(?P<agent>[^\"]*)")(?: (?P<request_time>[^ ]*) (?P<upstream_time>[^ ]*) (?P<pipe>[\.|p]))?

; the IP address (or host name) of the remote server
; use another Fluentd remote host
remote_host = 10.0.0.1

; the TCP port of the remote server
; use another Fluentd remote port
remote_port = 24225

Assuming the configuration is written properly, run the event loop to see its magic (well, not really):

$ flosculusd -c flosculus.ini

Credits

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for flosculus, version 0.3.0
Filename, size File type Python version Upload date Hashes
Filename, size flosculus-0.3.0.tar.gz (7.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page