Skip to main content

Pipe stuff to Loggly.

Project description

https://travis-ci.org/modcloth-labs/loggly-pipe.png?branch=master
  • Eats lines from stdin.
  • Sends batches to Loggly (or compatible).
  • Poops unaltered lines to stdout.
  • Poops mark JSON to stdout.

Installation

loggly-pipe is built to be executed as a standalone file, so installing directly from the raw github URL works great:

curl -L -o loggly-pipe https://raw.github.com/modcloth-labs/loggly-pipe/master/loggly_pipe.py
chmod +x loggly-pipe

You can also grab it via pip, which will install a loggly-pipe executable:

pip install loggly_pipe

Configuration

Configuration may be given via the command line or the env. The latest command line options are available like one might expect:

loggly-pipe --help

Here’s a list of the environment variables that are used as the option defaults:

  • LOGGLY_TOKEN REQUIRED one of the tokens from your Loggly account
  • LOGGLY_SERVER the base server URI (default https://logs-01.loggly.com)
  • LOGGLY_SHIPPER_COUNT number of shipper threads (default 1)
  • LOGGLY_TAGS comma-delimited tags to apply to the shipped records. This is passed directly as the last member of PATH_INFO, so it should be URL-encoded if necessary. (default python)
  • LOGGLY_BATCH_SIZE number of records to accumulate before shipping over HTTP (default 100)
  • LOGGLY_MAX_LINES exit after consuming this many lines (default 0, which disables)
  • LOGGLY_STDIN_SLEEP_INTERVAL interval in seconds to sleep when line read from stdin is empty (default 0.1)
  • LOGGLY_FLUSH_INTERVAL interval in seconds at which to flush the line buffer so that low activity doesn’t cause records to back up (default 10.0)
  • LOGGLY_SHIP_ATTEMPTS number of times to retry each batch over HTTP (default 1)
  • LOGGLY_ON_ERROR behavior when exceptions are caught, which is usually around shipping over HTTP, and will only take effect once LOGGLY_SHIP_ATTEMPTS has been reached (choices are raise, ignore; default raise)
  • LOGGLY_DEBUG print some debuggy stuff if non-empty

Usage

Given a stdin stream made of line-based JSON records, loggly-pipe will emit each record unaltered as well as a record with a single _mark key for each batch.

Here’s an example of a node app that uses bunyan for logging as well as the bunyan command line tool for human readability:

node server.js | loggly-pipe | bunyan

A trivial example that’s used in the black-box test of loggly-pipe itself can be found in the examples dir.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
loggly_pipe-0.2.1.tar.gz (5.7 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page