trendlines 0.3.0

Lightweight time-series recording.

Trendlines

A simple, lightweight alternative to Graphite.

What is Trendlines?

Trendlines is a tool for passively collecting and displaying time-based or sequential numeric data. Built on Flask, PeeWee, and Plotly, it provides a simple interface for adding and viewing your metrics.

It was created for a few reasons:

1. Graphite, while very awesome and powerful, does not provide a TCP way of sending in data - only UDP. Trendlines aims to provide both.
2. Graphite also does not allow you to view data using a simple sequential x-axis - it only uses time for the x-axis. Again, Trendlines allows you to use both.
3. It's always fun to start new projects.

Trendlines only accepts data that gets sent to it. It does not actively seek out data like many other monitoring programs out there (Zabbix, Nagios, Prometheus, etc.).

What can I use it for?

Anything, really. Well, anything that (a) you can assign a number to and (b) might change over time.

Examples include:

• Test coverage per project
• Code quality metrics per commit
• House temperature
• Distance driven per trip
• Distance per fillup (or per charge for the eco-friendly folk)
• How many times the dog barks
• How often some clicks the Big Red Button on your webpage

It's been designed to handle infrequent, variable interval data. Sometimes real-world data just doesn't appear at nice, regular intervals.

Installing

Docker

The easiest way to get up and running with Trendlines is with Docker:

$ docker run p 5000:80 dougthor42/trendlines:latest

Then open a browser to http://localhost:5000/ and you're all set. Add some data (see below) and refresh the page.

Note: Data will not persist when the container is destroyed!

Docker Compose

WIP!

If you're doing more than just playing around, you'll likely want to set up Docker Compose. I've included an example Compose file here.

Before using the example Compose file, you'll need to:

1. Make a directory to store the config file (and database file if using SQLite)
2. Make sure that dir is writable by docker-compose.
3. Create the configuration trendlines.cfg within that directory.

$ mkdir /var/www/trendlines
$ chmod -R a+w /var/www/trendlines
$ touch /var/www/trendlines/trendlines.cfg

Next, edit your new trendlines.cfg file as needed. At the very least, the following is needed:

# /var/www/trendlines/trendlines.cfg
SECRET_KEY = b"don't use the value written in this README file!"
DATABASE = "/data/internal.db"

You should be all set to bring Docker Compose up:

$ docker-compose -f path/to/docker-compose.yml up -d

Again, open up a browser to http://localhost and you're good to go. Add some data as outlined below and start playing around.

Note: If you get an error complaining about "Error starting userland proxy: Bind for 0.0.0.0:80: Unexpected error Permission denied", then try changing the port in docker-compose.yml to something else. I like 5000 myself:

ports:
  - 5000:80

Running behind a proxy

A typical case, for me at least, is adding this application to a server that's already running Apache for other things. In this case, make the following adjustments:

1. Add a proxy to the VirtualHost in your apache config.
2. Make sure to set the URL_PREFIX variable in your Trendlines config file.

# /etc/apache2/sites-enabled/your-site.conf
<VirtualHost *:80>
...
    # optionally replace all instances of "trendlines" with whatever you want
    # Make sure the port on ProxyPass and ProxyPassReverse matches what is
    # exposed in your docker-compose.yml file.
    <Location /trendlines>
        ProxyPreserveHost On
        ProxyPass http://0.0.0.0:5082/trendlines
        ProxyPassReverse http://0.0.0.0:5082/trendlines

        RequestHeader set X-Forwarded-Port 80
    </Location>
</VirtualHost>

# /var/www/trendlines/trendlines.cfg
URL_PREFIX = "/trendlines"    # Whatever you put in your Apache proxy

Running with Celery

Add the following services to your docker-compose.yml:

redis:
  image: redis
  ports:
    - "6379:6379"
celery:
  image: dougthor42/trendlines:latest
  ports:
    - "9999:9999"
  volumes:
    # should be the same as what's in the 'trendlines' service
    - type: bind
      source: /var/www/trendlines
      target: /data
  command: celery worker -l info -A trendlines.celery_app.celery

Usage

TODO.

Sending Data

Trendlines accepts 2 forms of data entry: TCP and UDP. TCP data entry is done via HTTP POST with a JSON payload. UDP data entry is done in a similar manner as Graphite (though not yet implemented).

To send in data using the HTTP POST method:

curl --data '{"metric": "foo.bar", "value": 52.88, "time": '${date +%s}'}' \
     --header "Content-Type: application/json" \
     --request POST \
     http://$SERVER/api/v1/data`

Or via UDP:

echo "foo.bar 52.88 `date +%s`" | nc $SERVER $PORT

The UDP string must follow the format metric_name value timestamp. This is the same format that Graphite uses for their plaintext protocol, so it is easy to switch from Trendlines to Graphite.

Viewing Data

Trendlines creates a simple set of web pages. A landing page at http://$SERVER provides a list of all known metrics. Clicking on a known metric shows you the graph of historical data.

You can also export the data as JSON by sending an HTTP GET request:

curl http://$SERVER/api/v1/data/$METRIC_NAME

Development

Assumptions:

• Docker and Docker-compose are installed and up-to-date
• Python 3.6 or higher

Running with scripts

Make sure the config/localhost.cfg file exists:

# ./config/localhost.cfg
DEBUG = True
DATABASE = "./internal.db"
TRENDLINES_API_URL = "http://localhost:5000/api/v1/data"
broker_url = "redis://localhost"

In 3 separate shells, run these 3 commands in order

1. docker run -p 6379:6379 redis
2. python runserver.py
3. python runworker.py

You can also optionally run each in the background (-d for docker and & for the others), but personally I like to see the logs scroll by.

From a 4th shell, send data:

echo "metric.name 12.345 `date +%s`" | nc localhost 9999

And view the data by opening http://localhost:5000 in your browser.

Running with Docker-Compose

The default configuration assumes running within docker-compose. If you need different settings, create config/trendlines.cfg and add your variables to it.

Build the images and bring up the stack:

$ docker-compose -f docker-build.yml build
$ docker-compose -f docker-build.yml up

Send data in

echo "metric.name 12.345 `date +%s`" | nc localhost 9999

And view the data by opening http://localhost:5000 in your browser.

Building Docker image

This is handled in CI, but in case you need to do it manually:

docker build -f docker/Dockerfile -t trendlines:latest -t dougthor42/trendlines:latest .
docker push dougthor42/trendlines:latest