Lightweight time-series recording.
Project description
Trendlines
A simple, lightweight alternative to Graphite.
What is Trendlines?
Trendlines is a tool for passively collecting and displaying time-based or sequential numeric data. Built on Flask, PeeWee, and Plotly, it provides a simple interface for adding and viewing your metrics.
It was created for a few reasons:
- Graphite, while very awesome and powerful, does not provide a TCP way of sending in data - only UDP. Trendlines aims to provide both.
- Graphite also does not allow you to view data using a simple sequential x-axis - it only uses time for the x-axis. Again, Trendlines allows you to use both.
- It's always fun to start new projects.
Trendlines only accepts data that gets sent to it. It does not actively seek out data like many other monitoring programs out there (Zabbix, Nagios, Prometheus, etc.).
What can I use it for?
Anything, really. Well, anything that (a) you can assign a number to and (b) might change over time.
Examples include:
- Test coverage per project
- Code quality metrics per commit
- House temperature
- Distance driven per trip
- Distance per fillup (or per charge for the eco-friendly folk)
- How many times the dog barks
- How often some clicks the Big Red Button on your webpage
It's been designed to handle infrequent, variable interval data. Sometimes real-world data just doesn't appear at nice, regular intervals.
Installing
Docker
The easiest way to get up and running with Trendlines is with Docker:
$ docker run p 5000:80 dougthor42/trendlines:latest
Then open a browser to http://localhost:5000/
and you're all set. Add some data
(see below) and refresh the page.
Note: Data will not persist when the container is destroyed!
Docker Compose
WIP!
If you're doing more than just playing around, you'll likely want to set up Docker Compose. I've included an example Compose file here.
Before using the example Compose file, you'll need to:
- Make a directory to store the config file (and database file if using SQLite)
- Make sure that dir is writable by docker-compose.
- Create the configuration
trendlines.cfg
within that directory.
$ mkdir /var/www/trendlines
$ chmod -R a+w /var/www/trendlines
$ touch /var/www/trendlines/trendlines.cfg
Next, edit your new trendlines.cfg
file as needed. At the very least, the
following is needed:
# /var/www/trendlines/trendlines.cfg
SECRET_KEY = b"don't use the value written in this README file!"
DATABASE = "/data/internal.db"
You should be all set to bring Docker Compose up:
$ docker-compose -f path/to/docker-compose.yml up -d
Again, open up a browser to http://localhost
and you're good to go. Add some
data as outlined below and start playing around.
Note: If you get an error complaining about "Error starting userland proxy: Bind for 0.0.0.0:80: Unexpected error Permission denied", then try changing the port in docker-compose.yml to something else. I like 5000 myself:
ports:
- 5000:80
Running behind a proxy
A typical case, for me at least, is adding this application to a server that's already running Apache for other things. In this case, make the following adjustments:
- Add a proxy to the
VirtualHost
in your apache config. - Make sure to set the
URL_PREFIX
variable in your Trendlines config file.
# /etc/apache2/sites-enabled/your-site.conf
<VirtualHost *:80>
...
# optionally replace all instances of "trendlines" with whatever you want
# Make sure the port on ProxyPass and ProxyPassReverse matches what is
# exposed in your docker-compose.yml file.
<Location /trendlines>
ProxyPreserveHost On
ProxyPass http://0.0.0.0:5082/trendlines
ProxyPassReverse http://0.0.0.0:5082/trendlines
RequestHeader set X-Forwarded-Port 80
</Location>
</VirtualHost>
# /var/www/trendlines/trendlines.cfg
URL_PREFIX = "/trendlines" # Whatever you put in your Apache proxy
Running with Celery
Add the following services to your docker-compose.yml
:
redis:
image: redis
ports:
- "6379:6379"
celery:
image: dougthor42/trendlines:latest
ports:
- "9999:9999"
volumes:
# should be the same as what's in the 'trendlines' service
- type: bind
source: /var/www/trendlines
target: /data
command: celery worker -l info -A trendlines.celery_app.celery
Usage
TODO.
Sending Data
Trendlines accepts 2 forms of data entry: TCP and UDP. TCP data entry is done via HTTP POST with a JSON payload. UDP data entry is done in a similar manner as Graphite (though not yet implemented).
To send in data using the HTTP POST method:
curl --data '{"metric": "foo.bar", "value": 52.88, "time": '${date +%s}'}' \
--header "Content-Type: application/json" \
--request POST \
http://$SERVER/api/v1/data`
Or via UDP:
echo "foo.bar 52.88 `date +%s`" | nc $SERVER $PORT
The UDP string must follow the format metric_name value timestamp
. This
is the same format that Graphite uses for their plaintext
protocol,
so it is easy to switch from Trendlines to Graphite.
Viewing Data
Trendlines creates a simple set of web pages. A landing page at
http://$SERVER
provides a list of all known metrics. Clicking on a known
metric shows you the graph of historical data.
You can also export the data as JSON by sending an HTTP GET request:
curl http://$SERVER/api/v1/data/$METRIC_NAME
Development
Assumptions:
- Docker and Docker-compose are installed and up-to-date
- Python 3.6 or higher
Running with scripts
Make sure the config/localhost.cfg
file exists:
# ./config/localhost.cfg
DEBUG = True
DATABASE = "./internal.db"
TRENDLINES_API_URL = "http://localhost:5000/api/v1/data"
broker_url = "redis://localhost"
In 3 separate shells, run these 3 commands in order
docker run -p 6379:6379 redis
python runserver.py
python runworker.py
You can also optionally run each in the background (-d
for docker and
&
for the others), but personally I like to see the logs scroll by.
From a 4th shell, send data:
echo "metric.name 12.345 `date +%s`" | nc localhost 9999
And view the data by opening http://localhost:5000
in your browser.
Running with Docker-Compose
The default configuration assumes running within docker-compose. If you need
different settings, create config/trendlines.cfg
and add your variables
to it.
Build the images and bring up the stack:
$ docker-compose -f docker-build.yml build
$ docker-compose -f docker-build.yml up
Send data in
echo "metric.name 12.345 `date +%s`" | nc localhost 9999
And view the data by opening http://localhost:5000
in your browser.
Building Docker image
This is handled in CI, but in case you need to do it manually:
docker build -f docker/Dockerfile -t trendlines:latest -t dougthor42/trendlines:latest .
docker push dougthor42/trendlines:latest
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for trendlines-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb2d55c955e15996512a0f3dd8f571f136d9aa650bd17e2486962e7b0926477f |
|
MD5 | 4fd6b70968e95f3f7799a609ffaf4b69 |
|
BLAKE2b-256 | 3d09d4d8d1fa57330b0b9db78f4112ed1be3462ac776fd257a4d55449849e8e0 |