Scrape only relevant metrics in Prometheus, according to your Grafana dashboards
Project description
frigga
Do you have a Grafana instance? frigga makes sure you don’t scrape metrics in Prometheus, which you don’t present in Grafana dashboards.
Scrape only relevant metrics in Prometheus, according to your Grafana dashboards, see the before and after snapshot. frigga generates keep
filters on metric_relabel_configs, and adds them to your prometheus.yml
file
frigga is extremely useful for Grafana Cloud customers since the pricing is per DataSeries ingestions.
Illustration
Expand/Collapse
Requirements
Python 3.6.7+
Installation
$ pip install frigga
Available Commands
Auto-generated by unfor19/replacer-action, see readme.yml
Usage: frigga [OPTIONS] COMMAND [ARGS]...
No confirmation prompts
Options:
-ci, --ci Use this flag to avoid confirmation prompts
--help Show this message and exit.
Commands:
grafana-list Alias: gl Provide Grafana URL and Grafana API Key...
prometheus-apply Alias: pa Applies .metrics.json for a given...
Getting Started
-
Grafana - Import the dashboard frigga - Jobs Usage (ID: 12537) to Grafana, and check out the number of DataSeries
-
Grafana - Generate an API Key for
Viewer
-
frigga - Get the list of metrics that are used in your Grafana dasboards
$ frigga gl # gl is grafana-list, or good luck :) Grafana url [http://localhost:3000]: http://my-grafana.grafana.net Grafana api key: (hidden) >> [LOG] Getting the list of words to ignore when scraping from Grafana ... >> [LOG] Found a total of 269 unique metrics to keep
.metrics.json
- automatically generated in pwd{ "all_metrics": [ "cadvisor_version_info", "container_cpu_usage_seconds_total", "container_last_seen", "container_memory_max_usage_bytes", ... ] }
-
Add the following snippet to the bottom of your
prometheus.yml
file. Check the example in docker-compose/prometheus-original.yml--- name: frigga exclude_jobs: []
-
frigga - Use the
.metrics.json
file to apply the rules to your existingprometheus.yml
$ frigga pa # pa is prometheus-apply, or pam-tada-dam Prom yaml path [docker-compose/prometheus.yml]: /etc/prometheus/prometheus.yml Metrics json path [./.metrics.json]: /home/willywonka/.metrics.json >> [LOG] Reading documents from docker-compose/prometheus.yml ... >> [LOG] Done! Now reload docker-compose/prometheus.yml with 'docker exec $PROM_CONTAINER_NAME kill -HUP 1'
-
As mentioned in the previous step, reload the
prometheus.yml
to Prometheus, here are two ways of doing it- "Kill" Prometheus
$ docker exec $PROM_CONTAINER_NAME kill -HUP 1
- Send a POST request to
/-/reload
- this requires Prometheus to be loaded with--web.enable-lifecycle
, for example, see docker-compose.yml$ curl -X POST http://localhost:9090/-/reload
- "Kill" Prometheus
-
Make sure the
prometheus.yml
was loaded successfully to Prometheus$ docker logs --tail 10 $PROM_CONTAINER_NAME level=info ts=2020-06-27T15:45:34.514Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml level=info ts=2020-06-27T15:45:34.686Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
-
Grafana - Now check
frigga - Jobs Usage
dashboard, the numbers should be signifcantly lower (up to 60% or even more)
Test it locally
Requirements
Getting Started
-
git clone this repository
-
Run Docker daemon (Docker for Desktop)
-
Make sure port 8080 is not in use
-
Deploy locally the services: Prometheus, Grafana, node-exporter and cadvisor
$ bash docker-compose/deploy_stack.sh Creating network frigga_net1 ... >> Grafana - Generating API Key - for Viewer eyJrIjoiT29hNGxGZjAwT2hZcU1BSmpPRXhndXVwUUE4ZVNFcGQiLCJuIjoibG9jYWwiLCJpZCI6MX0= # Save this key ^^^
-
Open your browser, navigate to http://localhost:3000
- Username and password are admin:admin
- You'll be prompted to update your password, so just keep using
admin
or hit Skip
-
Go to Jobs Usage dashboard, you'll see that Prometheus is processing ~2800 DataSeries
-
Get all the metrics that are used in your Grafana dasboards
$ export GRAFANA_API_KEY=the-key-that-was-generated-in-the-deploy-locally-step $ frigga gl -gurl http://localhost:3000 -gkey $GRAFANA_API_KEY >> [LOG] Getting the list of words to ignore when scraping from Grafana ... >> [LOG] Found a total of 269 unique metrics to keep # Generated .metrics.json in pwd
-
Apply the rules to
prometheus.yml
, keep the defaults$ frigga pa # prometheus-apply Prom yaml path [docker-compose/prometheus.yml]: Metrics json path [./.metrics.json]: ... >> [LOG] Done! Now reload docker-compose/prometheus.yml with 'docker exec $PROM_CONTAINER_NAME kill -HUP 1'
-
Reload
prometheus.yml
to Prometheus$ bash docker-compose/reload_prom_config.sh show >> Reloading prometheus.yml configuration file ... level=info ts=2020-06-27T16:25:17.656Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml
-
Go to Jobs Usage, you'll see that Prometheus is processing only ~1000 DataSeries (previously ~2800)
- In case you don't see the change, don't forget to hit the refersh button
-
Cleanup
$ docker-compose -p frigga --file docker-compose/docker-compose.yml down
Pros and Cons of this tool
Pros
- Grafana-Cloud - As a Grafana Cloud customer, the main reason for writing this tool was lowering the costs. This goal was achieved by sending only the relevant DataSeries to Grafana Cloud
- Saves disk-space on the machine running Prometheus
- Improves PromQL performance by querying less metrics; significant only when processing high volumes
Cons
- After applying the rules in
prometheus.yml
, it makes the file less readable. Due to the fact it's not a file that you play with on a daily basis, it's okayish - The memory usage of Prometheus increases slightly, around ~30MB, not significant, but worth mentioning
- If you intend to use more metrics, for example, you've added a new dashboard which uses more metrics, you'll need to do the same process again;
frigga gl
andfrigga pa
References
Contributing
Report issues/questions/feature requests on the Issues section.
Pull requests are welcome! Ideally, create a feature branch and issue for every single change you make. These are the steps:
- Fork this repo
- Create your feature branch from master (
git checkout -b my-new-feature
) - Install from source
$ git clone https://github.com/${GITHUB_OWNER}/frigga.git && cd frigga ... $ pip install --upgrade pip ... $ python -m venv ./ENV $ . ./ENV/bin/activate ... $ (ENV) pip install --editable . ... # Done! Now when you run 'frigga' it will get automatically updated when you modify the code
- Add the code of your new feature
- Test - make sure
frigga grafana-list
andfrigga prometheus-apply
commands work - Commit your remarkable changes (
git commit -am 'Added new feature'
) - Push to the branch (
git push --set-up-stream origin my-new-feature
) - Create a new Pull Request and tell us about your changes
Authors
Created and maintained by Meir Gabay
License
This project is licensed under the MIT License - see the LICENSE file for details
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for frigga-1.0.10rc2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c1d7309f751c67d4d89976949faf13fec58588165b308f6290e6591c7c9a72b3 |
|
MD5 | 340fc6c81d8bc9ccb4774f6f892b33af |
|
BLAKE2b-256 | 4b5faf24dea7c79147b2225dcc799c7639698b8c3bf88ab852ff3c4431fcaa3e |