FIWARE Prometheus Publisher

This software (fiprom) allows to publish metrics generated by Ceilometer in Prometheus. It allows to modify/enrich metrics before publishing them.

This project started from the need of collecting metrics generated by the FIWARE node located in Vicenza that is running an old version of Ceilometer (v 2015.1.1). Newer version of Ceilometer have a Prometheus publisher distributed along with Ceilometer itself and, therefore, there is no need for this custom publisher (excepting for customizing metrics metadata).

Installation

Fiprom works as a Ceilometer plugin loaded at runtime. It must be installed as library in the same environment where Ceilometer services are installed. It can be done simply:

pip install fiware-prometheus-publisher

An alternative way is to install the component in a Python virtualenv. In this case, also Ceilometer must be installed and executed from the virtualenv.

export PBR_VERSION=0.10.0
pip install -U 'pbr==0.11.0' testrepository setuptools

wget https://github.com/openstack/ceilometer/archive/2015.1.1.zip && unzip 2015.1.1.zip
RUN cd ceilometer-2015.1.1 && pip install .

pip install fiware-prometheus-publisher

Alternatively, the Docker image fiware-prometheus-publisher can be used. The image runs a Ceilometer Collector service configured to use fiprom as storage backend.

Configuration

Fiprom load its configuration through the ceilometer.conf file in the [fiprom] section. These are the supported keys and their default values:

[fiprom]

# URL of the Push Gateway where metrics created will be publihsed
push_gateway = http://localhost:9091/metrics/job/fiprom/infrastructure/fiware

# Configuration file for the conversion of Ceilometer samples to Prometheus metrics
converter_conf_file =

# Customize the location of the labels cache
cache_file=/tmp/fiprom.cache

# Mapping file for names of VMs, hosts and users
names_file =

# Mapping file for tenant groups
tenant_group_file =

# If set, logs samples received and metrics created in this file
log_file =

# If true, do not  make the call to the Push Gateway
dryrun = False

In addition to this configuration, the following has to be added:

• to use as Ceilomter pipeline publisher, add the following in your pipeline.yaml :

  - publishers:
    - fiprom://
  

• to use as backend for the Ceilometer Collector, add the following to your ceilometer.conf

  [database]
  metering_connection=fiprom://
  # also the key connection can be used
  

Further configuration files

The basic configuration (see Configuration) activate fiprom, but to be useful it needs further configuration on how to convert metrics from Ceilometer to Prometheus.

Converter Conf file

This file (converter_conf_file) tells fiprom which metrics should be converted and how labels of each metric should be mapped. It is in YAML format and must follow this schema:

enabled:
  - '<sample_name_pattern_1>'
  - '<sample_name_pattern_2>'
  - ...
labels:
  - <sample_name_pattern_x>:
    - <label_name_1>: '<python_expression>'
    - <label_name_2>: '<python_expression>'
    - ...
  - <sample_name_pattern_y>:
  	- ...

A working configuration with detailed comments can be found at fiprom_converter_conf.yaml file.

Names file

This file (names_file) contains the mapping between IDs of OpenStack objects and their names. Since most of the metrics created by Ceilometer only contain IDs of objects, this mapping file is required to add label with names (more human readable) to Prometheus metrics.

The file must contain one mapping for each line in the format:

<id> <delimiter> <name>

'=' or ':' characters are used as delimiter if found in the row, otherwise space, tab or comma characters are used. All of these are valid lines:

acede503-78e4-49cf-adbc-bd87390d4f6a helloworld
4c9e370ae557689709a64e9b543d50fd0f1538d0c5a22ac30c5afb41,  host1.eng.it
f7eb45f96d62423a805bd2ee5734d961: admin
f00a0cba70174563b6deef58dffebda2  = This is a demo, project 

Tenant Group file

This file (tenant_group_file) is used to assign a group to each tenant. Fiprom will add a tenant_group label when a mapping is found. The file is in the format:

<tenant_id_or_name> <delimiter> <group>

Same rules on delimiters applies of the names file.