tracker-exporter 0.1.19

Yandex.Tracker issue metrics exporter

Yandex.Tracker Exporter

Export issue metadata & agile metrics to OLAP data storage. Metrics based on issue changelog.

This is a working example of collecting metadata and metrics from Yandex.Tracker for analytics, based on the first version of the internal private tool. You can fork this repository and refine the tool the way you want. Or use it as it is - this will allow you to build basic analytics on the tasks from Yandex.Tracker.

Self-hosted arch example

So, you can install Clickhouse with Clickhouse Proxy via Ansible role inside project.
Edit the inventory file ansible/inventory/hosts.yml and just run ansible-playbook.

Attention: For the role to work correctly, docker must be installed on the target server.

Example:

pip3 install -r requirements-dev.txt
cd ansible
ansible-playbook -i inventory/hosts.yml playbooks/clickhouse.yml --limit agile

Usage

Native

Install from source

# prepare virtual environment
python3 -m venv venv
source venv/bin/activate
python3 setup.py install

# configure environment variables
export EXPORTER_TRACKER_TOKEN="xxxx"
export EXPORTER_TRACKER_ORG_ID="123456"

export EXPORTER_TRACKER_ISSUES_SEARCH_RANGE="6h"
export EXPORTER_TRACKER_FETCH_INTERVAL=30

export EXPORTER_CLICKHOUSE_USER="default"
export EXPORTER_CLICKHOUSE_PASSWORD="strongpassword"
export EXPORTER_CLICKHOUSE_HOST="clickhouse01.example.com"
export EXPORTER_CLICKHOUSE_HTTP_PORT="8121"

export EXPORTER_LOGLEVEL="info"
export EXPORTER_ENABLE_UPLOAD=true

# run
tracker-exporter

Install from pypi

pip3 install tracker-exporter
tracker-exporter

Use .env file

Read about the settings here

tracker-exporter --env-file /home/akimrx/tracker/.settings

Docker

cd yandex-tracker-exporter
touch .env  # prepare the environment variables file (dotenv), like the example above
docker-compose up -d --build
docker logs tracker-exporter -f

Serverless arch example

Create a Managed Clickhouse cluster

How to: https://cloud.yandex.com/en/docs/managed-clickhouse/operations/cluster-create

• Set user for exporter, example: agile
• Set a database name, example: agile
• Enable Serverless access flag
• For testing enable host public access
• Enable Access from the management console flag
• Run migration or manual create tables (see migration block here, see sql)

Create Cloud Function

How to: https://cloud.yandex.com/en/docs/functions/quickstart/create-function/python-function-quickstart

• Use Python >= 3.7
• Copy/paste example content from examples/serverless (code)
• Set entrypoint: main.handler (for code from examples)
• Set function timeout to 600, because the launch can be long if there are a lot of updated issues during the collection period
• Set memory to 512MB or more
• Add environment variables (see variables block here)

  EXPORTER_TRACKER_TOKEN=XXXXXXXXXXXXXXXX
  EXPORTER_TRACKER_ORG_ID=123456
  EXPORTER_CLICKHOUSE_PROTO=https
  EXPORTER_CLICKHOUSE_CERT=/etc/ssl/certs/ca-certificates.crt
  EXPORTER_CLICKHOUSE_HTTP_PORT=8443
  EXPORTER_CLICKHOUSE_HOST=rc1b-xxxxxx.mdb.yandexcloud.net
  EXPORTER_CLICKHOUSE_DATABASE=agile
  EXPORTER_CLICKHOUSE_USER=agile
  EXPORTER_CLICKHOUSE_PASSWORD=xxxx
  EXPORTER_ENABLE_UPLOAD=true
  EXPORTER_ISSUES_SEARCH_INTERVAL=2h
  

• Release function
• Run test
• See logs

Serverless database connection without public access

If you don't want to enable clickhouse public access, use service account with such permissions - serverless.mdbProxies.user and set environment variables below:

EXPORTER_CLICKHOUSE_HOST=akfd3bhqk3xxxxxxxxxxx.clickhouse-proxy.serverless.yandexcloud.net
EXPORTER_CLICKHOUSE_SERVERLESS_PROXY_ID=akfd3bhqk3xxxxxxxxxxxxx

How to create database connection: https://cloud.yandex.com/en/docs/functions/operations/database-connection

Also, the EXPORTER_CLICKHOUSE_PASSWORD variable with service account must be replaced by IAM-token. Keep this in mind. Probably, you should get it in the function code, because the IAM-token works for a limited period of time.

Create Trigger

How to: https://cloud.yandex.com/en/docs/functions/quickstart/create-trigger/timer-quickstart

• Create new trigger
• Choose type Timer
• Set interval every hour: 0 * ? * * *
• Select your function
• Create serverless service account or use an existing one
• Save trigger

Visualization

You can use any BI tool for visualization, for example:

• Yandex DataLens
• Apache Superset
• PowerBI
• Grafana

Migration

Based on go-migrate tool.

Download and install go-migrate tool

macOS

wget https://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.darwin-amd64.tar.gz -O migrate.tar.gz

tar xvf migrate.tar.gz
mv migrate ~/bin

Linux

wget https://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.linux-amd64.tar.gz -O migrate.tar.gz

tar -xvf migrate.tar.gz
mv migrate /usr/local/bin

Run migration

Example bash script

#!/usr/bin/env bash

set -Eeuo pipefail

CLICKHOUSE_HOST="localhost"
CLICKHOUSE_TCP_PORT=9000
CLICKHOUSE_HTTP_PORT=8123
CLICKHOUSE_USER="default"
CLICKHOUSE_PASSWORD="strongpassword"

MIGRATION_SOURCE_PATH="file://${PWD}/../migrations/clickhouse"
MIGRATION_HISTORY_TABLE="ci_gomigrate_migrations"
MIGRATION_DATABASE="agile"

MIGRATION_CLICKHOUSE_DSN="clickhouse://${CLICKHOUSE_HOST}:${CLICKHOUSE_TCP_PORT}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${MIGRATION_DATABASE}&x-multi-statement=true&x-migrations-table=${MIGRATION_HISTORY_TABLE}"

prepare_migration() {
    echo "CREATE DATABASE IF NOT EXISTS ${MIGRATION_DATABASE}" | \
        curl "http://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?user=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}" --data-binary @-

}

run_migration() {
    migrate -verbose \
        -source $MIGRATION_SOURCE_PATH \
        -database $MIGRATION_CLICKHOUSE_DSN \
        up

}

prepare_migration
run_migration

Environment variables (settings)

variable	require?	default	description
EXPORTER_LOGLEVEL	❌	info	One of: debug, info, warning, error, exception
EXPORTER_ENABLE_UPLOAD	❌	false	Enable/disable upload to Clickhouse storage
EXPORTER_MONITORING_ENABLED	❌	false	Enable send statsd metrics
EXPORTER_MONITORING_HOST	❌	localhost	Monitoring statsd hostname
EXPORTER_MONITORING_PORT	❌	8125	Monitoring statsd UDP port
EXPORTER_MONITORING_PREFIX	❌	tracker_exporter	Prefix for all sent metrics, i.e.: {prefix}_{metric_name}
EXPORTER_SENTRY_ENABLED	❌	false	Send exceptions and errors to Sentry
EXPORTER_SENTRY_DSN	❌	None	Sentry DSN like https://{id}@{sentry_url}
EXPORTER_TRACKER_TOKEN	✅	None	Yandex.Tracker OAuth token
EXPORTER_TRACKER_ORG_ID	✅	None	Yandex.Tracker organization ID for Yandex.Tracker
EXPORTER_TRACKER_ISSUES_SEARCH_RANGE	❌	4h	The query search range for recently updated issues, i.e: Updated >= now() - {VARIABLE}
EXPORTER_TRACKER_ISSUES_SEARCH_QUERY	❌	None	The query search string like Queue: SRE and status: closed
EXPORTER_TRACKER_ISSUES_FETCH_INTERVAL	❌	120	Exporter job run interval in minutes for issue and metrics
EXPORTER_CLOSED_ISSUES_STATUSES	❌	closed,rejected,released,resolved,cancelled	Lowercase comma-separated status, which will be flagged as is_closed
EXPORTER_CLICKHOUSE_PROTO	❌	http	Clickhouse protocol - HTTP or HTTPS
EXPORTER_CLICKHOUSE_HOST	❌	localhost	Clickhouse hostname
EXPORTER_CLICKHOUSE_HTTP_PORT	❌	8123	Clickhouse HTTP(S) port
EXPORTER_CLICKHOUSE_USER	❌	default	Clickhouse read-write username
EXPORTER_CLICKHOUSE_PASSWORD	✅	None	Clickhouse user password. If your clickhouse/user can work without password just ignore this variable.
EXPORTER_CLICKHOUSE_CACERT_PATH	❌	None	Path to CA certificate. Only for HTTPS
EXPORTER_CLICKHOUSE_SERVERLESS_PROXY_ID	❌	None	Database connection ID. Only for serverless
EXPORTER_CLICKHOUSE_DATABASE	❌	agile	Database for exporter CH tables
EXPORTER_CLICKHOUSE_ISSUES_TABLE	❌	issues	Table when store issues metadata
EXPORTER_CLICKHOUSE_ISSUE_METRICS_TABLE	❌	issue_metrics	Table when store issue metrics
EXPORTER_WORKHOURS_START	❌	9	The beginning of working hours for calculating business hour metrics
EXPORTER_WORKHOURS_END	❌	22	The end of working hours for calculating business hour metrics

Monitoring

Metric name	Metric type	Labels	Description
tracker_exporter_clickhouse_insert_time_seconds	time	project	Insert query time
tracker_exporter_clickhouse_optimize_time_seconds	time	project	Optimize query time
tracker_exporter_clickhouse_inserted_rows	gauge	project, database, table	Inserted rows to Clickhouse from last update
tracker_exporter_cycle_time_total_processing_time_seconds	time	project	Total issues processing time
tracker_exporter_issue_transform_time_seconds	time	project	Time of transformation of one issue into an object
tracker_exporter_issues_search_time_seconds	time	project	Yandex.Tracker search time
tracker_exporter_issues_processing_time_seconds	time	project	Time of transformation of batch issues into objects
tracker_exporter_issues_total_processed_count	count	project, source	Processed issues from Yandex.Tracker
tracker_exporter_issues_without_metrics	gauge	project	Issues with empty metrics
tracker_exporter_upload_status	gauge	project	Status of data upload to storage
tracker_exporter_last_update_timestamp	timestamp gauge	project	Timestamp of the last data upload to the storage