Skip to main content

Yandex.Tracker issue metrics exporter

Project description

Yandex.Tracker Exporter

Export issue metadata & agile metrics to OLAP data storage. Metrics based on issue changelog.

This is a working example of collecting metadata and metrics from Yandex.Tracker for analytics, based on the first version of the internal private tool. You can fork this repository and refine the tool the way you want. Or use it as it is - this will allow you to build basic analytics on the tasks from Yandex.Tracker.

Self-hosted arch example

So, you can install Clickhouse with Clickhouse Proxy via Ansible role inside project.
Edit the inventory file ansible/inventory/hosts.yml and just run ansible-playbook.

Attention: For the role to work correctly, docker must be installed on the target server.

Example:

pip3 install -r requirements-dev.txt
cd ansible
ansible-playbook -i inventory/hosts.yml playbooks/clickhouse.yml --limit agile

Usage

Native

Install from source

# prepare virtual environment
python3 -m venv venv
source venv/bin/activate
python3 setup.py install

# configure environment variables
export EXPORTER_TRACKER_TOKEN="xxxx"
export EXPORTER_TRACKER_ORG_ID="123456"

export EXPORTER_TRACKER_ISSUES_SEARCH_RANGE="6h"
export EXPORTER_TRACKER_FETCH_INTERVAL=30

export EXPORTER_CLICKHOUSE_USER="default"
export EXPORTER_CLICKHOUSE_PASSWORD="strongpassword"
export EXPORTER_CLICKHOUSE_HOST="clickhouse01.example.com"
export EXPORTER_CLICKHOUSE_HTTP_PORT="8121"

export EXPORTER_LOGLEVEL="info"
export EXPORTER_ENABLE_UPLOAD=true

# run
tracker-exporter

Install from pypi

pip3 install tracker-exporter
tracker-exporter

Use .env file

Read about the settings here

tracker-exporter --env-file /home/akimrx/tracker/.settings

Docker

cd yandex-tracker-exporter
touch .env  # prepare the environment variables file (dotenv), like the example above
docker-compose up -d --build
docker logs tracker-exporter -f

Serverless arch example

Create a Managed Clickhouse cluster

How to: https://cloud.yandex.com/en/docs/managed-clickhouse/operations/cluster-create

  • Set user for exporter, example: agile
  • Set a database name, example: agile
  • Enable Serverless access flag
  • For testing enable host public access
  • Enable Access from the management console flag
  • Run migration or manual create tables (see migration block here, see sql)

Create Cloud Function

How to: https://cloud.yandex.com/en/docs/functions/quickstart/create-function/python-function-quickstart

  • Use Python >= 3.7
  • Copy/paste example content from examples/serverless (code)
  • Set entrypoint: main.handler (for code from examples)
  • Set function timeout to 600, because the launch can be long if there are a lot of updated issues during the collection period
  • Set memory to 512MB or more
  • Add environment variables (see variables block here)
    EXPORTER_TRACKER_TOKEN=XXXXXXXXXXXXXXXX
    EXPORTER_TRACKER_ORG_ID=123456
    EXPORTER_CLICKHOUSE_PROTO=https
    EXPORTER_CLICKHOUSE_CERT=/etc/ssl/certs/ca-certificates.crt
    EXPORTER_CLICKHOUSE_HTTP_PORT=8443
    EXPORTER_CLICKHOUSE_HOST=rc1b-xxxxxx.mdb.yandexcloud.net
    EXPORTER_CLICKHOUSE_DATABASE=agile
    EXPORTER_CLICKHOUSE_USER=agile
    EXPORTER_CLICKHOUSE_PASSWORD=xxxx
    EXPORTER_ENABLE_UPLOAD=true
    EXPORTER_ISSUES_SEARCH_INTERVAL=2h
    
  • Release function
  • Run test
  • See logs

Serverless database connection without public access

If you don't want to enable clickhouse public access, use service account with such permissions - serverless.mdbProxies.user and set environment variables below:

EXPORTER_CLICKHOUSE_HOST=akfd3bhqk3xxxxxxxxxxx.clickhouse-proxy.serverless.yandexcloud.net
EXPORTER_CLICKHOUSE_SERVERLESS_PROXY_ID=akfd3bhqk3xxxxxxxxxxxxx

How to create database connection: https://cloud.yandex.com/en/docs/functions/operations/database-connection

Also, the EXPORTER_CLICKHOUSE_PASSWORD variable with service account must be replaced by IAM-token. Keep this in mind. Probably, you should get it in the function code, because the IAM-token works for a limited period of time.

Create Trigger

How to: https://cloud.yandex.com/en/docs/functions/quickstart/create-trigger/timer-quickstart

  • Create new trigger
  • Choose type Timer
  • Set interval every hour: 0 * ? * * *
  • Select your function
  • Create serverless service account or use an existing one
  • Save trigger

Visualization

You can use any BI tool for visualization, for example:

  • Yandex DataLens
  • Apache Superset
  • PowerBI
  • Grafana

Migration

Based on go-migrate tool.

Download and install go-migrate tool

macOS

wget https://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.darwin-amd64.tar.gz -O migrate.tar.gz

tar xvf migrate.tar.gz
mv migrate ~/bin

Linux

wget https://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.linux-amd64.tar.gz -O migrate.tar.gz

tar -xvf migrate.tar.gz
mv migrate /usr/local/bin

Run migration

Example bash script

#!/usr/bin/env bash

set -Eeuo pipefail

CLICKHOUSE_HOST="localhost"
CLICKHOUSE_TCP_PORT=9000
CLICKHOUSE_HTTP_PORT=8123
CLICKHOUSE_USER="default"
CLICKHOUSE_PASSWORD="strongpassword"

MIGRATION_SOURCE_PATH="file://${PWD}/../migrations/clickhouse"
MIGRATION_HISTORY_TABLE="ci_gomigrate_migrations"
MIGRATION_DATABASE="agile"

MIGRATION_CLICKHOUSE_DSN="clickhouse://${CLICKHOUSE_HOST}:${CLICKHOUSE_TCP_PORT}?username=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}&database=${MIGRATION_DATABASE}&x-multi-statement=true&x-migrations-table=${MIGRATION_HISTORY_TABLE}"

prepare_migration() {
    echo "CREATE DATABASE IF NOT EXISTS ${MIGRATION_DATABASE}" | \
        curl "http://${CLICKHOUSE_HOST}:${CLICKHOUSE_HTTP_PORT}/?user=${CLICKHOUSE_USER}&password=${CLICKHOUSE_PASSWORD}" --data-binary @-

}

run_migration() {
    migrate -verbose \
        -source $MIGRATION_SOURCE_PATH \
        -database $MIGRATION_CLICKHOUSE_DSN \
        up

}

prepare_migration
run_migration

Environment variables (settings)

variable require? default description
EXPORTER_LOGLEVEL info One of: debug, info, warning, error, exception
EXPORTER_ENABLE_UPLOAD false Enable/disable upload to Clickhouse storage
EXPORTER_MONITORING_ENABLED false Enable send statsd metrics
EXPORTER_MONITORING_HOST localhost Monitoring statsd hostname
EXPORTER_MONITORING_PORT 8125 Monitoring statsd UDP port
EXPORTER_MONITORING_PREFIX tracker_exporter Prefix for all sent metrics, i.e.: {prefix}_{metric_name}
EXPORTER_SENTRY_ENABLED false Send exceptions and errors to Sentry
EXPORTER_SENTRY_DSN None Sentry DSN like https://{id}@{sentry_url}
EXPORTER_TRACKER_TOKEN None Yandex.Tracker OAuth token
EXPORTER_TRACKER_ORG_ID None Yandex.Tracker organization ID for Yandex.Tracker
EXPORTER_TRACKER_ISSUES_SEARCH_RANGE 4h The query search range for recently updated issues, i.e: Updated >= now() - {VARIABLE}
EXPORTER_TRACKER_ISSUES_SEARCH_QUERY None The query search string like Queue: SRE and status: closed
EXPORTER_TRACKER_ISSUES_FETCH_INTERVAL 120 Exporter job run interval in minutes for issue and metrics
EXPORTER_CLOSED_ISSUES_STATUSES closed,rejected,released,resolved,cancelled Lowercase comma-separated status, which will be flagged as is_closed
EXPORTER_CLICKHOUSE_PROTO http Clickhouse protocol - HTTP or HTTPS
EXPORTER_CLICKHOUSE_HOST localhost Clickhouse hostname
EXPORTER_CLICKHOUSE_HTTP_PORT 8123 Clickhouse HTTP(S) port
EXPORTER_CLICKHOUSE_USER default Clickhouse read-write username
EXPORTER_CLICKHOUSE_PASSWORD None Clickhouse user password. If your clickhouse/user can work without password just ignore this variable.
EXPORTER_CLICKHOUSE_CACERT_PATH None Path to CA certificate. Only for HTTPS
EXPORTER_CLICKHOUSE_SERVERLESS_PROXY_ID None Database connection ID. Only for serverless
EXPORTER_CLICKHOUSE_DATABASE agile Database for exporter CH tables
EXPORTER_CLICKHOUSE_ISSUES_TABLE issues Table when store issues metadata
EXPORTER_CLICKHOUSE_ISSUE_METRICS_TABLE issue_metrics Table when store issue metrics
EXPORTER_WORKHOURS_START 9 The beginning of working hours for calculating business hour metrics
EXPORTER_WORKHOURS_END 22 The end of working hours for calculating business hour metrics

Monitoring

Metric name Metric type Labels Description
tracker_exporter_clickhouse_insert_time_seconds time project Insert query time
tracker_exporter_clickhouse_optimize_time_seconds time project Optimize query time
tracker_exporter_clickhouse_inserted_rows gauge project, database, table Inserted rows to Clickhouse from last update
tracker_exporter_cycle_time_total_processing_time_seconds time project Total issues processing time
tracker_exporter_issue_transform_time_seconds time project Time of transformation of one issue into an object
tracker_exporter_issues_search_time_seconds time project Yandex.Tracker search time
tracker_exporter_issues_processing_time_seconds time project Time of transformation of batch issues into objects
tracker_exporter_issues_total_processed_count count project, source Processed issues from Yandex.Tracker
tracker_exporter_issues_without_metrics gauge project Issues with empty metrics
tracker_exporter_upload_status gauge project Status of data upload to storage
tracker_exporter_last_update_timestamp timestamp gauge project Timestamp of the last data upload to the storage

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tracker-exporter-0.1.19.tar.gz (21.6 kB view hashes)

Uploaded Source

Built Distribution

tracker_exporter-0.1.19-py3-none-any.whl (21.4 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page