contek-tusk 2.1

Tusk for Clickhouse metric writing

Tusk Clickhouse Metric Writer

contek-tusk is a metric writer built upon Clickhouse.

1. Connection Initialization

import contek_tusk as tusk

tusk.init(host='localhost', user='tester', password='P@assw0rd')

2. Generic Metric

from contek_tusk import metric

cpu_usage = metric(table='system.cpu')

cpu_usage.write({
    'region': 'Tokyo',
    'instance_id': 'ABC',
    'usage': 0.43
})

cpu_usage.write({
    'region': 'Seoul',
    'instance_id': 'XYZ',
    'usage': 0.22
})

SELECT * FROM system.cpu

region	instance_id	datetime	usage
Tokyo	ABC	2022-03-10 06:42:42.746	0.43
Seoul	XYZ	2022-03-10 06:42:42.746	0.22

Note: the client will automatically use the first DateTime column to record current timestamp. Alternatively, it can be explicitly specified:

from contek_tusk import metric
from contek_tusk.table import Table

table = Table(database='system', table_name='cpu', time_column='datetime')
cpu_usage = metric(table)

3. Summing Counter

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage')

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
})  # default count is 1

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '2.2.2.4'
}, 3)

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
}, 4)  # the new count is 1 + 4 = 5

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '1.2.3.4'
}, 2)

SELECT * FROM download.quota_usage

resource_id	requester_ip	datetime	count
ABC111	1.2.3.4	2022-03-10 06:42:48.411	5
ABC122	1.2.3.4	2022-03-10 06:42:48.411	2
ABC122	2.2.2.4	2022-03-10 06:42:48.411	3

Note: the client will automatically use the first Int or UInt column to record the count. Alternatively, it can be explicitly specified:

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage', count_column='count')

4. Heartbeat Recorder

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app')

After initialization, a recurring task starts in the background and sends a heartbeat to the database every 30 seconds. The heartbeat cycle can be explicitly specified:

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app', heartbeat_period=15)

If the application is running multiple recurring tasks. Their heartbeats can also be registered:

from contek_tusk.heartbeat import beat

beat(task='scan_disk', heartbeat_period=60)

SELECT * FROM maintenance.heartbeats

app	task	datetime	sequence
my_app	main	2022-03-10 06:51:01.178	0
my_app	main	2022-03-10 06:51:31.178	1
my_app	main	2022-03-10 06:52:01.178	2
my_app	scan_disk	2022-03-10 06:52:04.215	0
my_app	main	2022-03-10 06:52:31.178	3
my_app	main	2022-03-10 06:53:01.178	4
my_app	scan_disk	2022-03-10 06:53:04.215	1

5.Logging

import logging
from contek_tusk.logging import init

logger = logging.getLogger(__name__)
init(table='maintenance.logs', app_name='my_app')

logger.info('Hello world!')
logger.warning('Fire drill test.')
logger.error('This is real fire.')
try:
    raise ValueError('Guess what?')
except ValueError:
    logger.exception('Oh no!')
logger.critical('I give up.')

SELECT * FROM maintenance.logs

app	datetime	level	logger	line	error	message	stacktrace
my_app	2022-03-10 08:53:03.558	4	example.py	7		Hello world!
my_app	2022-03-10 08:53:03.558	3	example.py	8		Fire drill test.
my_app	2022-03-10 08:53:03.558	2	example.py	9		This is real fire.
my_app	2022-03-10 08:53:03.558	2	example.py	13	ValueError	Oh no!	(traceback)
my_app	2022-03-10 08:53:03.558	1	example.py	14		I give up.

6. Full Initialization

Standard

import contek_tusk as tusk
import contek_tusk.heartbeat as heartbeat
import contek_tusk.logging as logging

tusk.init(host='localhost', user='tester', password='P@assw0rd')
tusk.set_app_name('my_app')
heartbeat.init(table='maintenance.heartbeats')
logging.init(table='maintenance.logs')

YAML

app: 'my_app'
host: 'localhost'
user: 'tester'
password: 'P@assw0rd'
heartbeat:
  database: 'maintenance'
  table: 'heartbeats'
logging:
  database: 'maintenance'
  table: 'logs'

import contek_tusk.yaml as tusk_yaml

tusk_yaml.for_yaml_file('example.yml')

7. Example Table Schema

maintenance.heartbeats

CREATE TABLE IF NOT EXISTS maintenance.heartbeats
(
    `app` String, 
    `datetime` DateTime64(3),
    `task` String,
    `sequence` UInt32,
    `expiry` DateTime64
)
ENGINE = MergeTree()
ORDER BY
(
    `app`,
    `datetime`,
    `task`,
    `sequence`
)
TTL toDateTime(datetime) + INTERVAL 28 DAY DELETE;

maintenance.logs

CREATE TABLE IF NOT EXISTS maintenance.logs
(
    `app` String,
    `datetime` DateTime64(3),
    `level` Int8,
    `logger` String,
    `line` UInt32,
    `error` String,
    `message` String,
    `stacktrace` String
)
ENGINE = MergeTree()
ORDER BY
(
    `category`,
    `host`,
    `app`,
    `datetime`,
    `level`,
    `logger`,
    `line`,
    `error`
)
TTL toDateTime(datetime) + INTERVAL 7 DAY DELETE;

8. Customization

To write to tables with different schemas. Specify the column names in TuskHeartbeatConfig and TuskLoggingConfig.