Skip to main content

Tusk for Clickhouse metric writing

Project description

Tusk Clickhouse Metric Writer

contek-tusk is a metric writer built upon Clickhouse.

1. Connection Initialization

import contek_tusk as tusk

tusk.init(host='localhost', user='tester', password='P@assw0rd')

2. Generic Metric

from contek_tusk import metric

cpu_usage = metric(table='system.cpu')

cpu_usage.write({
    'region': 'Tokyo',
    'instance_id': 'ABC',
    'usage': 0.43
})

cpu_usage.write({
    'region': 'Seoul',
    'instance_id': 'XYZ',
    'usage': 0.22
})
SELECT * FROM system.cpu
region instance_id datetime usage
Tokyo ABC 2022-03-10 06:42:42.746 0.43
Seoul XYZ 2022-03-10 06:42:42.746 0.22

Note: the client will automatically use the first DateTime column to record current timestamp. Alternatively, it can be explicitly specified:

from contek_tusk import metric
from contek_tusk.table import Table

table = Table(database='system', table_name='cpu', time_column='datetime')
cpu_usage = metric(table)

3. Summing Counter

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage')

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
})  # default count is 1

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '2.2.2.4'
}, 3)

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
}, 4)  # the new count is 1 + 4 = 5

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '1.2.3.4'
}, 2)
SELECT * FROM download.quota_usage
resource_id requester_ip datetime count
ABC111 1.2.3.4 2022-03-10 06:42:48.411 5
ABC122 1.2.3.4 2022-03-10 06:42:48.411 2
ABC122 2.2.2.4 2022-03-10 06:42:48.411 3

Note: the client will automatically use the first Int or UInt column to record the count. Alternatively, it can be explicitly specified:

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage', count_column='count')

4. Heartbeat Recorder

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app')

After initialization, a recurring task starts in the background and sends a heartbeat to the database every 30 seconds. The heartbeat cycle can be explicitly specified:

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app', heartbeat_period=15)

If the application is running multiple recurring tasks. Their heartbeats can also be registered:

from contek_tusk.heartbeat import beat

beat(task='scan_disk', heartbeat_period=60)
SELECT * FROM maintenance.heartbeats
app task datetime sequence
my_app main 2022-03-10 06:51:01.178 0
my_app main 2022-03-10 06:51:31.178 1
my_app main 2022-03-10 06:52:01.178 2
my_app scan_disk 2022-03-10 06:52:04.215 0
my_app main 2022-03-10 06:52:31.178 3
my_app main 2022-03-10 06:53:01.178 4
my_app scan_disk 2022-03-10 06:53:04.215 1

5.Logging

import logging
from contek_tusk.logging import init

logger = logging.getLogger(__name__)
init(table='maintenance.logs', app_name='my_app')

logger.info('Hello world!')
logger.warning('Fire drill test.')
logger.error('This is real fire.')
try:
    raise ValueError('Guess what?')
except ValueError:
    logger.exception('Oh no!')
logger.critical('I give up.')
SELECT * FROM maintenance.logs
app datetime level logger line error message stacktrace
my_app 2022-03-10 08:53:03.558 4 example.py 7 Hello world!
my_app 2022-03-10 08:53:03.558 3 example.py 8 Fire drill test.
my_app 2022-03-10 08:53:03.558 2 example.py 9 This is real fire.
my_app 2022-03-10 08:53:03.558 2 example.py 13 ValueError Oh no! (traceback)
my_app 2022-03-10 08:53:03.558 1 example.py 14 I give up.

6. Full Initialization

Standard

import contek_tusk as tusk
import contek_tusk.heartbeat as heartbeat
import contek_tusk.logging as logging

tusk.init(host='localhost', user='tester', password='P@assw0rd')
tusk.set_app_name('my_app')
heartbeat.init(table='maintenance.heartbeats')
logging.init(table='maintenance.logs')

YAML

app: 'my_app'
host: 'localhost'
user: 'tester'
password: 'P@assw0rd'
heartbeat:
  database: 'maintenance'
  table: 'heartbeats'
logging:
  database: 'maintenance'
  table: 'logs'
import contek_tusk.yaml as tusk_yaml

tusk_yaml.for_yaml_file('example.yml')

7. Example Table Schema

maintenance.heartbeats

CREATE TABLE IF NOT EXISTS maintenance.heartbeats
(
    `app` String, 
    `datetime` DateTime64(3),
    `task` String,
    `sequence` UInt32,
    `expiry` DateTime64
)
ENGINE = MergeTree()
ORDER BY
(
    `app`,
    `datetime`,
    `task`,
    `sequence`
)
TTL toDateTime(datetime) + INTERVAL 28 DAY DELETE;

maintenance.logs

CREATE TABLE IF NOT EXISTS maintenance.logs
(
    `app` String,
    `datetime` DateTime64(3),
    `level` Int8,
    `logger` String,
    `line` UInt32,
    `error` String,
    `message` String,
    `stacktrace` String
)
ENGINE = MergeTree()
ORDER BY
(
    `category`,
    `host`,
    `app`,
    `datetime`,
    `level`,
    `logger`,
    `line`,
    `error`
)
TTL toDateTime(datetime) + INTERVAL 7 DAY DELETE;

8. Customization

To write to tables with different schemas. Specify the column names in TuskHeartbeatConfig and TuskLoggingConfig.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

contek-tusk-1.0.tar.gz (16.2 kB view hashes)

Uploaded Source

Built Distribution

contek_tusk-1.0-py3-none-any.whl (22.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page