Tusk for Clickhouse metric writing
Project description
Tusk Clickhouse Metric Writer
contek-tusk
is a metric writer built upon Clickhouse.
1. Connection Initialization
import contek_tusk as tusk tusk.init(host='localhost', user='tester', password='P@assw0rd')
2. Generic Metric
from contek_tusk import metric cpu_usage = metric(table='system.cpu') cpu_usage.write({ 'region': 'Tokyo', 'instance_id': 'ABC', 'usage': 0.43 }) cpu_usage.write({ 'region': 'Seoul', 'instance_id': 'XYZ', 'usage': 0.22 })
SELECT * FROM system.cpu
region | instance_id | datetime | usage |
---|---|---|---|
Tokyo | ABC | 2022-03-10 06:42:42.746 | 0.43 |
Seoul | XYZ | 2022-03-10 06:42:42.746 | 0.22 |
Note: the client will automatically use the first DateTime
column to record current timestamp. Alternatively, it
can be explicitly specified:
from contek_tusk import metric from contek_tusk.table import Table table = Table(database='system', table_name='cpu', time_column='datetime') cpu_usage = metric(table)
3. Summing Counter
from contek_tusk.counter import counter counter = counter(table='download.quota_usage') counter.count({ 'resource_id': 'ABC111', 'requester_ip': '1.2.3.4' }) # default count is 1 counter.count({ 'resource_id': 'ABC122', 'requester_ip': '2.2.2.4' }, 3) counter.count({ 'resource_id': 'ABC111', 'requester_ip': '1.2.3.4' }, 4) # the new count is 1 + 4 = 5 counter.count({ 'resource_id': 'ABC122', 'requester_ip': '1.2.3.4' }, 2)
SELECT * FROM download.quota_usage
resource_id | requester_ip | datetime | count |
---|---|---|---|
ABC111 | 1.2.3.4 | 2022-03-10 06:42:48.411 | 5 |
ABC122 | 1.2.3.4 | 2022-03-10 06:42:48.411 | 2 |
ABC122 | 2.2.2.4 | 2022-03-10 06:42:48.411 | 3 |
Note: the client will automatically use the first Int
or UInt
column to record the count. Alternatively, it
can be explicitly specified:
from contek_tusk.counter import counter counter = counter(table='download.quota_usage', count_column='count')
4. Heartbeat Recorder
from contek_tusk.heartbeat import init init(table='maintenance.heartbeats', app_name='my_app')
After initialization, a recurring task starts in the background and sends a heartbeat to the database every 30 seconds. The heartbeat cycle can be explicitly specified:
from contek_tusk.heartbeat import init init(table='maintenance.heartbeats', app_name='my_app', heartbeat_period=15)
If the application is running multiple recurring tasks. Their heartbeats can also be registered:
from contek_tusk.heartbeat import beat beat(task='scan_disk', heartbeat_period=60)
SELECT * FROM maintenance.heartbeats
app | task | datetime | sequence |
---|---|---|---|
my_app | main | 2022-03-10 06:51:01.178 | 0 |
my_app | main | 2022-03-10 06:51:31.178 | 1 |
my_app | main | 2022-03-10 06:52:01.178 | 2 |
my_app | scan_disk | 2022-03-10 06:52:04.215 | 0 |
my_app | main | 2022-03-10 06:52:31.178 | 3 |
my_app | main | 2022-03-10 06:53:01.178 | 4 |
my_app | scan_disk | 2022-03-10 06:53:04.215 | 1 |
5.Logging
import logging from contek_tusk.logging import init logger = logging.getLogger(__name__) init(table='maintenance.logs', app_name='my_app') logger.info('Hello world!') logger.warning('Fire drill test.') logger.error('This is real fire.') try: raise ValueError('Guess what?') except ValueError: logger.exception('Oh no!') logger.critical('I give up.')
SELECT * FROM maintenance.logs
app | datetime | level | logger | line | error | message | stacktrace |
---|---|---|---|---|---|---|---|
my_app | 2022-03-10 08:53:03.558 | 4 | example.py | 7 | Hello world! | ||
my_app | 2022-03-10 08:53:03.558 | 3 | example.py | 8 | Fire drill test. | ||
my_app | 2022-03-10 08:53:03.558 | 2 | example.py | 9 | This is real fire. | ||
my_app | 2022-03-10 08:53:03.558 | 2 | example.py | 13 | ValueError | Oh no! | (traceback) |
my_app | 2022-03-10 08:53:03.558 | 1 | example.py | 14 | I give up. |
6. Full Initialization
Standard
import contek_tusk as tusk import contek_tusk.heartbeat as heartbeat import contek_tusk.logging as logging tusk.init(host='localhost', user='tester', password='P@assw0rd') tusk.set_app_name('my_app') heartbeat.init(table='maintenance.heartbeats') logging.init(table='maintenance.logs')
YAML
app: 'my_app' host: 'localhost' user: 'tester' password: 'P@assw0rd' heartbeat: database: 'maintenance' table: 'heartbeats' logging: database: 'maintenance' table: 'logs'
import contek_tusk.yaml as tusk_yaml tusk_yaml.for_yaml_file('example.yml')
7. Example Table Schema
maintenance.heartbeats
CREATE TABLE IF NOT EXISTS maintenance.heartbeats ( `app` String, `datetime` DateTime64(3), `task` String, `sequence` UInt32, `expiry` DateTime64 ) ENGINE = MergeTree() ORDER BY ( `app`, `datetime`, `task`, `sequence` ) TTL toDateTime(datetime) + INTERVAL 28 DAY DELETE;
maintenance.logs
CREATE TABLE IF NOT EXISTS maintenance.logs ( `app` String, `datetime` DateTime64(3), `level` Int8, `logger` String, `line` UInt32, `error` String, `message` String, `stacktrace` String ) ENGINE = MergeTree() ORDER BY ( `category`, `host`, `app`, `datetime`, `level`, `logger`, `line`, `error` ) TTL toDateTime(datetime) + INTERVAL 7 DAY DELETE;
8. Customization
To write to tables with different schemas. Specify the column names in TuskHeartbeatConfig
and
TuskLoggingConfig
.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for contek_tusk-1.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3befb8efc3d7f5db5f58b7b3ac9b5f2d59d7c61b44a2da5968fcfdf9cd21b40b |
|
MD5 | df33423bd8ab4e768f7abf2122dc21dd |
|
BLAKE2-256 | 1e05943a9fa1dcc172ef3401f0c529656cdf9beb06c65cdd6ec4782b079b418e |