Tusk for Clickhouse metric writing
Project description
Tusk Clickhouse Metric Writer
contek-tusk
is a metric writer built upon Clickhouse.
1. Connection Initialization
import contek_tusk as tusk
tusk.init(host='localhost', user='tester', password='P@assw0rd')
2. Generic Metric
from contek_tusk import metric
cpu_usage = metric(table='system.cpu')
cpu_usage.write({
'region': 'Tokyo',
'instance_id': 'ABC',
'usage': 0.43
})
cpu_usage.write({
'region': 'Seoul',
'instance_id': 'XYZ',
'usage': 0.22
})
SELECT * FROM system.cpu
region | instance_id | datetime | usage |
---|---|---|---|
Tokyo | ABC | 2022-03-10 06:42:42.746 | 0.43 |
Seoul | XYZ | 2022-03-10 06:42:42.746 | 0.22 |
Note: the client will automatically use the first DateTime
column to record current timestamp. Alternatively, it
can be explicitly specified:
from contek_tusk import metric
from contek_tusk.table import Table
table = Table(database='system', table_name='cpu', time_column='datetime')
cpu_usage = metric(table)
3. Summing Counter
from contek_tusk.counter import counter
counter = counter(table='download.quota_usage')
counter.count({
'resource_id': 'ABC111',
'requester_ip': '1.2.3.4'
}) # default count is 1
counter.count({
'resource_id': 'ABC122',
'requester_ip': '2.2.2.4'
}, 3)
counter.count({
'resource_id': 'ABC111',
'requester_ip': '1.2.3.4'
}, 4) # the new count is 1 + 4 = 5
counter.count({
'resource_id': 'ABC122',
'requester_ip': '1.2.3.4'
}, 2)
SELECT * FROM download.quota_usage
resource_id | requester_ip | datetime | count |
---|---|---|---|
ABC111 | 1.2.3.4 | 2022-03-10 06:42:48.411 | 5 |
ABC122 | 1.2.3.4 | 2022-03-10 06:42:48.411 | 2 |
ABC122 | 2.2.2.4 | 2022-03-10 06:42:48.411 | 3 |
Note: the client will automatically use the first Int
or UInt
column to record the count. Alternatively, it
can be explicitly specified:
from contek_tusk.counter import counter
counter = counter(table='download.quota_usage', count_column='count')
4. Heartbeat Recorder
from contek_tusk.heartbeat import init
init(table='maintenance.heartbeats', app_name='my_app')
After initialization, a recurring task starts in the background and sends a heartbeat to the database every 30 seconds. The heartbeat cycle can be explicitly specified:
from contek_tusk.heartbeat import init
init(table='maintenance.heartbeats', app_name='my_app', heartbeat_period=15)
If the application is running multiple recurring tasks. Their heartbeats can also be registered:
from contek_tusk.heartbeat import beat
beat(task='scan_disk', heartbeat_period=60)
SELECT * FROM maintenance.heartbeats
app | task | datetime | sequence |
---|---|---|---|
my_app | main | 2022-03-10 06:51:01.178 | 0 |
my_app | main | 2022-03-10 06:51:31.178 | 1 |
my_app | main | 2022-03-10 06:52:01.178 | 2 |
my_app | scan_disk | 2022-03-10 06:52:04.215 | 0 |
my_app | main | 2022-03-10 06:52:31.178 | 3 |
my_app | main | 2022-03-10 06:53:01.178 | 4 |
my_app | scan_disk | 2022-03-10 06:53:04.215 | 1 |
5.Logging
import logging
from contek_tusk.logging import init
logger = logging.getLogger(__name__)
init(table='maintenance.logs', app_name='my_app')
logger.info('Hello world!')
logger.warning('Fire drill test.')
logger.error('This is real fire.')
try:
raise ValueError('Guess what?')
except ValueError:
logger.exception('Oh no!')
logger.critical('I give up.')
SELECT * FROM maintenance.logs
app | datetime | level | logger | line | error | message | stacktrace |
---|---|---|---|---|---|---|---|
my_app | 2022-03-10 08:53:03.558 | 4 | example.py | 7 | Hello world! | ||
my_app | 2022-03-10 08:53:03.558 | 3 | example.py | 8 | Fire drill test. | ||
my_app | 2022-03-10 08:53:03.558 | 2 | example.py | 9 | This is real fire. | ||
my_app | 2022-03-10 08:53:03.558 | 2 | example.py | 13 | ValueError | Oh no! | (traceback) |
my_app | 2022-03-10 08:53:03.558 | 1 | example.py | 14 | I give up. |
6. Full Initialization
Standard
import contek_tusk as tusk
import contek_tusk.heartbeat as heartbeat
import contek_tusk.logging as logging
tusk.init(host='localhost', user='tester', password='P@assw0rd')
tusk.set_app_name('my_app')
heartbeat.init(table='maintenance.heartbeats')
logging.init(table='maintenance.logs')
YAML
app: 'my_app'
host: 'localhost'
user: 'tester'
password: 'P@assw0rd'
heartbeat:
database: 'maintenance'
table: 'heartbeats'
logging:
database: 'maintenance'
table: 'logs'
import contek_tusk.yaml as tusk_yaml
tusk_yaml.for_yaml_file('example.yml')
7. Example Table Schema
maintenance.heartbeats
CREATE TABLE IF NOT EXISTS maintenance.heartbeats
(
`app` String,
`datetime` DateTime64(3),
`task` String,
`sequence` UInt32,
`expiry` DateTime64
)
ENGINE = MergeTree()
ORDER BY
(
`app`,
`datetime`,
`task`,
`sequence`
)
TTL toDateTime(datetime) + INTERVAL 28 DAY DELETE;
maintenance.logs
CREATE TABLE IF NOT EXISTS maintenance.logs
(
`app` String,
`datetime` DateTime64(3),
`level` Int8,
`logger` String,
`line` UInt32,
`error` String,
`message` String,
`stacktrace` String
)
ENGINE = MergeTree()
ORDER BY
(
`category`,
`host`,
`app`,
`datetime`,
`level`,
`logger`,
`line`,
`error`
)
TTL toDateTime(datetime) + INTERVAL 7 DAY DELETE;
8. Customization
To write to tables with different schemas. Specify the column names in TuskHeartbeatConfig
and
TuskLoggingConfig
.
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for contek_tusk-1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ac264af6a20bd34c5e561ca50f4b6d794cc3270fb2b280ed60c559bd9d5597d0 |
|
MD5 | 1b5e7c6155460d3900ebdbe25d6b9738 |
|
BLAKE2b-256 | b678d0383bf8ec226a736c67cb6084960b8473914351ae7f161a36185e30fae6 |