Skip to main content

Tusk for Clickhouse metric writing

Project description

Tusk Clickhouse Metric Writer

contek-tusk is a metric writer built upon Clickhouse.

1. Connection Initialization

import contek_tusk as tusk

tusk.init(host='localhost', user='tester', password='P@assw0rd')

2. Generic Metric

from contek_tusk import metric

cpu_usage = metric(table='system.cpu')

cpu_usage.write({
    'region': 'Tokyo',
    'instance_id': 'ABC',
    'usage': 0.43
})

cpu_usage.write({
    'region': 'Seoul',
    'instance_id': 'XYZ',
    'usage': 0.22
})
SELECT * FROM system.cpu
region instance_id datetime usage
Tokyo ABC 2022-03-10 06:42:42.746 0.43
Seoul XYZ 2022-03-10 06:42:42.746 0.22

Note: the client will automatically use the first DateTime column to record current timestamp. Alternatively, it can be explicitly specified:

from contek_tusk import metric
from contek_tusk.table import Table

table = Table(database='system', table_name='cpu', time_column='datetime')
cpu_usage = metric(table)

3. Summing Counter

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage')

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
})  # default count is 1

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '2.2.2.4'
}, 3)

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
}, 4)  # the new count is 1 + 4 = 5

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '1.2.3.4'
}, 2)
SELECT * FROM download.quota_usage
resource_id requester_ip datetime count
ABC111 1.2.3.4 2022-03-10 06:42:48.411 5
ABC122 1.2.3.4 2022-03-10 06:42:48.411 2
ABC122 2.2.2.4 2022-03-10 06:42:48.411 3

Note: the client will automatically use the first Int or UInt column to record the count. Alternatively, it can be explicitly specified:

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage', count_column='count')

4. Heartbeat Recorder

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app')

After initialization, a recurring task starts in the background and sends a heartbeat to the database every 30 seconds. The heartbeat cycle can be explicitly specified:

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app', heartbeat_period=15)

If the application is running multiple recurring tasks. Their heartbeats can also be registered:

from contek_tusk.heartbeat import beat

beat(task='scan_disk', heartbeat_period=60)
SELECT * FROM maintenance.heartbeats
app task datetime sequence
my_app main 2022-03-10 06:51:01.178 0
my_app main 2022-03-10 06:51:31.178 1
my_app main 2022-03-10 06:52:01.178 2
my_app scan_disk 2022-03-10 06:52:04.215 0
my_app main 2022-03-10 06:52:31.178 3
my_app main 2022-03-10 06:53:01.178 4
my_app scan_disk 2022-03-10 06:53:04.215 1

5.Logging

import logging
from contek_tusk.logging import init

logger = logging.getLogger(__name__)
init(table='maintenance.logs', app_name='my_app')

logger.info('Hello world!')
logger.warning('Fire drill test.')
logger.error('This is real fire.')
try:
    raise ValueError('Guess what?')
except ValueError:
    logger.exception('Oh no!')
logger.critical('I give up.')
SELECT * FROM maintenance.logs
app datetime level logger line error message stacktrace
my_app 2022-03-10 08:53:03.558 4 example.py 7 Hello world!
my_app 2022-03-10 08:53:03.558 3 example.py 8 Fire drill test.
my_app 2022-03-10 08:53:03.558 2 example.py 9 This is real fire.
my_app 2022-03-10 08:53:03.558 2 example.py 13 ValueError Oh no! (traceback)
my_app 2022-03-10 08:53:03.558 1 example.py 14 I give up.

6. Full Initialization

Standard

import contek_tusk as tusk
import contek_tusk.heartbeat as heartbeat
import contek_tusk.logging as logging

tusk.init(host='localhost', user='tester', password='P@assw0rd')
tusk.set_app_name('my_app')
heartbeat.init(table='maintenance.heartbeats')
logging.init(table='maintenance.logs')

YAML

app: 'my_app'
host: 'localhost'
user: 'tester'
password: 'P@assw0rd'
heartbeat:
  database: 'maintenance'
  table: 'heartbeats'
logging:
  database: 'maintenance'
  table: 'logs'
import contek_tusk.yaml as tusk_yaml

tusk_yaml.for_yaml_file('example.yml')

7. Example Table Schema

maintenance.heartbeats

CREATE TABLE IF NOT EXISTS maintenance.heartbeats
(
    `app` String, 
    `datetime` DateTime64(3),
    `task` String,
    `sequence` UInt32,
    `expiry` DateTime64
)
ENGINE = MergeTree()
ORDER BY
(
    `app`,
    `datetime`,
    `task`,
    `sequence`
)
TTL toDateTime(datetime) + INTERVAL 28 DAY DELETE;

maintenance.logs

CREATE TABLE IF NOT EXISTS maintenance.logs
(
    `app` String,
    `datetime` DateTime64(3),
    `level` Int8,
    `logger` String,
    `line` UInt32,
    `error` String,
    `message` String,
    `stacktrace` String
)
ENGINE = MergeTree()
ORDER BY
(
    `category`,
    `host`,
    `app`,
    `datetime`,
    `level`,
    `logger`,
    `line`,
    `error`
)
TTL toDateTime(datetime) + INTERVAL 7 DAY DELETE;

8. Customization

To write to tables with different schemas. Specify the column names in TuskHeartbeatConfig and TuskLoggingConfig.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

contek-tusk-1.3.tar.gz (15.3 kB view details)

Uploaded Source

Built Distribution

contek_tusk-1.3-py3-none-any.whl (22.7 kB view details)

Uploaded Python 3

File details

Details for the file contek-tusk-1.3.tar.gz.

File metadata

  • Download URL: contek-tusk-1.3.tar.gz
  • Upload date:
  • Size: 15.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.1 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.0

File hashes

Hashes for contek-tusk-1.3.tar.gz
Algorithm Hash digest
SHA256 d642954d2e62eda4017cf1bd085e53f37bc1e50c33606535d858dfbeec66dd00
MD5 d84e61ff20ba0266e2cf322ebf3a266a
BLAKE2b-256 ad5d84a4671e0951c7f5930fddc44d85b026b63db9f2282764e440ad9e19fcae

See more details on using hashes here.

File details

Details for the file contek_tusk-1.3-py3-none-any.whl.

File metadata

  • Download URL: contek_tusk-1.3-py3-none-any.whl
  • Upload date:
  • Size: 22.7 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.1 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.0

File hashes

Hashes for contek_tusk-1.3-py3-none-any.whl
Algorithm Hash digest
SHA256 57eb87974c72f9359f6bb9cd045141af524a66b3d52bdccfae0f28306fe13c2e
MD5 d0c48d9945412367a949d99ebb2482ac
BLAKE2b-256 0055edd112b31f805930c6488b3dfd69a4b3d242c37efe152c5e6b354006acbf

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page