Skip to main content

Tusk Clickhouse Metric Writer

contek-tusk is a metric writer built upon Clickhouse.

1. Connection Initialization

import contek_tusk as tusk

tusk.init(host='localhost', user='tester', password='P@assw0rd')

2. Generic Metric

from contek_tusk import metric

cpu_usage = metric(table='system.cpu')

cpu_usage.write({
    'region': 'Tokyo',
    'instance_id': 'ABC',
    'usage': 0.43
})

cpu_usage.write({
    'region': 'Seoul',
    'instance_id': 'XYZ',
    'usage': 0.22
})
SELECT * FROM system.cpu
region instance_id datetime usage
Tokyo ABC 2022-03-10 06:42:42.746 0.43
Seoul XYZ 2022-03-10 06:42:42.746 0.22

Note: the client will automatically use the first DateTime column to record current timestamp. Alternatively, it can be explicitly specified:

from contek_tusk import metric
from contek_tusk.table import Table

table = Table(database='system', table_name='cpu', time_column='datetime')
cpu_usage = metric(table)

3. Summing Counter

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage')

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
})  # default count is 1

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '2.2.2.4'
}, 3)

counter.count({
    'resource_id': 'ABC111',
    'requester_ip': '1.2.3.4'
}, 4)  # the new count is 1 + 4 = 5

counter.count({
    'resource_id': 'ABC122',
    'requester_ip': '1.2.3.4'
}, 2)
SELECT * FROM download.quota_usage
resource_id requester_ip datetime count
ABC111 1.2.3.4 2022-03-10 06:42:48.411 5
ABC122 1.2.3.4 2022-03-10 06:42:48.411 2
ABC122 2.2.2.4 2022-03-10 06:42:48.411 3

Note: the client will automatically use the first Int or UInt column to record the count. Alternatively, it can be explicitly specified:

from contek_tusk.counter import counter

counter = counter(table='download.quota_usage', count_column='count')

4. Heartbeat Recorder

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app')

After initialization, a recurring task starts in the background and sends a heartbeat to the database every 30 seconds. The heartbeat cycle can be explicitly specified:

from contek_tusk.heartbeat import init

init(table='maintenance.heartbeats', app_name='my_app', heartbeat_period=15)

If the application is running multiple recurring tasks. Their heartbeats can also be registered:

from contek_tusk.heartbeat import beat

beat(task='scan_disk', heartbeat_period=60)
SELECT * FROM maintenance.heartbeats
app task datetime sequence
my_app main 2022-03-10 06:51:01.178 0
my_app main 2022-03-10 06:51:31.178 1
my_app main 2022-03-10 06:52:01.178 2
my_app scan_disk 2022-03-10 06:52:04.215 0
my_app main 2022-03-10 06:52:31.178 3
my_app main 2022-03-10 06:53:01.178 4
my_app scan_disk 2022-03-10 06:53:04.215 1

5.Logging

import logging
from contek_tusk.logging import init

logger = logging.getLogger(__name__)
init(table='maintenance.logs', app_name='my_app')

logger.info('Hello world!')
logger.warning('Fire drill test.')
logger.error('This is real fire.')
try:
    raise ValueError('Guess what?')
except ValueError:
    logger.exception('Oh no!')
logger.critical('I give up.')
SELECT * FROM maintenance.logs
app datetime level logger line error message stacktrace
my_app 2022-03-10 08:53:03.558 4 example.py 7 Hello world!
my_app 2022-03-10 08:53:03.558 3 example.py 8 Fire drill test.
my_app 2022-03-10 08:53:03.558 2 example.py 9 This is real fire.
my_app 2022-03-10 08:53:03.558 2 example.py 13 ValueError Oh no! (traceback)
my_app 2022-03-10 08:53:03.558 1 example.py 14 I give up.

6. Full Initialization

Standard

import contek_tusk as tusk
import contek_tusk.heartbeat as heartbeat
import contek_tusk.logging as logging

tusk.init(host='localhost', user='tester', password='P@assw0rd')
tusk.set_app_name('my_app')
heartbeat.init(table='maintenance.heartbeats')
logging.init(table='maintenance.logs')

YAML

app: 'my_app'
host: 'localhost'
user: 'tester'
password: 'P@assw0rd'
heartbeat:
  database: 'maintenance'
  table: 'heartbeats'
logging:
  database: 'maintenance'
  table: 'logs'
import contek_tusk.yaml as tusk_yaml

tusk_yaml.for_yaml_file('example.yml')

7. Example Table Schema

maintenance.heartbeats

CREATE TABLE IF NOT EXISTS maintenance.heartbeats
(
    `app` String, 
    `datetime` DateTime64(3),
    `task` String,
    `sequence` UInt32,
    `expiry` DateTime64
)
ENGINE = MergeTree()
ORDER BY
(
    `app`,
    `datetime`,
    `task`,
    `sequence`
)
TTL toDateTime(datetime) + INTERVAL 28 DAY DELETE;

maintenance.logs

CREATE TABLE IF NOT EXISTS maintenance.logs
(
    `app` String,
    `datetime` DateTime64(3),
    `level` Int8,
    `logger` String,
    `line` UInt32,
    `error` String,
    `message` String,
    `stacktrace` String
)
ENGINE = MergeTree()
ORDER BY
(
    `category`,
    `host`,
    `app`,
    `datetime`,
    `level`,
    `logger`,
    `line`,
    `error`
)
TTL toDateTime(datetime) + INTERVAL 7 DAY DELETE;

8. Customization

To write to tables with different schemas. Specify the column names in TuskHeartbeatConfig and TuskLoggingConfig.

Metadata

Release files for contek-tusk 2.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for contek-tusk 2.2
File Size Uploaded Provenance Actions
contek-tusk-2.2.tar.gz 15.5 kB Without provenance Details for contek-tusk-2.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for contek-tusk 2.2
File Interpreter ABI Platform Actions
contek_tusk-2.2-py3-none-any.whl Python 3 none any Details for contek_tusk-2.2-py3-none-any.whl

Total release size: 38.5 kB

Release files / contek-tusk-2.2.tar.gz

Download URL contek-tusk-2.2.tar.gz
Size 15.5 kB
Tags Source
SHA-256 checksum
How to use checksums
5e1ecc113c1e6458c14e52a7498ec9e1ca843b0f1b86dbcff8a82568b3be28e8
BLAKE2b-256 checksum
How to use checksums
3b109e9fe4db90b82e9e7f92ce43bed5a631607089f98417e2b7470a99b9482a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.1 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.0

Release files / contek_tusk-2.2-py3-none-any.whl

Download URL contek_tusk-2.2-py3-none-any.whl
Size 23.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
46939d1b469893670965966aaecbd1165745d972c2ca724877b0dac2a78842ae
BLAKE2b-256 checksum
How to use checksums
56c08734530d1ab0eef5cbd25fe9964da9f75a52892f4ef8834d87d00a850172
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.27.1 requests-toolbelt/0.9.1 urllib3/1.26.9 tqdm/4.63.1 importlib-metadata/4.11.3 keyring/23.5.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.10.0

Security for contek-tusk 2.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page