clickhouse_core

Shared ClickHouse connectivity library for Analytic AI services.

What it does

• Creates ClickHouse clients using a single config model.
• Supports both TCP (clickhouse-driver) and HTTP (clickhouse-connect).
• Provides sync and async APIs with consistent return types.
• Normalizes query results to list-of-dict rows.
• Supports per-database pooled clients via a dbPoolMap.
• Logs client creation and query execution.

Install

pip install -r requirements.txt

Usage (TCP)

from clickhouse_core import ClickHouseConfig, get_client

config = ClickHouseConfig(host="clickhouse", port=9000, database="default")
client = get_client(config)
rows = client.fetch_all("SELECT 1 AS one")

Usage (HTTP)

from clickhouse_core import ClickHouseConfig, get_client

config = ClickHouseConfig(host="clickhouse", port=8123, database="default", protocol="http")
client = get_client(config)
rows = client.fetch_all("SELECT 1 AS one")

Async usage

from clickhouse_core import ClickHouseConfig, get_async_client

config = ClickHouseConfig(host="clickhouse", port=9000, database="default")
client = get_async_client(config)
rows = await client.fetch_all("SELECT 1 AS one")

Pool manager (dbPoolMap)

from libs.clickhouse_core import ClickHouseConfig, ClickHousePoolFactory

base_config = ClickHouseConfig.from_env()
db_pool_map = {"default": 2, "analytics": 5}

pool = ClickHousePoolFactory(base_config, db_pool_map).build()

client = pool.get_client("default")
rows = client.fetch_all("SELECT 1")

async_client = pool.get_async_client("analytics")
rows = await async_client.fetch_all("SELECT 1")

Connection checks and per-DB credentials

• For each DB in dbPoolMap, the pool manager checks connectivity once (using SELECT 1).
• If connection fails, that DB is skipped and a warning is logged.
• Per-DB env overrides are supported:

<DBNAME>_USERNAME
<DBNAME>_PASSWORD

If these are not set, the library falls back to CLICKHOUSE_USERNAME and CLICKHOUSE_PASSWORD from the base config.

By default, on-demand clients are allowed for DBs not in dbPoolMap. You can disable this by passing allow_on_demand=False to ClickHousePoolFactory(...).

Configuration

Environment variables supported (prefix CLICKHOUSE_):

• CLICKHOUSE_HOST
• CLICKHOUSE_PORT
• CLICKHOUSE_DATABASE
• CLICKHOUSE_USERNAME
• CLICKHOUSE_PASSWORD
• CLICKHOUSE_PROTOCOL (tcp|http)
• CLICKHOUSE_SECURE (true/false)
• CLICKHOUSE_CONNECT_TIMEOUT
• CLICKHOUSE_SEND_RECEIVE_TIMEOUT
• CLICKHOUSE_HTTP_VERIFY
• CLICKHOUSE_COMPRESSION
• CLICKHOUSE_LOG_NAME

Logging

The library uses Python's standard logging. Configure logging in your service to see client creation and query events. The logger name is taken from CLICKHOUSE_LOG_NAME or defaults to clickhouse_core.

Versioning

See RELEASE.md for the git tag workflow and version bump rules.

Helper scripts:

python scripts/bump_clickhouse_core_version.py patch
python scripts/build_clickhouse_core.py

Build & install (real package)

From libs/clickhouse_core:

python -m pip install --upgrade build
python -m build

Install locally:

pip install dist/clickhouse_core-0.1.0-py3-none-any.whl

Install with HTTP support:

pip install dist/clickhouse_core-0.1.0-py3-none-any.whl[http]

Protected build (Cython .pyd inside wheel)

Set the environment variable before building:

set CLICKHOUSE_CORE_CYTHONIZE=1
python -m build

This produces a platform-specific wheel (e.g., cp312-win_amd64) containing .pyd files.

Monorepo usage

If you are using the library directly from this repo (not installed as a package):

from libs.clickhouse_core import ClickHouseConfig, get_client