norm_toolkit 2.0.1

ClickHouse backend

The DuckDB builder remains the source of truth. Build a DuckDB file with build_merged_duckdb, then upload its canonical tables into ClickHouse:

uv run python scripts/upload_clickhouse.py data/dbs_final/SmallMolecule.duckdb --database normalization

The upload shows a progress bar for each copied table; pass --no-progress to silence it.

Connection settings are read from .env with python-dotenv and use the official clickhouse-connect client. Set CH_HTTP, for example http://host:8123/normalization; CH_USER and CH_PASSWORD may be supplied separately and override URL credentials.

Use the ClickHouse backend from Python:

import asyncio

from norm_toolkit import ClickHouseNormalizer


async def main():
    normalizer = await ClickHouseNormalizer.create(database="normalization")
    result = await normalizer.normalize(["aspirin"], top_k=5)
    print(result)


asyncio.run(main())

Normalizer instances keep a 10,000-item LRU cache. Repeated items skip string normalization, database lookup, enrichment, and hierarchy expansion; misses in a mixed batch are queried together and added to the cache. Synonyms and all result-affecting options are part of the cache key.

normalizer = await ClickHouseNormalizer.create(
    database="normalization",
    normalization_cache_size=50_000,  # 0 disables; None is unbounded
)

print(normalizer.normalization_cache_info())
normalizer.clear_normalization_cache()  # also use after refreshing backing tables

You can also pass a DSN in code:

normalizer = await ClickHouseNormalizer.create(
    dsn="http://host:8123/normalization",
    database="normalization",
)