hyperunique-druid

Decode, merge and estimate Apache Druid hyperUnique sketches (org.apache.druid.hll, version 1 wire format -- not DataSketches HLL). A from-scratch Rust port of Druid 30.0.0's HyperLogLogCollector, with optional Python bindings (PyO3) and a ready-made ClickHouse executable-UDF loop.

Distributed as a PyPI package (hyperunique-druid): abi3 wheels for CPython

= 3.9, Rust inside. Not published to crates.io -- the Rust API is internal to the wheel.

Rust (inside the wheel)

use hyperunique_druid::hyperunique::merge_estimate;

let estimate: f64 = merge_estimate(&[sketch_a.as_slice(), sketch_b.as_slice()])?;

hyperunique::{decode, fold, estimate} are there for finer control.

Python

import hyperunique_druid

hyperunique_druid.merge_estimate([sketch_a, sketch_b])  # float; 0.0 for []
hyperunique_druid.serve()  # ClickHouse RowBinary UDF loop over stdin/stdout

serve() runs the whole read/merge/write loop natively with the GIL released, so a ClickHouse Cloud UDF is just:

# main.py
import hyperunique_druid

hyperunique_druid.serve()

# requirements.txt
hyperunique-druid==0.1.1

Layout

Cargo.toml / pyproject.toml   one crate, built by cargo (Rust) or maturin (Python)
src/hyperunique.rs            the sketch logic
src/rowbinary.rs, src/udf.rs  ClickHouse RowBinary framing + UDF loop
src/python.rs                 PyO3 bindings (feature `python`)
python/hyperunique_druid/     Python package wrapping the `_native` extension
package.sh                    builds everything publishable into dist/
verify-wheels.sh              smoke-tests every Linux wheel in containers

Develop

cargo test                       # Rust tests (no Python needed)
uv venv && uvx maturin develop   # build + install into .venv

Release

1. Bump version in Cargo.toml (the Python version is read from it too). Fill in license and repository there on the first release.

2. ./package.sh -- runs the tests and writes to dist/: manylinux2014 (glibc) wheels for linux x86_64 and aarch64. Requires rustup and uv.

3. ./verify-wheels.sh -- installs each Linux wheel into real containers (amd64 + arm64, Python 3.9 + 3.13) and runs it. Requires podman or docker.

4. Upload (versions are permanent; set UV_PUBLISH_TOKEN first):

   uv publish dist/*linux*.whl   # --dry-run to check first
   

5. Pin the new version in udfs/hyperunique-merge-pyo3/requirements.txt.

Wheels are glibc-only (manylinux2014), x86_64 and aarch64 -- the same two architectures as the Rust and Go UDFs. Cloud's Python runtime is glibc, so no Alpine/musl wheels. Other platforms would need a target added to package.sh; there is deliberately no sdist, so they get pip's clear "no matching distribution" error instead of a failed Rust build.