hyperunique-druid

Decode, merge and estimate Apache Druid hyperUnique sketches (org.apache.druid.hll, version 1 wire format -- not DataSketches HLL). A from-scratch Rust port of Druid 30.0.0's HyperLogLogCollector, with optional Python bindings (PyO3) and a ready-made ClickHouse executable-UDF loop.

Distributed as a PyPI package (hyperunique-druid): abi3 wheels for CPython

= 3.9, Rust inside. Not published to crates.io -- the Rust API is internal to the wheel.

Rust (inside the wheel)

use hyperunique_druid::hyperunique::merge_estimate;

let estimate: f64 = merge_estimate(&[sketch_a.as_slice(), sketch_b.as_slice()])?;

hyperunique::{decode, fold, estimate} are there for finer control.

Python

import hyperunique_druid

hyperunique_druid.merge_estimate([sketch_a, sketch_b])  # float; 0.0 for []
hyperunique_druid.serve()  # ClickHouse RowBinary UDF loop over stdin/stdout

serve() runs the whole read/merge/write loop natively with the GIL released, so a ClickHouse Cloud UDF is just:

# main.py
import hyperunique_druid

hyperunique_druid.serve()

# requirements.txt
hyperunique-druid==0.1.0

Layout

Cargo.toml / pyproject.toml   one crate, built by cargo (Rust) or maturin (Python)
src/hyperunique.rs            the sketch logic
src/rowbinary.rs, src/udf.rs  ClickHouse RowBinary framing + UDF loop
src/python.rs                 PyO3 bindings (feature `python`)
python/hyperunique_druid/     Python package wrapping the `_native` extension
package.sh                    builds everything publishable into dist/
verify-wheels.sh              smoke-tests every Linux wheel in containers

Develop

cargo test                       # Rust tests (no Python needed)
uv venv && uvx maturin develop   # build + install into .venv

Release

1. Bump version in Cargo.toml (the Python version is read from it too). Fill in license and repository there on the first release.

2. ./package.sh -- runs the tests and writes to dist/: manylinux2014 (glibc) and musllinux_1_2 (musl) wheels for linux x86_64 and aarch64, plus a native wheel for local use. Requires rustup and uv.

3. ./verify-wheels.sh -- installs each Linux wheel into real containers (glibc + musl, amd64 + arm64, Python 3.9 + 3.13) and runs it. Requires podman or docker.

4. Upload (versions are permanent; set MATURIN_PYPI_TOKEN first):

   uvx maturin upload dist/*linux*.whl
   

5. Pin the new version in udfs/hyperunique-merge-pyo3/requirements.txt.

pip picks the matching wheel, so Cloud's platform doesn't need to be known. Other platforms (armv7, ppc64le, ...) would need a target added to package.sh; there is deliberately no sdist, so those get pip's clear "no matching distribution" error instead of a failed Rust build.