llm-sketchkit

llm-sketchkit is a small Go and Python library for deterministic, mergeable summaries of high-cardinality LLM data. It provides matching semantics for text canonicalization, privacy-preserving keyed hashes, bounded sketches, and a deterministic protobuf wire format.

Raw prompts and identifiers do not need to enter sketch state. Producers can summarize locally and merge compatible sketches across processes or languages.

When This Fits

Use llm-sketchkit inside telemetry producers and processing components when exact per-key state would grow with cardinality or retain values that should not enter aggregate state.

Typical uses include:

• estimating distinct prompts, users, sessions, tools, or documents without keeping one counter per value;
• identifying token-heavy or request-heavy keys with deterministic lower and upper bounds;
• testing approximate set membership for bounded deduplication;
• comparing large sets using fixed-size similarity signatures; and
• producing summaries in Go that can be read and merged in Python with the same profiles and wire semantics.

Inputs can be canonicalized and keyed before entering sketch state. This keeps raw values out of the sketch, but the resulting hashes remain pseudonymous and linkable while the same secret is in use.

This is a sketch library, not a trace collector, sampling processor, storage backend, dashboard, or differential-privacy system. The FAQ answers common questions about fit, memory, accuracy, and interoperability.

Included Sketches

The Go and Python implementations share the same profiles, hash domains, test vectors, and serialized representation.

Evidence At A Glance

Measurements use deterministic workloads and report the least favorable of five Linux benchmark runs where applicable.

• HMAC-SHA256-64 sustained at least 1.65 million 64-byte inputs/s/core and 850,340 1 KiB inputs/s/core on an Intel Xeon Platinum 8573C with Go 1.26.5.
• HLL++ and weighted frequent-items updates took at most 10.58 ns/op and 145.6 ns/op, respectively, with 0 allocations/op in the measured paths.
• The HLL++ small profile's maximum observed relative error was 2.3301% across the characterization grid, within its 2.4375% enforced bound.
• Bloom profile false-positive rates were at or below their configured targets in the measured trials, with zero false negatives among inserted hashes.
• MinHash mean absolute error fell from 0.02845 at k=128 to 0.02009 at k=256, closely following the expected inverse-square-root relationship.
• Both weighted frequent-items oracle workloads retained 100% true top-20 recall and valid no-false-positive query results in both implementations.

See the visual scorecard, raw measurement records, and DataSketches implementation rationale for methods, limitations, and reproduction commands.

Requirements

• Go 1.25 or newer, with the latest security patch for that release line
• Python 3.11 or newer

Install

Go:

go get github.com/llm-measurement/llm-sketchkit@latest

Python from a checkout:

git clone https://github.com/llm-measurement/llm-sketchkit.git
cd llm-sketchkit
python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip==26.2 setuptools==83.0.0
python -m pip install .

For development checks, install the optional tools:

python -m pip install -e '.[dev]'

Quick Start

Generate a deployment secret and expose it to the process:

export LLM_SKETCHKIT_SECRET="$(python -c 'import secrets; print(secrets.token_hex(32))')"

Go HLL++ example:

package main

import (
	"fmt"
	"log"

	"github.com/llm-measurement/llm-sketchkit/go/sketchkit/canon"
	sketchhash "github.com/llm-measurement/llm-sketchkit/go/sketchkit/hash"
	"github.com/llm-measurement/llm-sketchkit/go/sketchkit/hllpp"
)

func main() {
	secret, err := sketchhash.SecretFromEnv("LLM_SKETCHKIT_SECRET")
	if err != nil {
		log.Fatal(err)
	}

	sketch, err := hllpp.New(
		hllpp.ProfileSmall,
		sketchhash.PromptV1,
		sketchhash.HMACSHA25664,
	)
	if err != nil {
		log.Fatal(err)
	}

	canonical, err := canon.CanonicalizeString(canon.TextV1, "  cafe\u0301\r\n")
	if err != nil {
		log.Fatal(err)
	}
	digest, err := sketchhash.Hash64(secret, sketchhash.PromptV1, canonical)
	if err != nil {
		log.Fatal(err)
	}

	sketch.AddHash(digest)
	fmt.Printf("estimated distinct prompts: %.0f\n", sketch.Estimate())
}

Python HLL++ example:

from llm_sketchkit import PROMPT_V1, canonicalize_text_v1, hash64, hllpp
from llm_sketchkit import secret_from_env

secret = secret_from_env("LLM_SKETCHKIT_SECRET")
sketch = hllpp.new("small", PROMPT_V1)

canonical = canonicalize_text_v1("  cafe\u0301\r\n")
sketch.add_hash(hash64(secret, PROMPT_V1, canonical))

print(f"estimated distinct prompts: {sketch.estimate():.0f}")

Merge Sketches

Sketches merge only when their kind, profile, hash domain, hash algorithm, and shape metadata match. A mismatch is an error rather than an implicit conversion.

left = hllpp.new("small", PROMPT_V1)
right = hllpp.new("small", PROMPT_V1)

left.add_hash(hash64(secret, PROMPT_V1, canonicalize_text_v1("alpha")))
right.add_hash(hash64(secret, PROMPT_V1, canonicalize_text_v1("beta")))

left.merge(right)
print(f"merged distinct prompts: {left.estimate():.0f}")

Security And Privacy

• Hash inputs with a registered domain and a high-entropy secret before adding them to a sketch. The built-in secret loaders require at least 16 bytes and reject known placeholder values.
• Keyed hashes are pseudonymous, not anonymous. Anyone with the secret can test candidate values, and repeated hashes remain linkable while the same secret and domain are in use.
• Never log, serialize, or commit the hash secret. Rotate it when the trust boundary changes; rotation intentionally breaks comparison with older state.
• Bound raw input size before canonicalization. Canonicalization operates on in-memory text and intentionally leaves application-specific limits to callers.
• Sketches reveal bounded aggregate information and may reveal membership or recurrence. They do not provide differential privacy.
• Treat serialized sketches as untrusted input at process boundaries. The parse APIs cap input size and reject invalid profiles, domains, shapes, counters, and register values.

See SECURITY.md for private vulnerability reporting.

Wire Compatibility

Deterministic protobuf encoding is part of the compatibility surface. Go and Python are checked against the same canonicalization, hashing, sketch, and cross-language merge fixtures in vectors/.

Run all local checks:

go test ./... -race
python -m pytest -q
ruff check .
mypy --strict

The optional Apache DataSketches comparison checks weighted frequent-items query behavior against an independent implementation:

python -m pip install -e '.[oracle]'
python scripts/datasketches_oracle.py --check

Reference

• spec/ defines canonicalization, hashing, profiles, and wire encoding.
• vectors/ contains executable conformance fixtures.
• reports/ contains benchmark, accuracy, and oracle results.
• bench/ contains the Go benchmark harnesses.
• docs/FAQ.md answers common adoption questions.
• CHANGELOG.md records release-level changes.

Status

llm-sketchkit is an alpha library. The compatibility surface consists of the specifications, the Go and Python APIs exercised by the vectors, and the checked-in conformance fixtures.

License

Apache-2.0.