Hazy

CI Python 3.9+

A dependency-free Python toolkit for fast, mergeable probabilistic data structures, implemented in Rust.

Start with the problem

If you know the engineering constraint but not the sketch parameters, Hazy can choose them:

from hazy import plan

recommendation = plan(
    "count_unique",
    expected_items=100_000_000,
    error_rate=0.01,
    max_memory="16 MiB",
)
counter = recommendation.create()

print(recommendation.explain())

plan() turns a goal, error target, and memory budget into a configured data structure. You can still construct every structure directly when you want full control.

Why Hazy?

When working with massive datasets, exact answers become expensive. Counting unique visitors across billions of events? A precise solution needs gigabytes of memory. Checking if a URL exists in a blocklist of millions? Exact lookups are slow.

Probabilistic data structures solve this by trading perfect accuracy for dramatic improvements in speed and memory. A HyperLogLog can count 1 billion unique items using just 16KB of memory (with ~1% error). A Bloom filter can check membership in a set of 10 million items using 12MB instead of hundreds of megabytes.

Hazy provides compact implementations with one consistent Python API, batch operations, deterministic hashing, and a Rust core. Use it when you need to:

  • Count unique items in streams too large to fit in memory
  • Check set membership without storing every element
  • Estimate frequencies of items in high-throughput streams
  • Find similar documents without comparing every pair
  • Track top-K items in real-time leaderboards

Features

  • Bloom Filter: Space-efficient set membership testing
  • Counting Bloom Filter: Bloom filter with deletion support
  • Scalable Bloom Filter: Auto-scaling Bloom filter for unknown cardinality
  • Cuckoo Filter: Fixed-capacity membership filter with deletion
  • HyperLogLog: Cardinality estimation with ~2% error using minimal memory
  • Count-Min Sketch: Frequency estimation for streaming data
  • MinHash: Set similarity estimation using Jaccard index
  • Top-K (Space-Saving): Find the most frequent items in a stream

Engineering features

  • Rust backend with GIL-free batch operations
  • xxHash3 for fast, high-quality hashing
  • Pre-hashed input for interoperable high-throughput pipelines
  • Goal-oriented planning from accuracy and memory requirements
  • Byte and JSON serialization for storage and transport
  • File I/O with save()/load() for persistence

Installation

pip install hazy

Building from source

Requires a current stable Rust toolchain and Python 3.9+:

# Install Rust if needed
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Build and install
pip install maturin
maturin develop --release

Quick Start

from hazy import BloomFilter, HyperLogLog, CountMinSketch

# Bloom Filter - set membership
bf = BloomFilter(expected_items=10000, false_positive_rate=0.01)
bf.add("hello")
print("hello" in bf)  # True
print("world" in bf)  # False (probably)

# HyperLogLog - count unique items
hll = HyperLogLog(precision=14)
hll.update_many(f"user_{i}" for i in range(1_000_000))
print(f"Unique users: {hll.cardinality():.0f}")  # ~1,000,000

# Count-Min Sketch - frequency estimation
cms = CountMinSketch(width=10000, depth=5)
cms.add("apple")
cms.add("apple")
cms.add("banana")
print(f"Apple count: {cms['apple']}")  # >= 2

Example: unique users from an event stream

This processes one million events without building a set of user IDs:

from hazy import plan

recommendation = plan(
    "count_unique",
    expected_items=50_000_000,
    error_rate=0.01,
    max_memory="16 MiB",
)
counter = recommendation.create()

events = ({"user_id": f"user-{i % 250_000}"} for i in range(1_000_000))
for event in events:
    counter.add(event["user_id"])

print(f"Estimated unique users: {counter.cardinality():,.0f}")
print(f"Sketch memory: {counter.size_in_bytes / 1024:,.0f} KiB")

The runnable version is in examples/unique_users.py.

Direct parameter selection

Use the estimation helpers to choose optimal parameters:

from hazy import estimate_bloom_params, estimate_hll_params

# Bloom filter for 1M items at 1% FPR
params = estimate_bloom_params(expected_items=1_000_000, false_positive_rate=0.01)
print(f"Memory needed: {params.memory_mb:.1f} MB")
print(f"Hash functions: {params.num_hashes}")

# HyperLogLog for 1% error
params = estimate_hll_params(expected_cardinality=1_000_000, error_rate=0.01)
print(f"Precision: {params.precision}")
print(f"Memory: {params.memory_bytes} bytes")

API Overview

All structures share a consistent API:

# Add items
structure.add(item)           # Add single item
structure.update(items)       # Add multiple items
structure.update_many(items)  # Explicit optimized batch path

# Query
structure.query(item)         # Query (meaning varies by structure)
structure.query_many(items)   # Batch query where applicable
item in structure             # Membership test (where applicable)

# Combine
structure.merge(other)        # Combine two structures
result = structure | other    # Union operator (where applicable)

# Serialize
data = structure.to_bytes()   # Binary serialization
structure = Type.from_bytes(data)

json_str = structure.to_json()
structure = Type.from_json(json_str)

# Introspection
len(structure)                # Approximate count
structure.size_in_bytes       # Memory footprint

# File I/O
structure.save("filter.hazy")
structure = Type.load("filter.hazy")

Data Structures

BloomFilter

from hazy import BloomFilter

bf = BloomFilter(expected_items=10000)
bf.add("hello")
assert "hello" in bf
assert bf.false_positive_rate < 0.02

CountingBloomFilter

from hazy import CountingBloomFilter

cbf = CountingBloomFilter(expected_items=10000)
cbf.add("hello")
cbf.add("hello")
cbf.remove("hello")  # Still contains "hello"
cbf.remove("hello")  # Now removed

Only remove items known to have been inserted. Like other probabilistic filters, a positive membership check can be a false positive; removing an item that was never added can decrement counters shared with real items.

ScalableBloomFilter

from hazy import ScalableBloomFilter

# Automatically grows as you add items
sbf = ScalableBloomFilter(initial_capacity=1000)
for i in range(1_000_000):  # Way more than initial capacity
    sbf.add(f"item_{i}")

print(f"Slices: {sbf.num_slices}")  # Multiple slices created
print("item_500" in sbf)  # True

CuckooFilter

from hazy import CuckooFilter

cf = CuckooFilter(capacity=10000)
cf.add("hello")
cf.remove("hello")
assert "hello" not in cf

For the same reason, only remove keys known to have been inserted: an absent key can share a fingerprint with a stored key.

HyperLogLog

from hazy import HyperLogLog

hll = HyperLogLog(precision=14)  # 16KB memory, ~0.8% error
hll.update([f"item_{i}" for i in range(1000000)])
print(f"Cardinality: {hll.cardinality():.0f}")

CountMinSketch

from hazy import CountMinSketch

cms = CountMinSketch(width=10000, depth=5)
# Or: cms = CountMinSketch(error_rate=0.001, confidence=0.99)
cms.add("apple", count=10)
print(f"Apple frequency: {cms['apple']}")

MinHash

from hazy import MinHash

mh1 = MinHash(num_hashes=128)
mh1.update(["a", "b", "c", "d"])

mh2 = MinHash(num_hashes=128)
mh2.update(["c", "d", "e", "f"])

print(f"Jaccard similarity: {mh1.jaccard(mh2):.2f}")  # ~0.33

TopK

from hazy import TopK

tk = TopK(k=10)
for word in ["apple"] * 100 + ["banana"] * 50 + ["cherry"] * 25:
    tk.add(word)

for item, count in tk.top(3):
    print(f"{item}: {count}")

Visualization

Install with visualization support:

pip install hazy[viz]

Plotting

from hazy import BloomFilter, HyperLogLog, CountMinSketch, TopK
from hazy.viz import plot_bloom, plot_hll, plot_cms, plot_topk, show

# Bloom filter bit array heatmap
bf = BloomFilter(expected_items=10000)
bf.update([f"item_{i}" for i in range(5000)])
plot_bloom(bf)

# HyperLogLog register histogram
hll = HyperLogLog(precision=12)
hll.update([f"user_{i}" for i in range(100000)])
plot_hll(hll)

# Count-Min Sketch heatmap
cms = CountMinSketch(width=100, depth=5)
for word in ["apple"] * 50 + ["banana"] * 30 + ["cherry"] * 10:
    cms.add(word)
plot_cms(cms)

# Top-K bar chart
tk = TopK(k=10)
tk.update(["apple"] * 100 + ["banana"] * 50 + ["cherry"] * 25)
plot_topk(tk)

show()  # Display all figures

Jupyter Notebooks

Enable rich HTML display in Jupyter:

import hazy
hazy.enable_notebook_display()

bf = hazy.BloomFilter(expected_items=1000)
bf.add("hello")
bf  # Displays rich HTML with stats and progress bar

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Build release wheel
maturin build --release

Reproducible comparisons with pyprobables, datasketch, pybloom-live, and zpds are in benchmarks/.

License

MIT

Project links

Data verified by PyPI on Aug 11, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for haoud from gravatar.com haoud

Credits

Author: Caroline Haoud

GitHub Statistics

Data verified by PyPI on Aug 11, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.9

Provides Extra

all dev docs numpy viz

Tags

bloom-filter hyperloglog count-min-sketch cuckoo-filter probabilistic data-structures cardinality streaming

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hazy-0.3.1.tar.gz (141.4 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

hazy-0.3.1-cp39-abi3-win_amd64.whl (398.6 kB view details)

Uploaded CPython 3.9+Windows x86-64

hazy-0.3.1-cp39-abi3-musllinux_1_2_x86_64.whl (718.8 kB view details)

Uploaded CPython 3.9+musllinux: musl 1.2+ x86-64

hazy-0.3.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (505.7 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

hazy-0.3.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (495.7 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

hazy-0.3.1-cp39-abi3-macosx_11_0_arm64.whl (463.5 kB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

hazy-0.3.1-cp39-abi3-macosx_10_12_x86_64.whl (480.7 kB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

File details

Details for the file hazy-0.3.1.tar.gz.

File metadata

  • Download URL: hazy-0.3.1.tar.gz
  • Upload date:
  • Size: 141.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.1.tar.gz
Algorithm Hash digest
SHA256 de92d93cb16bd4cf755db65c70286f475ceed7a75635dabb97f9a0d0847d9182
MD5 a2b9f0fcffdc50c2a8d20ea7db361035
BLAKE2b-256 799379dc692f2f7534e3ccc84865447fa5730f9d0dec496fe3d0acf364ea5b89

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.1.tar.gz:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.1-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: hazy-0.3.1-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 398.6 kB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.1-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 26ffa042f49974f0be9161599bd28253d33fda3beee8cd8f2e3540415295a9c8
MD5 a5c7ce0a39d13e4af3a785236d4f0ed7
BLAKE2b-256 4579b4882d1877c8a27bb7d9c17be14f6da3725517bb011a512fc06802ef26d1

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.1-cp39-abi3-win_amd64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.1-cp39-abi3-musllinux_1_2_x86_64.whl.

File metadata

  • Download URL: hazy-0.3.1-cp39-abi3-musllinux_1_2_x86_64.whl
  • Upload date:
  • Size: 718.8 kB
  • Tags: CPython 3.9+, musllinux: musl 1.2+ x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.1-cp39-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 0f4b284af25c3586e46b4a94171abb1f1aadcca468aaf5cf817e1cb159457aa7
MD5 99311fa751c81fceec36d190f67f27fe
BLAKE2b-256 dd0f1108c1ecdb1b06f3641f26d038e0d1ae515653f383f9f6fdbb9351a96818

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.1-cp39-abi3-musllinux_1_2_x86_64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hazy-0.3.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 0f88134dfa77719c12d2ac290066773de8947ba30b2c8557c2d9de09286536b1
MD5 b81710f290189f157d3d526abe169e96
BLAKE2b-256 82f427f5bd53628dd9b31196571c41684870b2eb69ee726874d3420f1ea9d3a6

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for hazy-0.3.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 287d1d09dec6a8c73212162a460476c396acd2ec4c3eeb749bc402d31439a139
MD5 7902b0faa14c51f7a9e7ed062a4ce2c1
BLAKE2b-256 9a44900d7211f0c9ed27f543b68184c4e2fb20c7681ae5074c267a7bfe4b2275

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.1-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

  • Download URL: hazy-0.3.1-cp39-abi3-macosx_11_0_arm64.whl
  • Upload date:
  • Size: 463.5 kB
  • Tags: CPython 3.9+, macOS 11.0+ ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.1-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6a16df69f7978ae5f35257a0cf0e084cdef97b1263ee7b0d114a9cc8fcaf2308
MD5 608357cce00fe57b90249af4fe58b041
BLAKE2b-256 84b7229aa9e459369b8e41b4104578a560362bef24f86f55405dd973dfa8f840

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.1-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.1-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

  • Download URL: hazy-0.3.1-cp39-abi3-macosx_10_12_x86_64.whl
  • Upload date:
  • Size: 480.7 kB
  • Tags: CPython 3.9+, macOS 10.12+ x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.1-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 4d71b7ed01c7856bcf76cfb903cb76c066b43682246b786d3e11a6b40f983f86
MD5 653385c08a6b55110410f422ab010b84
BLAKE2b-256 6956fd97d61f271fdb93c36a32a994908ec83d1c1d578dc8c96d00b353fef288

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.1-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

0.3.1 This release

7 files

0.3.0

7 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page