Skip to main content

Hazy

CI Python 3.9+

A dependency-free Python toolkit for fast, mergeable probabilistic data structures, implemented in Rust.

Why Hazy?

When working with massive datasets, exact answers become expensive. Counting unique visitors across billions of events? A precise solution needs gigabytes of memory. Checking if a URL exists in a blocklist of millions? Exact lookups are slow.

Probabilistic data structures solve this by trading perfect accuracy for dramatic improvements in speed and memory. A HyperLogLog can count 1 billion unique items using just 16KB of memory (with ~1% error). A Bloom filter can check membership in a set of 10 million items using 12MB instead of hundreds of megabytes.

Hazy provides compact implementations with one consistent Python API, batch operations, deterministic hashing, and a Rust core. Use it when you need to:

  • Count unique items in streams too large to fit in memory
  • Check set membership without storing every element
  • Estimate frequencies of items in high-throughput streams
  • Find similar documents without comparing every pair
  • Track top-K items in real-time leaderboards

Features

  • Bloom Filter: Space-efficient set membership testing
  • Counting Bloom Filter: Bloom filter with deletion support
  • Scalable Bloom Filter: Auto-scaling Bloom filter for unknown cardinality
  • Cuckoo Filter: Fixed-capacity membership filter with deletion
  • HyperLogLog: Cardinality estimation with ~2% error using minimal memory
  • Count-Min Sketch: Frequency estimation for streaming data
  • MinHash: Set similarity estimation using Jaccard index
  • Top-K (Space-Saving): Find the most frequent items in a stream

Engineering features

  • Rust backend with GIL-free batch operations
  • xxHash3 for fast, high-quality hashing
  • Pre-hashed input for interoperable high-throughput pipelines
  • Goal-oriented planning from accuracy and memory requirements
  • Byte and JSON serialization for storage and transport
  • File I/O with save()/load() for persistence

Installation

pip install hazy

Building from source

Requires a current stable Rust toolchain and Python 3.9+:

# Install Rust if needed
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Build and install
pip install maturin
maturin develop --release

Quick Start

from hazy import BloomFilter, HyperLogLog, CountMinSketch

# Bloom Filter - set membership
bf = BloomFilter(expected_items=10000, false_positive_rate=0.01)
bf.add("hello")
print("hello" in bf)  # True
print("world" in bf)  # False (probably)

# HyperLogLog - count unique items
hll = HyperLogLog(precision=14)
hll.update_many(f"user_{i}" for i in range(1_000_000))
print(f"Unique users: {hll.cardinality():.0f}")  # ~1,000,000

# Count-Min Sketch - frequency estimation
cms = CountMinSketch(width=10000, depth=5)
cms.add("apple")
cms.add("apple")
cms.add("banana")
print(f"Apple count: {cms['apple']}")  # >= 2

Plan from a goal

You do not need to start with the sketch mathematics:

from hazy import plan

recommendation = plan(
    "count_unique",
    expected_items=100_000_000,
    error_rate=0.01,
    max_memory="16 MiB",
)
counter = recommendation.create()
print(recommendation.explain())

Direct parameter selection

Use the estimation helpers to choose optimal parameters:

from hazy import estimate_bloom_params, estimate_hll_params

# Bloom filter for 1M items at 1% FPR
params = estimate_bloom_params(expected_items=1_000_000, false_positive_rate=0.01)
print(f"Memory needed: {params.memory_mb:.1f} MB")
print(f"Hash functions: {params.num_hashes}")

# HyperLogLog for 1% error
params = estimate_hll_params(expected_cardinality=1_000_000, error_rate=0.01)
print(f"Precision: {params.precision}")
print(f"Memory: {params.memory_bytes} bytes")

API Overview

All structures share a consistent API:

# Add items
structure.add(item)           # Add single item
structure.update(items)       # Add multiple items
structure.update_many(items)  # Explicit optimized batch path

# Query
structure.query(item)         # Query (meaning varies by structure)
structure.query_many(items)   # Batch query where applicable
item in structure             # Membership test (where applicable)

# Combine
structure.merge(other)        # Combine two structures
result = structure | other    # Union operator (where applicable)

# Serialize
data = structure.to_bytes()   # Binary serialization
structure = Type.from_bytes(data)

json_str = structure.to_json()
structure = Type.from_json(json_str)

# Introspection
len(structure)                # Approximate count
structure.size_in_bytes       # Memory footprint

# File I/O
structure.save("filter.hazy")
structure = Type.load("filter.hazy")

Data Structures

BloomFilter

from hazy import BloomFilter

bf = BloomFilter(expected_items=10000)
bf.add("hello")
assert "hello" in bf
assert bf.false_positive_rate < 0.02

CountingBloomFilter

from hazy import CountingBloomFilter

cbf = CountingBloomFilter(expected_items=10000)
cbf.add("hello")
cbf.add("hello")
cbf.remove("hello")  # Still contains "hello"
cbf.remove("hello")  # Now removed

Only remove items known to have been inserted. Like other probabilistic filters, a positive membership check can be a false positive; removing an item that was never added can decrement counters shared with real items.

ScalableBloomFilter

from hazy import ScalableBloomFilter

# Automatically grows as you add items
sbf = ScalableBloomFilter(initial_capacity=1000)
for i in range(1_000_000):  # Way more than initial capacity
    sbf.add(f"item_{i}")

print(f"Slices: {sbf.num_slices}")  # Multiple slices created
print("item_500" in sbf)  # True

CuckooFilter

from hazy import CuckooFilter

cf = CuckooFilter(capacity=10000)
cf.add("hello")
cf.remove("hello")
assert "hello" not in cf

For the same reason, only remove keys known to have been inserted: an absent key can share a fingerprint with a stored key.

HyperLogLog

from hazy import HyperLogLog

hll = HyperLogLog(precision=14)  # 16KB memory, ~0.8% error
hll.update([f"item_{i}" for i in range(1000000)])
print(f"Cardinality: {hll.cardinality():.0f}")

CountMinSketch

from hazy import CountMinSketch

cms = CountMinSketch(width=10000, depth=5)
# Or: cms = CountMinSketch(error_rate=0.001, confidence=0.99)
cms.add("apple", count=10)
print(f"Apple frequency: {cms['apple']}")

MinHash

from hazy import MinHash

mh1 = MinHash(num_hashes=128)
mh1.update(["a", "b", "c", "d"])

mh2 = MinHash(num_hashes=128)
mh2.update(["c", "d", "e", "f"])

print(f"Jaccard similarity: {mh1.jaccard(mh2):.2f}")  # ~0.33

TopK

from hazy import TopK

tk = TopK(k=10)
for word in ["apple"] * 100 + ["banana"] * 50 + ["cherry"] * 25:
    tk.add(word)

for item, count in tk.top(3):
    print(f"{item}: {count}")

Visualization

Install with visualization support:

pip install hazy[viz]

Plotting

from hazy import BloomFilter, HyperLogLog, CountMinSketch, TopK
from hazy.viz import plot_bloom, plot_hll, plot_cms, plot_topk, show

# Bloom filter bit array heatmap
bf = BloomFilter(expected_items=10000)
bf.update([f"item_{i}" for i in range(5000)])
plot_bloom(bf)

# HyperLogLog register histogram
hll = HyperLogLog(precision=12)
hll.update([f"user_{i}" for i in range(100000)])
plot_hll(hll)

# Count-Min Sketch heatmap
cms = CountMinSketch(width=100, depth=5)
for word in ["apple"] * 50 + ["banana"] * 30 + ["cherry"] * 10:
    cms.add(word)
plot_cms(cms)

# Top-K bar chart
tk = TopK(k=10)
tk.update(["apple"] * 100 + ["banana"] * 50 + ["cherry"] * 25)
plot_topk(tk)

show()  # Display all figures

Jupyter Notebooks

Enable rich HTML display in Jupyter:

import hazy
hazy.enable_notebook_display()

bf = hazy.BloomFilter(expected_items=1000)
bf.add("hello")
bf  # Displays rich HTML with stats and progress bar

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Build release wheel
maturin build --release

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

hazy-0.3.0.tar.gz (134.5 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

hazy-0.3.0-cp39-abi3-win_amd64.whl (398.3 kB view details)

Uploaded CPython 3.9+Windows x86-64

hazy-0.3.0-cp39-abi3-musllinux_1_2_x86_64.whl (718.6 kB view details)

Uploaded CPython 3.9+musllinux: musl 1.2+ x86-64

hazy-0.3.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (505.4 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ x86-64

hazy-0.3.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (495.4 kB view details)

Uploaded CPython 3.9+manylinux: glibc 2.17+ ARM64

hazy-0.3.0-cp39-abi3-macosx_11_0_arm64.whl (463.1 kB view details)

Uploaded CPython 3.9+macOS 11.0+ ARM64

hazy-0.3.0-cp39-abi3-macosx_10_12_x86_64.whl (480.4 kB view details)

Uploaded CPython 3.9+macOS 10.12+ x86-64

File details

Details for the file hazy-0.3.0.tar.gz.

File metadata

  • Download URL: hazy-0.3.0.tar.gz
  • Upload date:
  • Size: 134.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.0.tar.gz
Algorithm Hash digest
SHA256 e1ec356d4385dc9afd9bd576d0ff7c7e586507e8a4afb2fc1578bfb8c5cb9992
MD5 fb08c3f3bd3b479fbb936fb16bab1360
BLAKE2b-256 06a0bcdb1392c2e426192bef0a87b60c2a95c4c186ba22e6192e2306694143c4

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.0.tar.gz:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.0-cp39-abi3-win_amd64.whl.

File metadata

  • Download URL: hazy-0.3.0-cp39-abi3-win_amd64.whl
  • Upload date:
  • Size: 398.3 kB
  • Tags: CPython 3.9+, Windows x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.0-cp39-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 7069e99efa856f404e94193b73c5b97af20f58df3fb20732dd831d6f903ea090
MD5 d7804d060adf7febb2c262b819c9c1fa
BLAKE2b-256 dc59842ed23eeb30ddf85c807210db6fb5920cee1de362c6f7cbb826548113fc

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.0-cp39-abi3-win_amd64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.0-cp39-abi3-musllinux_1_2_x86_64.whl.

File metadata

  • Download URL: hazy-0.3.0-cp39-abi3-musllinux_1_2_x86_64.whl
  • Upload date:
  • Size: 718.6 kB
  • Tags: CPython 3.9+, musllinux: musl 1.2+ x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.0-cp39-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 3159ac0cee2bb22ffedaabda743ded310de966a2f70e456137920b734dc1adc5
MD5 76b1ef00e69a05bb4b16a34a58181ce0
BLAKE2b-256 01261c5f6b8df29564b0ed78b6733647d9566b880e2fb6c7a1fcaffad9f8c4b0

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.0-cp39-abi3-musllinux_1_2_x86_64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for hazy-0.3.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 e76a5f668ad7ccf51b71058481a8c7b8907be6b305405fe643a48ff949746ef7
MD5 27353ffffc2f86c6e556cc23e8e57791
BLAKE2b-256 e67e30fbf4b52d0e3ffa1bb5670d5271af891a7cf601478546b3ce8e2c5ffb37

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for hazy-0.3.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 ead533cc803a1b33655f803386e671da0a1bf246f78b646c245b6b215fca6665
MD5 5bc3f1c9c81d5a8d796543ff0487a26d
BLAKE2b-256 b99897d20f7a8f935ba2f9f56cc2995d236ed0f2b7ea87d178104d19c84a1b64

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.0-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.0-cp39-abi3-macosx_11_0_arm64.whl.

File metadata

  • Download URL: hazy-0.3.0-cp39-abi3-macosx_11_0_arm64.whl
  • Upload date:
  • Size: 463.1 kB
  • Tags: CPython 3.9+, macOS 11.0+ ARM64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.0-cp39-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 09c0d57d6384b02fd446c9014ca85fe3427a3b89a9f8fb4969edfbc6a45ad330
MD5 5926d70e05d032a2f23e8aeba9c85c90
BLAKE2b-256 a7cadbc64bf1e795522e9620cc5278b040012d24949092a78d25479f6fdcc5bb

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.0-cp39-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file hazy-0.3.0-cp39-abi3-macosx_10_12_x86_64.whl.

File metadata

  • Download URL: hazy-0.3.0-cp39-abi3-macosx_10_12_x86_64.whl
  • Upload date:
  • Size: 480.4 kB
  • Tags: CPython 3.9+, macOS 10.12+ x86-64
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for hazy-0.3.0-cp39-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 404c82a1d42eec2378a71634d59c0b16e30f7e5a2686233b83f620eb1fdaab37
MD5 bf5431dcc8e560c18ccc043e843e2e78
BLAKE2b-256 3108d8a0701ebbc14c8543d49cce4897c6052dcc2e529229274b66b48da5558f

See more details on using hashes here.

Provenance

The following attestation bundles were made for hazy-0.3.0-cp39-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on carolinehaoud/hazy

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.3.1

7 files

This release

0.3.0 This release

7 files

0.1.0

2 files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page