Skip to main content

Probabilistic data structures for processing and searching very large datasets

Project description

datasketch gives you probabilistic data structures that can process and search very large amount of data super fast, with little loss of accuracy.

This package contains the following data sketches:

Data Sketch



estimate Jaccard similarity and cardinality

Weighted MinHash

estimate weighted Jaccard similarity


estimate cardinality


estimate cardinality

The following indexes for data sketches are provided to support sub-linear query time:


For Data Sketch

Supported Query Type

MinHash LSH

MinHash, Weighted MinHash

Jaccard Threshold

MinHash LSH Forest

MinHash, Weighted MinHash

Jaccard Top-K

MinHash LSH Ensemble


Containment Threshold



Custom Metric Top-K

datasketch must be used with Python 3.7 or above, NumPy 1.11 or above, and Scipy.

Note that MinHash LSH and MinHash LSH Ensemble also support Redis and Cassandra storage layer (see MinHash LSH at Scale).


To install datasketch using pip:

pip install datasketch

This will also install NumPy as dependency.

To install with Redis dependency:

pip install datasketch[redis]

To install with Cassandra dependency:

pip install datasketch[cassandra]

Project details

Release history Release notifications | RSS feed

This version


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datasketch-1.6.4.tar.gz (91.3 kB view hashes)

Uploaded source

Built Distribution

datasketch-1.6.4-py3-none-any.whl (88.3 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page