datasketch: Big Data Looks Small

https://static.pepy.tech/badge/datasketch/month

https://zenodo.org/badge/DOI/10.5281/zenodo.598238.svg

https://codecov.io/gh/ekzhu/datasketch/branch/master/graph/badge.svg

datasketch gives you probabilistic data structures that can process and search very large amount of data super fast, with little loss of accuracy.

This package contains the following data sketches:

Data Sketch	Usage
MinHash	estimate Jaccard similarity and cardinality
Weighted MinHash	estimate weighted Jaccard similarity
HyperLogLog	estimate cardinality
HyperLogLog++	estimate cardinality

The following indexes for data sketches are provided to support sub-linear query time:

Index	For Data Sketch	Supported Query Type
MinHash LSH	MinHash, Weighted MinHash	Jaccard Threshold
LSHBloom	MinHash, Weighted MinHash	Jaccard Threshold
MinHash LSH Forest	MinHash, Weighted MinHash	Jaccard Top-K
MinHash LSH Ensemble	MinHash	Containment Threshold
HNSW	Any	Custom Metric Top-K

datasketch must be used with Python 3.9 or above, NumPy 1.11 or above, and Scipy.

Note that MinHash LSH and MinHash LSH Ensemble also support Redis and Cassandra storage layer (see MinHash LSH at Scale).

Install

To install datasketch using pip:

pip install datasketch

This will also install NumPy as dependency.

To install with Redis dependency:

pip install datasketch[redis]

To install with Cassandra dependency:

pip install datasketch[cassandra]

To install with Bloom filter dependency:

pip install datasketch[bloom]

Contributing

We welcome contributions from everyone. Whether you’re fixing bugs, adding features, improving documentation, or helping with tests, your contributions are valuable.

Development Setup

The project uses uv for fast and reliable Python package management. Follow these steps to set up your development environment:

Install uv: Follow the official installation guide at https://docs.astral.sh/uv/getting-started/installation/

Clone the repository:

git clone https://github.com/ekzhu/datasketch.git
cd datasketch

Set up the environment:

# Create a virtual environment
# (Optional: specify Python version with --python 3.x)
uv venv
# Activate the virtual environment (optional, uv run commands work without it)
source .venv/bin/activate

# Install all dependencies
uv sync

Verify installation:

# Run tests to ensure everything works
uv run pytest

Optional dependencies (for specific development needs):

# For testing
uv sync --extra test

# For Cassandra support
uv sync --extra cassandra

# For Redis support
uv sync --extra redis

# For all extras
uv sync --all-extras

Learn more about uv at https://docs.astral.sh/uv/

Development Workflow

Fork the repository on GitHub if you haven’t already.

Create a feature branch for your changes:

git checkout -b feature/your-feature-name
# Or for bug fixes:
git checkout -b fix/issue-description

Make your changes following the project’s coding standards.
Run the tests to ensure nothing is broken:
```
uv run pytest
```

Check code quality with ruff:

# Check for issues
uvx ruff check .

# Auto-fix formatting issues
uvx ruff format .

Commit your changes with a clear, descriptive commit message:

git commit -m "Add feature: brief description of what was changed"

Push to your fork and create a pull request on GitHub:
```
git push origin your-branch-name
```
Respond to feedback from maintainers and iterate on your changes.

Guidelines

Follow PEP 8 style guidelines
Write tests for new features
Update documentation as needed
Keep commits focused and atomic
Be respectful in discussions

For more information, check the GitHub issues for current priorities or areas needing help. You can also join the discussion on project roadmap and priorities.

Metadata

Release files for datasketch 2.0.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for datasketch 2.0.0
File	Size	Uploaded
datasketch-2.0.0.tar.gz	98.7 kB	Jul 5, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for datasketch 2.0.0
File	Interpreter	ABI	Platform	Reset
datasketch-2.0.0-py3-none-any.whl 107.2 kB Jul 5, 2026	Python 3	none	any	Details

Total release size: 205.9 kB

Release files / datasketch-2.0.0.tar.gz

Download URL	datasketch-2.0.0.tar.gz
Size	98.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`e0570e170f7e64b8d6fb1cc2e4ce36a9f7036c5100167e50a0770addc50558c2`
BLAKE2b-256 checksum How to use checksums	`daaebbcdeab67ebfe27747896618bca0f8113e79d534c5af2b06ce6693c981de`
Upload date	Jul 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.13`

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/ekzhu/datasketch
Publishing commit: github.com/ekzhu/datasketch/tree/3da96a591c954952c9dd367567db720cfdc151e0

Workflow

Publishing configuration: github.com/ekzhu/datasketch/blob/3da96a591c954952c9dd367567db720cfdc151e0/.github/workflows/pypi.yml
Publishing logs: github.com/ekzhu/datasketch/actions/runs/28732165939/attempts/1

Artifact

Subject: datasketch-2.0.0.tar.gz
SHA-256 checksum: e0570e170f7e64b8d6fb1cc2e4ce36a9f7036c5100167e50a0770addc50558c2 Verifying attestations

Release files / datasketch-2.0.0-py3-none-any.whl

Download URL	datasketch-2.0.0-py3-none-any.whl
Size	107.2 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`aea5ffafcce776e03d085740e78b874e778d779b07ee11ca636ca51b3fef09ed`
BLAKE2b-256 checksum How to use checksums	`54df9e79bf1fb2f660507dbd00d24730d107595a2f599f2524a96344149c95f1`
Upload date	Jul 5, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`twine/6.1.0 CPython/3.13.13`

Provenance

Identity

Publishing platform: GitHub Actions
Publishing repository: github.com/ekzhu/datasketch
Publishing commit: github.com/ekzhu/datasketch/tree/3da96a591c954952c9dd367567db720cfdc151e0

Workflow

Publishing configuration: github.com/ekzhu/datasketch/blob/3da96a591c954952c9dd367567db720cfdc151e0/.github/workflows/pypi.yml
Publishing logs: github.com/ekzhu/datasketch/actions/runs/28732165939/attempts/1

Artifact

Subject: datasketch-2.0.0-py3-none-any.whl
SHA-256 checksum: aea5ffafcce776e03d085740e78b874e778d779b07ee11ca636ca51b3fef09ed Verifying attestations

Release history Release notifications | RSS feed

This release

2.0.0 This release

Jul 5, 2026 2 release files

1.10.0

Apr 17, 2026 2 release files

1.9.0

Jan 18, 2026 2 release files

1.8.0

Nov 28, 2025 2 release files

1.7.0

Nov 5, 2025 2 release files

1.6.5

Jun 4, 2024 2 release files

1.6.4

Oct 3, 2023 2 release files

1.6.3

Sep 12, 2023 2 release files

1.6.2

Sep 6, 2023 2 release files

1.6.1

Sep 6, 2023 2 release files

1.6.0

Aug 30, 2023 2 release files

1.5.9

Feb 19, 2023 2 release files

1.5.8

Aug 21, 2022 2 release files

1.5.7

Feb 4, 2022 2 release files

1.5.6

Dec 27, 2021 2 release files

1.5.4

Dec 4, 2021 2 release files

1.5.3

Dec 23, 2020 2 release files

1.5.2

Dec 15, 2020 2 release files

1.5.1

Mar 12, 2020 2 release files

1.5.0

Nov 26, 2019 2 release files

1.4.11

Nov 24, 2019 2 release files

1.4.10

Sep 20, 2019 2 release files

1.4.9

Sep 20, 2019 2 release files

1.4.8

Sep 5, 2019 2 release files

1.4.7

Jul 11, 2019 2 release files

1.4.5

May 31, 2019 2 release files

1.4.4

May 26, 2019 2 release files

1.4.3

Mar 25, 2019 2 release files

1.4.2

Mar 3, 2019 2 release files

1.4.1

Jan 17, 2019 2 release files

1.4.0

Jan 6, 2019 2 release files

1.3.0

Dec 27, 2018 2 release files

1.2.10

Nov 2, 2018 2 release files

1.2.9

Oct 22, 2018 2 release files

1.2.8

Oct 17, 2018 1 release file

1.2.7

Jul 27, 2018 2 release files

1.2.6

Jul 21, 2018 2 release files

1.2.5

Nov 21, 2017 2 release files

1.2.4

Nov 8, 2017 2 release files

1.2.3

Sep 8, 2017 2 release files

1.2.2

Sep 7, 2017 1 release file

1.2.1

Apr 26, 2017 1 release file

1.2.0

Mar 31, 2017 1 release file

1.1.3

Mar 26, 2017 1 release file

1.1.2

Mar 15, 2017 1 release file

1.1.1

Mar 15, 2017 1 release file

1.1.0

Mar 11, 2017 1 release file

0.2.6

Jan 7, 2017 2 release files

0.2.5

Nov 12, 2016 1 release file

0.2.4

Jul 10, 2016 1 release file

0.2.3

Apr 12, 2016 1 release file

0.2.1

Mar 18, 2016 1 release file

0.2.0

Mar 18, 2016

0.1.36

Mar 17, 2016 1 release file

0.1.35

Jan 29, 2016 1 release file

0.1.34

Jan 26, 2016 1 release file

0.1.33

Jan 20, 2016 1 release file

0.1.32

Jan 9, 2016 1 release file

0.1.31

Jan 9, 2016 1 release file

0.1.30

Dec 9, 2015 1 release file

0.1.29

Dec 8, 2015 1 release file

0.1.28

Dec 8, 2015 1 release file

0.1.27

Dec 7, 2015 1 release file

0.1.26

Aug 10, 2015 1 release file

0.1.25

Jul 24, 2015 1 release file

0.1.24

Jul 13, 2015 1 release file

0.1.23

Jul 13, 2015 1 release file

0.1.22

Jul 8, 2015 1 release file

0.1.21

Jul 8, 2015 1 release file

0.1.20

Jul 7, 2015 1 release file

0.1.19

Jun 25, 2015 1 release file

0.1.18

Jun 25, 2015 1 release file

0.1.17

Jun 12, 2015 1 release file

0.1.16

Jun 12, 2015 1 release file

0.1.15

Jun 10, 2015 1 release file

0.1.14

Jun 9, 2015 1 release file

0.1.13

Jun 9, 2015 1 release file

0.1.12

Jun 2, 2015 1 release file

0.1.11

May 13, 2015 1 release file

0.1.10

May 11, 2015 1 release file

0.1.9

Apr 20, 2015 1 release file

0.1.8

Apr 8, 2015 1 release file

0.1.7

Apr 7, 2015 1 release file

0.1.6

Apr 5, 2015 1 release file

0.1.5

Apr 5, 2015 1 release file

0.1.4

Apr 5, 2015 1 release file

0.1.3

Apr 1, 2015 1 release file

0.1.2

Apr 1, 2015 1 release file

0.1.1

Apr 1, 2015 1 release file

datasketch 2.0.0

datasketch: Big Data Looks Small

Install

Contributing

Development Setup

Development Workflow

Guidelines

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License

Requires

Provides Extra

Tags

Classifiers

Metadata

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License

Requires

Provides Extra

Tags

Classifiers

Release files for datasketch 2.0.0

Source distribution (sdist)

Built distribution (wheel)

Release files / datasketch-2.0.0.tar.gz

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release files / datasketch-2.0.0-py3-none-any.whl

Provenance

PyPI Publish Attestation

Identity

Workflow

Artifact

Release history Release notifications | RSS feed