https://static.pepy.tech/badge/datasketch/month https://zenodo.org/badge/DOI/10.5281/zenodo.598238.svg https://codecov.io/gh/ekzhu/datasketch/branch/master/graph/badge.svg

datasketch gives you probabilistic data structures that can process and search very large amount of data super fast, with little loss of accuracy.

This package contains the following data sketches:

Data Sketch

Usage

MinHash

estimate Jaccard similarity and cardinality

Weighted MinHash

estimate weighted Jaccard similarity

HyperLogLog

estimate cardinality

HyperLogLog++

estimate cardinality

The following indexes for data sketches are provided to support sub-linear query time:

Index

For Data Sketch

Supported Query Type

MinHash LSH

MinHash, Weighted MinHash

Jaccard Threshold

LSHBloom

MinHash, Weighted MinHash

Jaccard Threshold

MinHash LSH Forest

MinHash, Weighted MinHash

Jaccard Top-K

MinHash LSH Ensemble

MinHash

Containment Threshold

HNSW

Any

Custom Metric Top-K

datasketch must be used with Python 3.9 or above, NumPy 1.11 or above, and Scipy.

Note that MinHash LSH and MinHash LSH Ensemble also support Redis and Cassandra storage layer (see MinHash LSH at Scale).

Install

To install datasketch using pip:

pip install datasketch

This will also install NumPy as dependency.

To install with Redis dependency:

pip install datasketch[redis]

To install with Cassandra dependency:

pip install datasketch[cassandra]

To install with Bloom filter dependency:

pip install datasketch[bloom]

Contributing

We welcome contributions from everyone. Whether you’re fixing bugs, adding features, improving documentation, or helping with tests, your contributions are valuable.

Development Setup

The project uses uv for fast and reliable Python package management. Follow these steps to set up your development environment:

  1. Install uv: Follow the official installation guide at https://docs.astral.sh/uv/getting-started/installation/

  2. Clone the repository:

    git clone https://github.com/ekzhu/datasketch.git
cd datasketch

  3. Set up the environment:

    # Create a virtual environment
# (Optional: specify Python version with --python 3.x)
uv venv
# Activate the virtual environment (optional, uv run commands work without it)
source .venv/bin/activate

# Install all dependencies
uv sync

  4. Verify installation:

    # Run tests to ensure everything works
uv run pytest

  5. Optional dependencies (for specific development needs):

    # For testing
uv sync --extra test

# For Cassandra support
uv sync --extra cassandra

# For Redis support
uv sync --extra redis

# For all extras
uv sync --all-extras

Learn more about uv at https://docs.astral.sh/uv/

Development Workflow

  1. Fork the repository on GitHub if you haven’t already.

  2. Create a feature branch for your changes:

    git checkout -b feature/your-feature-name
# Or for bug fixes:
git checkout -b fix/issue-description

  3. Make your changes following the project’s coding standards.

  4. Run the tests to ensure nothing is broken:

    uv run pytest

  5. Check code quality with ruff:

    # Check for issues
uvx ruff check .

# Auto-fix formatting issues
uvx ruff format .

  6. Commit your changes with a clear, descriptive commit message:

    git commit -m "Add feature: brief description of what was changed"

  7. Push to your fork and create a pull request on GitHub:

    git push origin your-branch-name

  8. Respond to feedback from maintainers and iterate on your changes.

Guidelines

  • Follow PEP 8 style guidelines

  • Write tests for new features

  • Update documentation as needed

  • Keep commits focused and atomic

  • Be respectful in discussions

For more information, check the GitHub issues for current priorities or areas needing help. You can also join the discussion on project roadmap and priorities.

Project links

Data verified by PyPI on Jul 5, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for ekzhu from gravatar.com ekzhu

Credits

Author: ekzhu

GitHub Statistics

Data verified by PyPI on Jul 5, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License

MIT License (MIT)

Requires

Python >=3.9

Provides Extra

aio benchmark bloom cassandra experimental-aio redis test

Tags

database datamining

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datasketch-2.0.0.tar.gz (98.7 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

datasketch-2.0.0-py3-none-any.whl (107.2 kB view details)

Uploaded Python 3

File details

Details for the file datasketch-2.0.0.tar.gz.

File metadata

  • Download URL: datasketch-2.0.0.tar.gz
  • Upload date:
  • Size: 98.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for datasketch-2.0.0.tar.gz
Algorithm Hash digest
SHA256 e0570e170f7e64b8d6fb1cc2e4ce36a9f7036c5100167e50a0770addc50558c2
MD5 aeae38f9060b8d71f0b14288225282a4
BLAKE2b-256 daaebbcdeab67ebfe27747896618bca0f8113e79d534c5af2b06ce6693c981de

See more details on using hashes here.

Provenance

The following attestation bundles were made for datasketch-2.0.0.tar.gz:

Publisher: pypi.yml on ekzhu/datasketch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file datasketch-2.0.0-py3-none-any.whl.

File metadata

  • Download URL: datasketch-2.0.0-py3-none-any.whl
  • Upload date:
  • Size: 107.2 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.13

File hashes

Hashes for datasketch-2.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 aea5ffafcce776e03d085740e78b874e778d779b07ee11ca636ca51b3fef09ed
MD5 a1175f9f5ebae6eda24890a1592e96fb
BLAKE2b-256 54df9e79bf1fb2f660507dbd00d24730d107595a2f599f2524a96344149c95f1

See more details on using hashes here.

Provenance

The following attestation bundles were made for datasketch-2.0.0-py3-none-any.whl:

Publisher: pypi.yml on ekzhu/datasketch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

2.0.0

1.10.0

1.9.0

1.8.0

1.7.0

1.6.5

1.6.4

1.6.3

1.6.2

1.6.1

1.6.0

1.5.9

1.5.8

1.5.7

1.5.6

1.5.4

1.5.3

1.5.2

1.5.1

1.5.0

1.4.11

1.4.10

1.4.9

1.4.8

1.4.7

1.4.5

1.4.4

1.4.3

1.4.2

1.4.1

1.4.0

1.3.0

1.2.10

1.2.9

1.2.8

1.2.7

1.2.6

1.2.5

1.2.4

1.2.3

1.2.2

1.2.1

1.2.0

1.1.3

1.1.2

1.1.1

1.1.0

0.2.6

0.2.5

0.2.4

0.2.3

0.2.1

0.2.0

0.1.36

0.1.35

0.1.34

0.1.33

0.1.32

0.1.31

0.1.30

0.1.29

0.1.28

0.1.27

0.1.26

0.1.25

0.1.24

0.1.23

0.1.22

0.1.21

0.1.20

0.1.19

0.1.18

0.1.17

0.1.16

0.1.15

0.1.14

0.1.13

0.1.12

0.1.11

0.1.10

0.1.9

0.1.8

0.1.7

0.1.6

0.1.5

0.1.4

0.1.3

0.1.2

0.1.1

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page