Skip to main content

A fast implementation of Bloom filter for Python 3 built on mmap

Project description

pybloomfiltermmap3

pybloomfiltermmap3 is a Python 3 compatible fork of pybloomfiltermmap by @axiak.

The goal of pybloomfiltermmap3 is simple: to provide a fast, simple, scalable, correct library for Bloom filters in Python.

Build Status Documentation Status PyPI PyPI PyPI

Why pybloomfiltermmap3?

There are a couple reasons to use this module:

  • It natively uses mmapped files.
  • It is fast (see benchmarks).
  • It natively does the set things you want a Bloom filter to do.

Quickstart

After you install, the interface to use is a cross between a file interface and an ste interface. As an example:

    >>> import pybloomfilter
    >>> fruit = pybloomfilter.BloomFilter(100000, 0.1, '/tmp/words.bloom')
    >>> fruit.update(('apple', 'pear', 'orange', 'apple'))
    >>> len(fruit)
    3
    >>> 'mike' in fruit
    False
    >>> 'apple' in fruit
    True

To create an in-memory filter, simply omit the file location:

    >>> fruit = pybloomfilter.BloomFilter(10000, 0.1)
    >>> fruit.add('apple')
    >>> 'apple' in fruit
    True

These in-memory filters can be pickled and reloaded:

    >>> import pickle
    >>> pickled_fruit = pickle.dumps(fruit)
    >>> unpickled_fruit = pickle.loads(pickled_fruit)
    >>> 'apple' in unpickled_fruit
    True

Caveat: it is currently not possible to persist this filter later as an mmap file.

Docs

Current docs are available at pybloomfiltermmap3.rtfd.io.

Install

To install:

pip install pybloomfiltermmap3

and you should be set.

Note to Python 2 to < 3.5 users

This library is specifically meant for Python 3.5 and above. As of 2020, we strongly advise you to switch to an actively maintained distribution of Python 3. If for any reason your current environment is restricted to Python 2, please see pybloomfiltermmap. Please note that the latter is not actively maintained and will lack bug fixes and new features.

History and Future

pybloomfiltermmap is an excellent Bloom filter implementation for Python 2 by @axiak and contributors. I, @prashnts, made initial changes to add support for Python 3 sometime in 2016 as the current pybloomfiltermmap3 on PyPI. Since then, with the help of contributors, there have been incremental improvements and bug fixes while maintaining the API from versions 0.4.x and below.

Some new features and changes were first introduced in version 0.5.0. From this point on, the goal is to reach stability, as well as add a few more APIs to expand upon the use cases. While we can't guarantee that we won't change the current interface, the transition from versions 0.4.x and below should be quick one liners. Please open an issue if we broke your build!

Suggestions, bug reports, and / or patches are welcome!

Contributions and development

When contributing, you should set up an appropriate Python 3 environment and install the dependencies listed in requirements-dev.txt. Package installation depends on a generated pybloomfilter.c file, which requires Cython module to be in your current environment.

Environment setup

# Installs the venv and python3-dev packages
sudo apt install python3.10-venv python3-dev

# Creates a virtual env called "env"
python -m venv env

# Activates the created virtual env
source ./env/bin/activate

Dependencies

python -m pip install --upgrade pip
pip install cython

Build

python setup.py develop

Test

python setup.py test

Maintainers

License

See the LICENSE file. It's under the MIT License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pybloomfiltermmap3-0.6.0.tar.gz (505.3 kB view details)

Uploaded Source

File details

Details for the file pybloomfiltermmap3-0.6.0.tar.gz.

File metadata

  • Download URL: pybloomfiltermmap3-0.6.0.tar.gz
  • Upload date:
  • Size: 505.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.0 CPython/3.11.8

File hashes

Hashes for pybloomfiltermmap3-0.6.0.tar.gz
Algorithm Hash digest
SHA256 ce9dbe24c25bbbb3244e573a2bdb7af3ef203c6bc8fd512abbba1ee6248fd619
MD5 77f0aaea066352a3a4f5e476d4e9efe8
BLAKE2b-256 056a38f1518366eec574499890dc635e9fdb457197c4256f2ce45a0970bc453d

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page