Skip to main content

Sorted containers with state-of-the-art query performance and compressed memory usage

Project description

pygm

PyGM is a Python library that enables fast query operations on sorted lists of numbers (like integers and floats) with a tiny memory overhead. Internally, it features the PGM-index, a state-of-the-art learned data structure that robustly scales to billions of elements in just a few tens of megabytes.

Build status Code coverage PyPI License GitHub stars GitHub forks

Quick start

Install with pip:

pip install pygm

PyGM supports both standard and other useful list and set operations:

>>> from pygm import SortedList, SortedSet
>>> sl = SortedList([0, 1, 34, 144, 1, 55, 233, 2, 3, 21, 89, 5, 8, 13])
>>> sl
SortedList([0, 1, 1, ..., 144, 233])
>>> sl.find_gt(9)                                   # smallest element > 9
13
>>> sl.count(1)                                     # frequency of 1
2
>>> 42 in sl                                        # membership test
False
>>> list(sl.range(0, 21, inclusive=(False, True)))  # elements 0 < x <= 21
[1, 1, 2, 3, 5, 8, 13, 21]
>>> sl[2:10:3]                                      # slicing syntax support
SortedList([1, 5, 21])
>>> (sl + [-3, -2, -1]).rank(0)                     # number of elements <= 0
4
>>> ss = SortedSet([1, 2, 3, 4]) ^ {3, 4, 5}        # set symmetric difference
>>> ss.find_lt(5)
2

The full documentation is available online and in the Python interpreter via the help() built-in function.

Installation

PyGM is compatible with Python 3.3+. The easiest way to install it is through PyPI:

pip install pygm

Otherwise, you can clone the repo, build it from source and install it as follows:

if [[ "$(uname)" == "Darwin" ]]; then brew install libomp; fi
git clone https://github.com/gvinciguerra/PyGM.git
cd PyGM
git submodule update --init --recursive
pip install .

Remember to leave the source directory PyGM/ and its parent before running Python.

Performance

Here are some plots that compare the performance of PyGM with two popular libraries, sortedcontainers and blist, on synthetic data.

Query performance of Python packages implementing sorted lists

All the graphs are log-log and show, for increasing data sizes, the average time it takes to perform each operation (lower is better). In particular, the __init__ plot shows the construction time, __contains__ measures membership queries, and __getitem__ shows the cost of accessing an element given its position.

The interesting operations on sorted lists are: (i) index, which returns the position of the first occurrence of a given element; (ii) count, which returns the number of occurrences of a given element; (iii) bisect_left, which returns the insertion point for a given value in the list to maintain the sorted order (and is used to implement find_[ge|gt|le|lt]).

You can run and plot the experiments on your computer and your data with the notebook in tests/benchmark.ipynb.

License

This project is licensed under the terms of the Apache License 2.0.

If you use the library in an academic setting, please cite the following paper:

Paolo Ferragina and Giorgio Vinciguerra. The PGM-index: a fully-dynamic compressed learned index with provable worst-case bounds. PVLDB, 13(8): 1162-1175, 2020.

@article{Ferragina:2020pgm,
  Author = {Paolo Ferragina and Giorgio Vinciguerra},
  Title = {The {PGM-index}: a fully-dynamic compressed learned index with provable worst-case bounds},
  Year = {2020},
  Volume = {13},
  Number = {8},
  Pages = {1162--1175},
  Doi = {10.14778/3389133.3389135},
  Url = {https://pgm.di.unipi.it},
  Issn = {2150-8097},
  Journal = {{PVLDB}}}

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pygm-1.0.1.tar.gz (29.0 kB view hashes)

Uploaded Source

Built Distributions

pygm-1.0.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (367.3 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pygm-1.0.1-pp310-pypy310_pp73-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded PyPy macOS 10.9+ x86-64

pygm-1.0.1-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (366.7 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pygm-1.0.1-pp39-pypy39_pp73-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded PyPy macOS 10.9+ x86-64

pygm-1.0.1-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (367.2 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pygm-1.0.1-pp38-pypy38_pp73-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded PyPy macOS 10.9+ x86-64

pygm-1.0.1-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (367.5 kB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

pygm-1.0.1-pp37-pypy37_pp73-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded PyPy macOS 10.9+ x86-64

pygm-1.0.1-cp312-cp312-musllinux_1_1_x86_64.whl (5.2 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.1+ x86-64

pygm-1.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

pygm-1.0.1-cp312-cp312-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded CPython 3.12 macOS 10.9+ x86-64

pygm-1.0.1-cp311-cp311-musllinux_1_1_x86_64.whl (5.1 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

pygm-1.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

pygm-1.0.1-cp311-cp311-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded CPython 3.11 macOS 10.9+ x86-64

pygm-1.0.1-cp310-cp310-musllinux_1_1_x86_64.whl (5.1 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

pygm-1.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

pygm-1.0.1-cp310-cp310-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded CPython 3.10 macOS 10.9+ x86-64

pygm-1.0.1-cp39-cp39-musllinux_1_1_x86_64.whl (5.1 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

pygm-1.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

pygm-1.0.1-cp39-cp39-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded CPython 3.9 macOS 10.9+ x86-64

pygm-1.0.1-cp38-cp38-musllinux_1_1_x86_64.whl (5.1 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

pygm-1.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

pygm-1.0.1-cp38-cp38-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded CPython 3.8 macOS 10.9+ x86-64

pygm-1.0.1-cp37-cp37m-musllinux_1_1_x86_64.whl (5.3 MB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ x86-64

pygm-1.0.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

pygm-1.0.1-cp37-cp37m-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded CPython 3.7m macOS 10.9+ x86-64

pygm-1.0.1-cp36-cp36m-musllinux_1_1_x86_64.whl (5.3 MB view hashes)

Uploaded CPython 3.6m musllinux: musl 1.1+ x86-64

pygm-1.0.1-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ x86-64

pygm-1.0.1-cp36-cp36m-macosx_10_9_x86_64.whl (1.6 MB view hashes)

Uploaded CPython 3.6m macOS 10.9+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page