fext 0.2.0

Extensions to standard Python's heapq for performance applications

fext
----

Fast CPython extensions to Python standard library with focus on performance.

This library provides CPython native extensions to mimic some of the well known
built-in types. The implementation relies on enforced protocol - all the
objects and abstract data types are implemented in C/C++ to provide highly
effective manipulation.

Extended heapq - fext.ExtHeapQueue
==================================

The extended heap queue acts as a min-heap queue from the standard Python
library. It uses a hash table for storing information about indexes (where
values sit in the min-heap queue) to optimize removals from the heap
to O(log(N)).

.. figure:: https://raw.githubusercontent.com/thoth-station/fext/master/fig/fext_extheapq.png
:scale: 40%
  :align: center

Using fext in a C++ project
===========================

The design of this library allows you to use sources in your C++ project as
well. The ``eheapq.hpp`` file defines the extended heap queue and ``edict.hpp`` the
extended dictionary. Python files then act as a bindings to their respective
Python interfaces. Mind the API design for the templated classes - it was meant to
be used with pointers to objects (so avoid possible copy constructors).

Building the extensions
=======================

To build extensions, install the following packages (Fedora):

.. code-block:: console

dnf install -y python3-devel g++ gcc

Now you can build extensions:

.. code-block:: console

python3 setup.py build

If you would like to produce binaries with debug information:

.. code-block:: console

CFLAGS='-Wall -O0 -ggdb' python3 setup.py build

Check sections below for more info on testing the C/C++ parts of extensions.

Reference count and memory leak checks
======================================

You can find ``Makefile`` in the Git repo. This repo defines targets to
perform leak checks and reference count checks. Note they use different Python
interpreters (with/without debug information) so make sure you do not mix
virtual environments when running the tests.

.. code-block:: console

make check

Developing the extension
========================

First, prepare your environment:

.. code-block:: console

dnf install -y make
make deps

To develop or adjust sources, simply change sources and verify your
change is accepted by the test suite:

.. code-block::

make check

The ``check`` target will run the actual test suite (see also ``make test``).
Besides it, the test suite will be executed two more times to check test suite
and its behaviour with respect to Python object reference counting
(``python3-debug`` dependency will be automatically installed with the provided
``make deps``). This part of the test suite can be executed using ``make
check-refcount``. The last part of the test suite runs valgrind against the
test suite - you can explicitly trigger this part by calling ``make
check-leaks``.

Mind ``make check-refcount`` and ``make check-leaks`` will take some time given the
checks and processing that is done on the background. To verify your changes
more iterativelly, ``make test`` should do the trick (don't forget to do ``make
check`` after that though).

To clean up your environment, perform:

.. code-block:: console

make clean

Building and releasing
======================

The release can be done from a containerized environment:

.. code-block:: console

podman run --rm --workdir /io --entrypoint bash -it --volume `pwd`:/io:Z quay.io/pypa/manylinux2014_x86_64 -c "yum install -y make && make all"

To check what's happening, let's run a containerized environment - this can be
helpful when you are testing or developing the extension:

.. code-block:: console

podman run --rm --workdir /io --entrypoint bash -it --volume `pwd`:/io:Z quay.io/pypa/manylinux2014_x86_64

The following commands (run in the container stated above) will install all
the necessary tools:

.. code-block:: console

yum install -y make
make deps

Once tests pass, clean the environment:

.. code-block:: console

make clean

Now we should be ready to produce ``bdist_wheel`` and ``sdist`` distribution
for PyPI:

.. code-block:: console

python3 setup.py bdist_wheel
python3 setup.py sdist

Finally, upload artifacts to PyPI:

.. code-block:: console

auditwheel repair fext/*.whl
twine upload wheelhouse/*.whl

Alternativelly you can let ``make all`` happen.

Installation
============

The project is `hosted on PyPI <https://pypi.org/project/fext/>`_. You can
install it via ``pip`` or ``Pipenv``:

.. code-block:: console

pipenv install fext
# pip3 install fext

If there is no release conforming your system, a build process is triggered
during the installation - requires ``python3-devel`` and ``gcc/g++``.

Usage
=====

These data structures were designed for Thoth's adviser - for data kept in
resolver's internal state as well as in the reinforcement learning part.