funktionsklempner 1.0.0

Plumbing C++ with a Python frontend via ctypes.

Funktionsklempner

A library and framework for plumbing C++ code to Python via ctypes.

Why ctypes? It has the advantage of being widely available with Python itself, and interfaces C ABIs that can remain stable for long periods of time. This way, a compiled library can work across many different Python versions in which a dynamically linked extension would have to be recompiled due to binary incompatibility. The maintenance burden due to the recurring need to recompile packages is reduced. This comes at the cost of the additional overhead for having to pipe data through a C/ctypes API. Funktionsklempner generates this wrapping code automatically, which reduces the cost to the computational overhead. You may find Funktionsklempner useful for your open source project with low version iteration rate.

Funktionsklempner's basic workflow is as follows:

1. Declare the interface in a C++ header (i.e. functions whose equivalents shall be passed through to the Python world).
2. Declare all such headers in a project-wide funktionsklempner.toml file.
3. Generated wrapper source files via the funktionsklempner command.
4. Check the generated wrapper files and check them into your version control system.
5. Keep the auto-generated files updated via a git pre-commit hook.

Usage

Funktionsklempner assumes that you have created a C++ library with an interface that uses the Funktionsklempner (marker) types. For the sake of brevity, let's use the example in the example subdirectory; i.e. a project that has an example.cpp source file and whose interface with the Python world is designed in example.hpp. Specifically, the important part of the example's interface can be illustrated using the single function

int test_function(
    const char* name,
    size_t i,
    double* result
);

that takes a string name and an integer i, does something with it, and returns a double as result. Note in this interface the pointer-to-double type of result: to transfer values to Python via ctypes, Funktionsklempner uses preallocated ctypes variables and passes pointers to them when calling the compiled library. The C++ side is then expected to write the return value into this provided pointer. The actual return value of the function is unused; Funktionsklempner convention is to use void.

Note that the actual function in example.hpp looks differently! What I have just described is the mechanism that Funktionsklempner makes use of, which is a bit cumbersome to do by hand. To write more explicit code and to make the parser of Funktionsklempner understand the intention of this function definition, it is annoted using macros and alias types defined in funktionsklempner/funktionsklempner.hpp and funktionsklempner/scalar.hpp:

FUNKTIONSKLEMPNER_EXPORT_FUNCTION
int test_function(
    funktionsklempner::input<const char*> name,
    funktionsklempner::input<size_t> i,
    funktionsklempner::output<double> result
);

The first macro FUNKTIONSKLEMPNER_EXPORT_FUNCTION tells the parser that what follows is the declaration of a function that shall be callable from the Python world. The templated type alias funktionsklempner::input<T> tells the parser that the variables name and i are input variables passed by value T, and the alias funktionsklempner::output<T> tells the parser that result is a return value of the function (corresponding to a T* input parameter). Note that you shall name your variables when exporting via the FUNKTIONSKLEMPNER_EXPORT_FUNCTION macro.

What does Funktionsklempner do with this C++ function declaration? On the Python side, it will generate a function that takes a string name, an int i, and returns a float, i.e.

def test_function(
      name: str,
      i: int
    ) -> float:
    ...

On the C++ side, it will wrap test_function defined in the header by an exception handling function (any uncaught exceptions passing through to the Python world would crash the program!) that caches any occurring error messages. Between this C++ wrapper and the Python test_function, glue code will initialize ctypes variables and watch out for such error messages, raising errors on the Python side appropriately.

All glue code is generated via the funktionsklempner generate command after the header example.hpp has been properly described in the project-wide funktionsklempner.toml. The following two sections describe both the funktionsklempner command line tool as well as the funktionsklempner.toml configuration file format.

funktionsklempner.toml

The funktionsklempner.toml file collects all files in a source code repository that shall be wrapped via the funktionsklempner command. It has to be placed in the root of the packages source code directory and it is required for the funktionsklempner command line utility to work. As an example of its layout, consider the funktionsklempner.toml file of the example subdirectory:

[package]
name = "funkybadger"
root = "funkybadger"

[funkybadger]
[funkybadger.example]
header = "cpp/example.hpp"
sources = [ "cpp/example.cpp" ]

First, note that the [package] entry---including the name and root fields---is mandatory. The package.name field describes the name of the Python packge, and the package.root field describes the location, relative to the source code directory's root, of the package's source code (see src layout vs flat layout, where package.root would correspond to the awesome_package folder).

Second, there must be one entry per wrapped shared library. In the example, this is [funkybadger.example]. All of these entries must be under a root entry that corresponds to the package name (here [funkybadger]). The entry's key must correspond to the module name which shall be generated and from which the wrapped functions shall be imported (here the module funkybadger.example). Each shared library entry must contain two fields:

1. header, a string denoting the path to the Funktionsklempner-annotated, API-defining header file (here example/example.hpp). The path shall be relative to the source directory's root.
2. sources, a list of (strings denoting paths to) source files which encompass the shared library.

Optional fields

There are further optional fields that can be used to tweak the code that Funktionsklempner injects into the Meson build file. This way, necessary information (such as a static library that must be linked to) can be passed to the autogenerated Meson build code. The following optional fields are available:

1. link_with, a list of strings denoting static_library targets of the injection-targeted Meson build file which the Python shared library shall be linked with.
2. dependencies, a list of strings denoting dependency objects of the injection-targeted Meson build files on which the Python shared library depends.
3. wrapper_source, a file name specifying the wrapper C++ source file relative to the repository's root.
4. include_root, a directory name specifying the root include directory for the library. If not given, will default to the directory in which header resides.

As an illustration, the code

[package.library]
header = "..."
sources = [ "..." ]
link_with = [ "linklib" ]
dependencies = [ "a_dependency" ]
wrapper_source = "cpp/src/library_wrapped.cpp"

in the funktionsklempner.toml configuration file will generate a wrapper source file at cpp/src/library_wrapped.cpp (relative to the repository root), and inject the following code into the project's Meson build file:

package_library = shared_library(
    'library',
    # ...
    'cpp/src/library_wrapped.cpp',
    dependencies: [
        a_dependency,
    ],
    link_with: [
        linklib,
    ],
    # ...
)

Command line

The Funktionsklempner package comes with the funktionsklempner command line utility. This program sets up the Meson subprojects and git pre-commit hook, injects the project's main Meson build file, and generates the wrapper code. It has three main commands, two of which are typically used by users themselves.

funktionsklempner init

This command needs to be called once after the funktionsklempner.toml configuration file has been written. The command performs the following tasks:

1. Setup the funktionsklempner, objektbuch, gcem-1.18.0, and (optional) boost-1.90.0 Meson subprojects that are required for the automatic error propagation to the Python side.
2. Setup the git pre-commit hook that internally uses funktionsklempner verify to check that the generated code currently residing in the repository is up to date.
3. Start injecting the Funktionsklempner code into the project's Meson build file.

Boost will be preferably used from the system side, but particulary during the pip package building, Boost might not be found. In this case, the reduced subset of Boost (shipping code relevant to Boost.Units) will act as a fallback subproject.

funktionsklempner generate

This command is the main workhorse. It generates or updates wrapper codes for all libraries described in funktionsklempner.toml, and updates the injection into the project's Meson build file. Call this command whenever a definition in one of the API-defining headers changes.

funktionsklempner verify

This command checks whether the potential result of funktionsklempner generate is identical to what is currently found in the source tree, raising an error if not. This command is used by the git pre-commit hook to ensure that the autogenerated code stays up to date with the API header definitions.

Header annotation syntax

The introductory example lays out the basic syntax for annotating C++ headers using FUNKTIONSKLEMPNER_* macros. The following macros are available:

Macro	Purpose	Scope
FUNKTIONSKLEMPNER_EXPORT_FUNCTION	Export the following function to Python module	Top level
FUNKTIONSKLEMPNER_EXPORT_STRUCT	Export the following structure to Python module	Top level
FUNKTIONSKLEMPNER_SIZE_CONTROL("X", "Y")	Declare variable X as an (output) array whose size is controlled by the (input) array(s) or integer Y	Function export

FUNKTIONSKLEMPNER_EXPORT_FUNCTION

This function will generate a wrapper for the following function and export it to the Python module.

FUNKTIONSKLEMPNER_EXPORT_STRUCT

This function will generate a ctypes.Structure wrapping the following struct and export it to the Python module. Functions accepting pointers to this struct (including as return values!) will then be able to pass this data type to and from the Python world. This can be helpful when complex data types shall be transferred with a limited amount of ctypes calls.

FUNKTIONSKLEMPNER_SIZE_CONTROL("X", "Y")

A way to return NumPy arrays with known size. Works with and is required for funktionsklempner::OutputNDArray and funktionsklempner::OutputQuantityArray-type variables. Declares that the variable X is an array of size given by the variable Y. On the Python side, the two C++ parameters X and Y will be translated into a single NumPy array X of shape Y.shape (if Y is an input array) or (Y,) (if Y is not an input parameter of the C++ function, in which case it will be generated as an integer input on the Python side). This allows handling the frequent task of performing computational tasks on vector data kept in Numpy arrays.

The simplest way to automatically derive the size of an output array is by following the size of an input array:

FUNKTIONSKLEMPNER_EXPORT_FUNCTION
FUNKTIONSKLEMPNER_SIZE_CONTROL("out", "in_")
void a_function(
    funktionsklempner::InputNDArray<double> in_,
    funktionsklempner::OutputNDArray<double> out
);

Here, in_ is an automatically converted input array, and the return value out of the Python wrapper to this function will be a newly generated Numpy array of size out.size = in_.size.

Multidimensional size control

Sometimes, following the size of a single array will not be enough. Think of a two-parameter function evaluated on a meshgrid, for instance. For this particular use case, one can use the size multiplication syntax:

FUNKTIONSKLEMPNER_EXPORT_FUNCTION
FUNKTIONSKLEMPNER_SIZE_CONTROL("X", "Y,Z")
void a_function(
    funktionsklempner::InputNDArray<double> Y,
    funktionsklempner::InputQuantityArray<boost::units::si::length, double> Z,
    funktionsklempner::OutputNDArray<double> out
);

Here, Y is an automatically converted input array of shape Y.shape, Z is an automatically converted input array of shape Z.shape and length dimension, and the return value X of the Python wrapper to this function will be a newly generated Numpy array of shape X.shape = (*Y.shape, *Z.shape).

The size control works equivalently with QuantityArrays instead of NDArrays.

funktionsklempner::NDArray<NumPyDType T>

This class is a thin random access range wrapper around a scalar pointer (e.g., a NumPy array buffer). It is accessible in the funktionsklempner namespace as InputNDArray<T> (=NDArray<const T>, read-only access to the underlying buffer) and OutputNDArray<T> (=NDArray<T>, can be written to).

The API of the class is as follows:

template<NumPyDType T>
class NDArray
{
public:
    /* Constructors: */
    constexpr NDArray(T* data, size_t N, const char* dtype);
    constexpr NDArray(NDArray&&) noexcept = default;
    constexpr NDArray(const NDArray&) noexcept = default;

    /* Sized range: */
    constexpr size_t size() const noexcept;

    /* Random access iterators, pointers to the underlying buffer: */
    constexpr T* begin() const noexcept;
    constexpr T* end() const noexcept;

    /* Random index-based access: */
    constexpr T& operator[](size_t i) const;
};

Note that NumPyDType is a concept that captures the basic scalar types of NumPy array buffers (integer and floating point types).

funktionsklempner::QuantityArray<Unit U, NumPyDType T>

A rather thin random access wrapper around a scalar pointer that represents values of a certain physical (SI) unit via Boost.Units. Iterating over the quantity array yields value views that act, basically, as physical quantities of the given unit U (boost::unit::quantity<U, T>). It is accessible in the funktionsklempner namespace as InputQuantityArray<U, T> (=QuantityArray<U, const T>, read-only access to the underlying buffer) and OutputQuantityArray<U, T> (=QuantityArray<U, T>, can be written to).

The value type of the iterators of QuantityArray is the the QuantityView<U,T> class, which internally contains a pointer to the numerical buffer of the QuantityArray and a boost::unit::quantity<U, T> that represents the physical unit. The QuantityView can be casted to boost::unit::quantity<U, T>, and addition, subtraction, multiplication, and division operators of QuantityView<U,T> with boost::unit::quantity<U, T> are provided. In case of the OutputQuantityArray, assignment and inplace addition and subtraction operators with boost::unit::quantity<U, T> are provided as well. Essentially, QuantityArray<U,T> can be used to wrap a floating point buffer T* in a way that looks like a buffer of quantities, boost::unit::quantity<U, T>*.

The API of the class is as follows:

template<Unit U, NumPyDType T>
class QuantityArray
{
public:
    /* Constructor with given unit of same dimension as `U`. */
    constexpr QuantityArray(
        T* data,
        size_t N,
        const char* dtype,
        const char* unit
    );

    /* Constructor with default unit `U` */
    constexpr QuantityArray(
        T* data,
        size_t N,
        const char* dtype
    );

    /* Sized range: */
    constexpr size_t size() const noexcept;

    /* Random access iterators, pointing to the underlying buffer: */
    constexpr QuantityIterator<U, T> begin() const;
    constexpr const std::decay_t<T>* end() const noexcept;

    /* Random index-based access: */
    constexpr QuantityView<U, T> operator[](size_t i) const;

    /* Setting the unit to `u`, a quantity of unit `U`.
     * Note: this method is enabled only if `T` is not a const value
     *       (i.e. for OutputQuantityArray<U,T>).
     */
    constexpr void set_unit(const boost::units::quantity<U>& u);
};

Note that Unit is a concept that captures boost::units::unit<Dim, System> of some dimension Dim and system System (which essentially has to be the SI system from boost::units::si to be able to interact with the remainder of Funktionsklempner). As before, NumPyDType is a concept that captures the basic scalar types of NumPy array buffers (integer and floating point types).

Caveats

Some known caveats to keep in mind when using Funktionsklempner:

1. This uses a simplified grammar.
2. There is some effort in this code base to handle comments. However, this effort is designed empirically rather than after a standard. You might run into complications with more 'unusual' comment or preprocessor layouts. If you hit such a bug, you may try to file a bug report--but there is no guarantee that all such bugs will be fixed (in which case reduction to the simplified grammar may be the only way to run Funktionsklempner).

Note, furthermore, the terms listed in the LICENSE.

License

Funktionsklempner itself is licensed under the European Public License, v1.2 (EUPL-1.2, see LICENSE file in this directory). The template code for the code generation is licensed under the BSD 3-Clause license (BSD-3-Clause, noted in corresponding source files). The example under the example/ subfolder is equally licensed under the BSD-3-Clause license.

Funktionsklempner vendors a subset of the Boost C++ Libraries, version 1.90.0 in subprojects/packagefiles/boost-1.90.0_units.tar.gz. This code is licensed under the Boost Software License as indicated in the archive. You can find the full Boost library at boost.org.

Changelog

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[1.0.0] - 2026-03-09

Added

• Start of the changelog.