cmake-build-extension

Setuptools extension to build and package CMake projects.

This project aims to simplify the integration of C++ projects based on CMake with Python packaging tools. CMake provides out-of-the-box support to either SWIG and pybind11, that are two among the most used projects to create Python bindings from C++ sources.

If you have any experience with these hybrid projects, you know the challenges to make packaging right! This project takes inspiration from pre-existing examples (pybind/cmake_example, among many others) and provides a simple, flexible, and reusable setuptools extension with the following features:

• Bridge between CMake projects and Python packaging.
• Configure and build the CMake project from setup.py.
• Install the CMake project in the resulting Python package.
• Allow passing custom CMake options.
• Allow creating a top-level __init__.py.
• Expose C++ executables to the Python environment.
• Provide a context manager to import reliably CPython modules on all major OSs.
• Disable the C++ extension in editable installations (requiring to manually call CMake to install the C++ project).

Have a look to the example for an overview of what can be done with this extension. It shows how to create SWIG and pybind11 bindings for a project composed by a small C++ library with NumPy support and an executable.

Advanced features

1. This extension supports creating packages PEP517 and PEP518 compliant (more details).
2. If the CMake project exports the targets, downstream projects can:
   1. Extend their CMAKE_MODULE_PATH with the root of your installed Python package, that could be obtained with:

      python -c "import <pkg>, pathlib; print(pathlib.Path(<pkg>.__file__).parent)"
      

      and consume the exported CMake targets.
   2. Use cmake-build-extension with the cmake_depends_on option and link against the exported CMake targets during the downstream packaging.

Note that the second feature allows distributing C++ dependencies through PyPI. The resulting package structure is similar to other projects like pybind11 and CasADi. Be aware that ensuring ABI compatibility could be problematic in edge cases, and the usage of a proper compatible release pattern (~=) could be necessary.

Installation

From PyPI:

pip install cmake-build-extension

From the repository:

pip install git+https://github.com/diegoferigo/cmake-build-extension

Usage

Once both CMake project and setup.py|setup.cfg|pyproject.toml of your hybrid package are correctly configured to use the resources provided by cmake-build-extension, the following commands can be used:

# ============
# Create sdist
# ============

# Calling setup.py
python setup.py sdist

# Using pypa/build
python -m build --sdist

# ============
# Create wheel
# ============

# Calling setup.py
python setup.py bdist_wheel

# Using pip
pip wheel -w dist/ .

# Using pypa/build
python -m build --wheel

# ==========================================================
# Create wheel or install package passing additional options
# ==========================================================

# Calling setup.py
python setup.py {bdist_wheel|install} build_ext -D"BAR=Foo;VAR=TRUE"

# Using pip
pip {wheel|install} --global-option="build_ext" --global-option="-DBAR=Foo;VAR=TRUE" .

# Using pypa/build (only wheel creation)
python -m build --wheel "-C--global-option=build_ext" "-C--global-option=-DBAR=Foo;VAR=TRUE"

Caveats

manylinux* support

This extension, beyond packaging the hybrid C++ / Python project, also allows the inclusion of the exported CMake targets in the resulting wheel. This result depends on how the CMake project is configured, and whether the exported targets are installed together with the other files.

Such hybrid packages can then be uploaded to PyPI. Though, on GNU/Linux, the generated wheel is not compliant by default with any manylinux* standard. Tools such auditwheel exist to fix these wheels, but they require running on selected distributions. Luckily, other projects like cibuildwheel greatly simplify the process in CI.

This being said, manylinux* guidelines could still work against you. In fact, wheels supporting manylinux2010|manylinux2014 are built with gcc4 that does not support the new C++11 ABIs. In few words, this means that the exported libraries bundled in the wheel cannot be imported in a downstream project using relatively new C++ standards! For more details visit robotology/idyntree#776.

Luckily, the situation changed thanks to the finalization of PEP600, i.e. manylinuxX_YY :tada: If you build a PEP600 compliant wheel (nowadays compatible with most of the commonly used distributions), your exported CMake project bundled in the wheel can be successfully imported downstream. If you want to support this use case, make sure to produce and distribute wheels compliant with PEP600.

Loading CPython modules in Windows

Python 3.8 changed how DLL are resolved. By default, modules that could be imported in Python 3.7 stopped working, and using the new os.add_dll_directory is now necessary.

In order to streamline the process, cmake-build-extension implements a context manager that can be used to import reliably the bindings module:

import cmake_build_extension

with cmake_build_extension.build_extension_env():
    from . import bindings

It will take care to temporarily fix the search path.

For more details, refer to #8 and #12.

setup.py|setup.cfg|pyproject.toml files in subfolder

Sometimes hybrid projects are C++ centric, and keeping these files in the top-level folder is not desirable. In this setup, however, problems occur if the main CMakeLists.txt is kept in the top-level folder (see pypa/build#322). To solve this problem, cmake-build-extension provides custom commands to create source distribution. You can use one of the following custom sdist options in setup.py:

setuptools.setup(
    cmdclass=dict(
        # [...]
        # Pack the whole git folder:
        sdist=cmake_build_extension.GitSdistFolder,
        # Pack only the git tree:
        sdist=cmake_build_extension.GitSdistTree,
        # Or, inherit from cmake_build_extension.sdist_command.GitSdistABC and
        # make your own custom sdist including only the files you are interested
    ),
)

Downstream projects

If the provided example is not enough complex, find here below a list of projects using cmake-build-extension:

• robotology/idyntree
• robotology/yarp
• robotology/ycm
• diegoferigo/gazebo-yarp-synchronizer
• robotology/gym-ignition@scenario
• dic-iit/gazebo-scenario-plugins
• dic-iit/bipedal-locomotion-framework
• artivis/manif

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

License

MIT