skbuild-conan: A conan extension for scikit-build

PyBind11 and scitkit-build enable us to easily write native C++-modules for Python. However, you get problems if your C++-code has dependencies. This extension tries to make defining C++-dependencies as easy as defining the Python-dependencies. This way you can easily add any C++-library that has a conan recipe to your Python-project.

This project originates from our need to use complex C++-libraries in Python projects and missing any nice option to include C++-dependencies. For a few projects, we wrote individual code to fetch the dependencies or just added instructions on how to install them (which of course can scare pure Python users).

This project is currently just a nice interface to hacks we accumulated. We try to make it as universal and robust as possible as we rely on this for multiple projects, but we are working fast-paced and will quickly abandon this tool once there is something better.

We (TU Braunschweig, Algorithms Group) are not affiliated with scikit-build or conan.

Installation

For skbuild-conan to work, you need to have a full Python and C++ development environment. This means that you need to have a compiler installed and the Python development files. The Python development files are usually called python3-dev or python3-devel and the compiler is usually called gcc or clang. On most Linux distributions, these are installed by default.

You can simply add "skbuild_conan", to requires=[...] in pyproject.toml. E.g. The pyproject.toml could look like this

[build-system]
requires = [
    "conan>=2.0.0",
    "setuptools",
    "scikit-build>=0.17.3",
    "skbuild-conan",
    "cmake>=3.23",
    "ninja",
]
build-backend = "setuptools.build_meta"

If you want to use for example setup.py build, you need to install skbuild_conan to your environment. You can do so by pip install skbuild_conan.

Transparency and Logging

skbuild-conan provides comprehensive logging to help you understand what's happening during the build process.

Using pip/setup.py verbosity flags (easiest)

skbuild-conan automatically respects standard --verbose and --quiet flags:

# Standard output (default)
pip install .

# Verbose output (shows all operations and phases)
pip install --verbose .

# Very verbose output (includes full conan output)
pip install -vv .

# Quiet output (errors only)
pip install --quiet .

The same works with setup.py:

python setup.py install --verbose
python setup.py build -vv

Using environment variable (for more control)

You can also use the SKBUILD_CONAN_LOG_LEVEL environment variable:

# Minimal output (errors only)
export SKBUILD_CONAN_LOG_LEVEL=quiet
pip install .

# Standard output (recommended)
export SKBUILD_CONAN_LOG_LEVEL=normal  # This is the default
pip install .

# Detailed output (shows all operations)
export SKBUILD_CONAN_LOG_LEVEL=verbose
pip install .

# Debug output (includes full conan output)
export SKBUILD_CONAN_LOG_LEVEL=debug
pip install .

Priority: Command-line flags take precedence over the environment variable. If you set both, the --verbose/--quiet flags will be used.

After installation, a dependency report is generated at .conan/<build_type_lowercase>/dependency-report.txt (e.g. .conan/release/dependency-report.txt) showing:

• What dependencies were requested
• What versions were resolved
• Build configuration used
• Local recipes installed

This transparency helps with:

• Understanding exactly what's being built
• Debugging version conflicts
• Security auditing
• Reproducing builds

Usage

The usage is very similar to scitkit-build (and setuptools). We just added a few additional arguments to setup().

See how to use scikit-build first, as this is just a small extension to it.

The added options are

• conanfile: Path to the folder with the conanfile.[py|txt]. By default the root is assumed. The conanfile can be used to define the dependencies. Alternatively, you can also use conan_requirements to define the conan dependencies without a conanfile. This option is exclusive. If you define conan_requirements, this option is ignored.
• conan_recipes: List of paths to further conan recipes. The conan package index is far from perfect, so often you need to build your own recipes. You don't always want to upload those, so this argument gives you the option to integrate local recipes. Just the path to the folder containing the conanfile.py. Each recipe is exported into the cache on every build via conan create --build=missing, using the same conan_profile_settings as the rest of your dependencies. Conan skips the compile when a matching binary is already cached, so a repeated build only pays for the export.
• conan_requirements: Instead of providing a conanfile, you can simply state the dependencies here. E.g. ["fmt/[>=10.0.0]"] to add fmt in version >=10.0.0.
• conan_profile_settings: Overwrite conan profile settings. You should use this to pin the C++ standard, e.g. {"compiler.cppstd": "20"} (see Setting the C++ standard). Also necessary for ABI-problems, etc.
• wrapped_setup: The setup-method that is going to be wrapped. This would allow you to extend already extended setup functions. By default, it is the setup of skbuild, which extends the setup of setuptools.
• conan_output_folder: The folder where conan will write the generated files. No real reason to change it unless the default creates conflicts with some other tool.
• cmake_args: This is actually an argument of skbuild but we will extend it. It hands cmake custom arguments. We use it to tell cmake about the conan modules.
• conan_profile: The name of the conan profile to use. By default, it is skbuild_conan_py. This profile is created automatically and should work for most cases. If you need to change it, you can do so by editing ~/.conan2/profiles/skbuild_conan_py.
• conan_env: Environment variables that are used for the conan calls. By default it will override CC and CXX with empty strings. This is necessary to work around problems with anaconda, but it should not cause any problems with other setups. You could define CONAN_HOME to ./conan/cache to use a local cache and not install anything to the user space. A relative CONAN_HOME is resolved against the current working directory, because conan itself only accepts absolute paths. Note that a project-local cache is not shared with your other projects, so everything is downloaded and built again for this one.

An example usage could be as follows

from skbuild_conan import setup
from setuptools import find_packages

setup(  # https://scikit-build.readthedocs.io/en/latest/usage.html#setup-options
    name="simple_skbuild_conan_example",
    version="0.1.1",
    packages=find_packages("src"),  # Include all packages in `./src`.
    package_dir={"": "src"},  # The root for our python package is in `./src`.
    python_requires=">=3.7",  # lowest python version supported.
    install_requires=[],  # Python Dependencies
    conan_requirements=["fmt/[>=10.0.0]"],  # C++ Dependencies
    conan_profile_settings={"compiler.cppstd": "17"},  # C++ standard
    cmake_minimum_required_version="3.23",
)

See ./examples/simple_skbuild_conan_example for a full example.

Setting the C++ standard

Always pin the C++ standard your bindings are built against.

setup(
    ...,
    conan_profile_settings={"compiler.cppstd": "20"},
)

If you do not, conan uses the value it auto-detected via conan profile detect. That value is the standard your compiler enables without any flags, which is not the latest one it supports — it can be as low as 14, and it differs between platforms and compiler versions. This is a common cause of builds that work on your machine but fail elsewhere: CGAL, for example, checks for C++17 and aborts the build if the profile says 14. Windows/MSVC and older GCC/Clang are the usual offenders.

Pinning the standard also makes the build more reproducible: the standard is part of the conan package id, so your dependencies are built and cached against the standard you asked for rather than against whatever the machine happened to detect.

Note that conan does not treat the package id as an exact requirement. If no binary matches, its binary compatibility plugin looks for a compatible one and will accept, for example, a gnu20 build for a compiler.cppstd=20 request — giving you the language extensions you just turned off, without an error. Pinning the setting everywhere (which skbuild-conan now does for local recipes too) is what keeps that fallback from being reached in the first place.

A few details worth knowing:

• "20" vs "gnu20": the gnu prefix enables compiler extensions (-std=gnu++20 instead of -std=c++20). Prefer the plain value unless you actually need the extensions. Valid values are e.g. "11", "14", "17", "20", "23" and their gnu variants.

• Do not set CMAKE_CXX_STANDARD in your CMakeLists.txt. The conan toolchain already sets it (plus CMAKE_CXX_STANDARD_REQUIRED) from compiler.cppstd. Overriding it after project() compiles your bindings against a different standard than your dependencies, which can lead to subtle ODR/ABI breakage. Conan will print a warning if you do:

  Warning: Standard CMAKE_CXX_STANDARD value defined in conan_toolchain.cmake
  to 20 has been modified to 17 by .../CMakeLists.txt
  

  If a target needs a minimum standard, declare it on the target instead — this composes with a higher value from the toolchain rather than fighting it:

  target_compile_features(_bindings PRIVATE cxx_std_17)
  

• The setting applies to everything skbuild-conan builds, including the local recipes you pass via conan_recipes.

• Changing the standard later rebuilds your dependencies. The standard is part of the package id, so conan has to build binaries it does not have yet. That is expected, and a one-off cost per standard you use.

All examples in ./examples follow this pattern.

Examples

If you do not have any C++-dependencies, you can just use scikit-build which also provides a set of examples.

Simple Example with fmt

The example in ./examples/simple_skbuild_conan_example provides a minimal example of how to use fmt in Python using PyBind11 and skbuild_conan. fmt is a nice library for formatting strings and is used by many other libraries. Python, of course, comes with extensive inbuilt string formatting, thus, this example is not very useful. However, it is a good starting point to understand how to use skbuild_conan with a simple external library.

Complex Example with CGAL: Using CGAL in Python

Sometimes, your dependencies are significantly more complex. For example, you may want to do some geometry processing and use CGAL. CGAL is a very complex library with many dependencies, but also the most powerful library for geometric operations and often the only choice for many problems. CGAL has a conan recipe, but it took a while until it received updates for conan2, such that we wrote our own recipe. In the meantime, the official recipe was updated, but for the sake of the example, we will use our own recipe. In case you are faced with the problem of an outdated conan recipe (or none at all), you can use the same trick.

See ./examples/cgal_skbuild_conan_example for an example of how to use CGAL via a custom conan recipe in Python using PyBind11 and skbuild_conan.

Note that there is also the cgalpy project by my friends at TAU (which I visited for a few months in 2022/2023), which is a nearly complete and efficient wrapper of CGAL. It may need some more documentation, but Efi put a lot of thought into efficiency and configurability.

Common problems

Feel free to copy these comments. Attribution is appreciated but not necessary.

ABI problems: Undefined symbole ...__cxx1112basic_stringIcSt11char_...

This problem should be automatically fixed. Please open an issue if you still encounter it.

See https://docs.conan.io/1/howtos/manage_gcc_abi.html for more details.

C++ standard problems: requires C++17 / no member named 'optional' in namespace 'std'

If a dependency refuses to build with a message such as

ERROR: cgal/6.0.1: Invalid: cgal requires C++17. Your compiler is set to C++14.

or your own code fails to compile with errors about C++17/C++20 features not existing, your conan profile has a lower compiler.cppstd than you expect. The auto-detected profile uses the standard that is active without any compiler flags, not the newest one the compiler supports.

Pin it explicitly in your setup.py:

setup(
    ...,
    conan_profile_settings={"compiler.cppstd": "17"},  # or "20", "23", ...
)

See Setting the C++ standard for details.

glibcxx problems:

If you get an error such as

ImportError: /home/krupke/anaconda3/envs/mo310/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /home/krupke/anaconda3/envs/mo310/lib/python3.10/site-packages/samplns/cds/_cds_bindings.cpython-310-x86_64-linux-gnu.so)

you are probably using conda (good!) but need to update glibcxx. Install the latest version by

conda install -c conda-forge libstdcxx-ng

In some cases, this still is not enough, especially if you are using a very up to date rolling-release distribution, such as Arch Linux, or if you installed libstdcxx-ng some time ago. This could lead to the system having a slightly newer version of glibcxx than conda. First try to upgrade libstdcxx-ng with

conda upgrade -c conda-forge --all

If this does not help, you can try to install g++ (caveat: Linux only, Mac OS needs clang) into your conda environment and use it to compile the package.

conda install -c conda-forge gxx_linux-64  # This should enforce a modern g++ version.
conda install -c conda-forge cxx-compiler  # This should make sure that the compiler is used.

Note that just the second command first may install an outdated g++ version (at least I observed that it installed gcc11 instead of gcc13, messing up my whole environment as this is too old). When compiling from source, you probably should delete the _skbuild-folder and do a proper uninstall of the previous installation first.

conan problems

If you encounter problems with conan, you can try to delete the conan profile and let it be recreated.

rm ~/.conan2/profiles/skbuild_conan_py

Maybe you can also just take a look at the file and see if conan detected your compiler correctly. A proper profile on Linux should for example look like this (different for other systems):

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=13
os=Linux

If the problem persists, you can try to delete the conan cache.

rm -rf ~/.conan2

fatal error: Python.h: No such file or directory

This is a rare problem with your Python installation. Make sure that you have the development files installed.

sudo apt-get install python3-dev  # Ubuntu
sudo yum install python3-devel  # CentOS, RHEL
sudo dnf install python3-devel  # Fedora
sudo zypper in python3-devel  # OpenSUSE
sudo apk add python3-dev  # Alpine
sudo apk add python3-dev  # Cygwin

In case you have multiple Python versions installed, you may need to specify the Python version.

If you still encounter problems, please open an issue.

MetadataPathFinder.invalidate_caches() missing 1 required positional argument: 'cls'

A very odd bug that we encountered with a distribution of Python 3.12 was the following:

...
TypeError: MetadataPathFinder.invalidate_caches() missing 1 required positional argument: 'cls'

ERROR: MetadataPathFinder.invalidate_caches() missing 1 required positional argument: 'cls'

The reason for this was surprisingly a bug in the Python distribution of importlib itself. Someone forgot to add the @classmethod to the MetadataPathFinder. Updating the Python distribution, in this case in conda via conda update python solved the problem.

Contributing

We welcome contributions from the community! Whether you're fixing a bug, adding a feature, or improving documentation, your help is appreciated.

Quick Start:

• 📖 Read CONTRIBUTING.md for detailed contribution guidelines
• 🔧 See DEVELOPMENT.md for technical development documentation
• 🐛 Report bugs via GitHub Issues
• 💡 Suggest features via GitHub Discussions

Good First Issues: Look for issues labeled good first issue to get started.

Please note that response times may vary as we prioritize based on available time and resources.

Changelog

• 1.6.0 C++ standard documentation plus a set of fixes around conan settings and local recipes.

  • Documented setting the C++ standard via conan_profile_settings={"compiler.cppstd": ...}, with all examples updated to pin it. (#16)
  • The examples no longer set CMAKE_CXX_STANDARD in their CMakeLists.txt, as that overrides the value the conan toolchain derives from compiler.cppstd. They declare a per-target minimum with target_compile_features instead.
  • Local recipes are now built with your conan_profile_settings. Previously only build_type was forwarded to conan create, so a recipe was built with the profile defaults. Conan's binary compatibility fallback then resolved that mismatched binary for the subsequent install — asking for compiler.cppstd=20 could silently get you a gnu20 build, i.e. language extensions you had turned off.
  • Local recipes are no longer skipped when their name/version is already in the cache. That check ignored where the cached package came from, so a local recipe was silently replaced by an unrelated remote package of the same name — the bundled CGAL recipe is cgal/6.0.1, exactly the reference ConanCenter serves. It also ignored the settings of the cached binary, so changing compiler.cppstd never rebuilt it. Every recipe is now handed to conan create --build=missing on each run; conan skips the build itself when a matching binary exists, so a warm rebuild costs an export rather than a compile.
  • conan_env is now actually applied to conan. ConanAPI() was constructed before the environment override was installed, and it resolves CONAN_HOME and loads the configuration in its constructor — so the documented conan_env={"CONAN_HOME": "./conan/cache"} silently had no effect and the user-wide cache was used instead. A relative CONAN_HOME is now resolved against the current directory, since conan itself rejects one.
  • build_type passed via conan_profile_settings is now dropped with a warning instead of emitting a conan command line that sets it twice with conflicting values. Use the --build-type argument.
  • The examples now include <fmt/format.h> instead of the <fmt/core.h> that no longer provides fmt::format since fmt 12.
  • Developer tooling was modernized: ruff (lint + format, replacing the separate pyupgrade hook) and ty (type checking) run in pre-commit, with the rule set pinned in pyproject.toml so a local ruff check matches the hook. The cmake-format hook had its additional_dependencies/types/files keys at the repo level, where pre-commit ignores them; they now sit on the hook. Type hints across the package were updated to PEP 585/604 syntax (the >=3.9 floor is kept via from __future__ import annotations).

• 1.5.0 Support for conan's cmake_layout in conanfiles. Previously, using [layout] cmake_layout in a conanfile.txt (or cmake_layout(self) in conanfile.py) caused a "conan_toolchain.cmake not found" error because the generators are placed under build/{BuildType}/generators/ instead of directly in the output folder. (#7)

• 1.4.0 Major transparency and usability improvements:

  • Structured logging with configurable verbosity levels (quiet/normal/verbose/debug)
  • Auto-detect verbosity from pip/setup.py --verbose/-v flags
  • Dependency resolution reports showing what was installed and why
  • Context-aware error messages with specific remediation suggestions
  • Input validation to catch configuration errors early
  • Automatic retry for network operations with exponential backoff
  • Version compatibility checks with warnings for known issues
  • Cross-platform color support via colorama

• 1.3.1 Replace pkg_resources with importlib.metadata to fix compatibility with setuptools v76+.

• 1.3.0 The Debug/Release will propagate to the conan profile. (thanks to @xandox)

• 1.2.0 Workaround for Windows and MSVC found by Ramin Kosfeld (TU Braunschweig).

• 1.1.1 Fixing problem if the conan default profile has been renamed via environment variable.

• 1.1.0 conan is now called directly. This is kind of hacky, but circumvents problems with conan not being in the path if only installed for build.

• 1.0.0 Custom conan profile and workaround for anaconda problem.

• 0.2.0 Improved logging.

• 0.1.4 Fixing problem with paths that contain spaces. Switching back to manual versioning.

• 0.1.3 Fixing bug if no settings are given.

• 0.1.2 Moved workaround into setup to make it more explicit.

• 0.1.1 First tested and apparently working version.