Seamless integration of Cmake build system to setuptools/distutils
Project description
Setuptools extensions for CMake: Seamless integration of Cmake build system to setuptools
This Python package provides an extension to setuptools to integrate CMake into setuptools workflow. CMake build tool is tasked to build/install a complete Python distribution package with binary extensions and necessary data files. Then, setuptools follows to package up the bundled files for binary distribution (bdist_wheel
/bdist_egg
/etc.) or the CMake source directory for source distribution (sdist
).
Features
- setup() Wrapper:
cmaketools.setup()
wrapssetuptools.setup()
to provide one-stopsetup()
call for both CMake and setupstools. - Source Distributable:
cmaketools
let you create apip
-installable source distribution via Setuptools'sdist
command. This enables the use oftox
to perform multi-environment testing. - Automatic Source Content Detection: By taking advantage of the source directory structure imposed by CMake project,
setup.py
's critical keywords:package_dir
,packages
,package_data
,ext_modules
, andcmdclass
. - Source File Protection: Neither CMake nor Python setuptools will modify any content of the source directory under any command. It will not be cluttered by
__pycache__
or other runtime artifacts. - Git Submodule Aware: If a project contains git submodules, the submodules will be automatically cloned during
pip
installation, and the pinned commit of each submodule will be checked out before build. - Support for CMake Command-Line Options: The most of the CMake command line options are made available as the
build_ext
command options. For example,python setup.py build_ext -GNinja
will build the CMake project with Ninja build system. - Integaration of Native Code Tests: CMake ships with a test driver program, called ctest. It could be called to run the CMake build tests from Python via
cmaketools.cmakeutil.ctest()
.
Usage Examples
You can experiment cmaketools
with different Python/native interfaces availeble from following GitHub templates:
- CPython: https://github.com/python-cmaketools/cpython-example
- Pybind11: https://github.com/python-cmaketools/pybind-example
- Boost-Python: https://github.com/python-cmaketools/boost-python-example
Package Authoring Guide
Source Directory Structure
The structure of the source directory and placements of CMakeLists.txt
must adhere to the requirements below for cmaketools
to detect the package structure correctly. Here are some key tips in structuring the source directory:
- Source Directory (
src
) corresponds to the root package (orLib\site-packages
in Python directory). It could be named arbitrarily so long as it is assigned tosrc_dir
attribute ofCMakeBuilder
. - Package Directory Source subdirectories with
__init__.py
file are included inpackages
setup
argument. - Pure Python Modules Place all
.py
module files where they belong within the package structure. - Binary Extension Module To define a binary module, create a directory under a package directory it belongs to. In the example,
src/mypkg/example_module
is one such directory then we expectmypkg.example_module
binary module. Make sure the directory name matches the module name defined in C/C++ source file. Each binary module directory should containCMakeLists.txt
file which defines the library target. For example, theCMakeLists.txt
file in module directory shall callpybind11_add_module
to include apybind11
-based module to the build project. This is a requirement for the auto-configuration ofext_modules
setup
argument. - Additional Files Any "owned" additional data files needed to build the binary modules or to be used by the package shall be placed somewhere in
src
as it is the directory packaged bysdist
. - 3rd-Pary Files If downloadable or git-clonable, place them outside of
src
to keepsdist
package small. Script CMake to install them to their final in-package location to keep your package platform agnostic. This can be done via git submodules or CMakefile(DOWNLOAD <url> <file> ...)
command, then build it if necessary and install the files relative toCMAKE_INSTALL_PREFIX
. Only if they must be distributed insdist
package, place them insidesrc
.
CMakeLists.txt
Authoring Tips
First, to learn how to author CMake scripts, visit Official CMake Tutorial.
The CMake integration relies on CMake's ability to traverse directory hierarchies, i.e., to encapsulate the build process of each directory via its CMakeLists.txt
script and traverse directries. Some script snippets are repetitive and reusable as described below.
Here are general tips:
-
In general,
CMakeLists.txt
is expected in the source directory and its (sub)directories (possibly excluding resource/asset directories). ParentCMakeLists.txt
must calladd_subdirectory()
for each of its subdirectories. -
Base Source Directory shall define a
SRC_DIR
variable byset(SRC_DIR ${CMAKE_CURRENT_SOURCE_DIR})
so relative paths of subdirectories can be evaluated later.
-
Package Directories with pure Python modules must contain
install(FILES <file>...)
command to copy all.py
files to the install target folder (typicallydist/<package_name>
):file(RELATIVE_PATH DST_DIR ${SRC_DIR} ${CMAKE_CURRENT_SOURCE_DIR}) file(GLOB PYFILES LIST_DIRECTORIES false RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} "*.py") install(FILES ${PYFILES} DESTINATION ${DST_DIR} COMPONENT "PY")
Note
COMPONENT "PY"
designation ininstall
. This letssetuptools
'sbuild_py
to run CMake to install these files (andpackage_data
files). -
External Module Directories runs
add_library(<name> SHARED | MODULE ...)
command either directly or indirectly. Here, it is imperative to setname
of the library target to match its directory name. Then the target is copied to the final destination withinstall(TARGETS <target>...)
command.# match target name to folder name get_filename_component(TARGET ${CMAKE_CURRENT_SOURCE_DIR} NAME) # build commands add_library(${TARGET} ...) # typically Python3_add_library or pybind11_add_module set_target_properties(${TARGET} PROPERTIES PREFIX "${PYTHON_MODULE_PREFIX}") set_target_properties(${TARGET} PROPERTIES SUFFIX "${PYTHON_MODULE_EXTENSION}") # ... more build commands to follow # install commands get_filename_component(CURRENT_SRC_DIR ${CMAKE_CURRENT_SOURCE_DIR} DIRECTORY) if(${SRC_DIR} STREQUAL ${CURRENT_SRC_DIR}) set(DST_DIR ".") # if parent is the base source folder else() file(RELATIVE_PATH DST_DIR ${SRC_DIR} ${CURRENT_SRC_DIR}) endif() install(TARGETS ${TARGET} DESTINATION ${DST_DIR} COMPONENT "EXT")
Here we register the install as
EXT
component sobuild_ext
will only copy external modules to their final locations. -
Own Package Data Files are handled in a similar fashion as the pure Python modules with
install(FILES <file>...)
command asPY
component.# to install a package data file 'data.txt' file(RELATIVE_PATH DST_DIR ${SRC_DIR} ${CMAKE_CURRENT_SOURCE_DIR}) install(FILES "data.txt" DESTINATION ${DST_DIR} COMPONENT "PY")
-
3rd-Party Package Data Files are a bit trickier. The most intuitive way perhaps is to call the
install
command from the source folder, which matches the folder where the 3rd-party file is placed in the package. For example, suppose this skeletal directory model:# After 'cmake --install build' project-root/ ├── build/ | └── lib/ | └── 3rd-party-tool/ | └── libtool.dll # <=original ├── dist/ | └── mypkg/ | └── lib/ | └── libtool.dll # <=distro-ready file (the install destination) ├── lib/ | └── 3rd-party-tool/ # lib source files in here to be built └── src/ └── mypkg/ └── lib/ └── CMakeLists.txt # <=issue install command in this file
The source files of a 3rd-party library is included to the project via git submodule in
lib/3rd-party-tool/
and when built let's assume its DLL (assuming Windows) file will be found atbuild/lib/3rd-party-tool/libtool.dll
. We want this DLL file to be placed inlib
folder of the Python package, which means CMake must install (copy)libtool.dll
todist/mypkg/lib/libtool.dll
. The install command shall be issued bysrc/mypkg/lib/CMakeLists.txt
even ifsrc/mypkg/lib/
would otherwise be empty.# to install a package data file SET(DLL_NAME "libtool.dll") SET(DLL_PATH "${CMAKE_BINARY_DIR}/lib/3rd-party-tool/${DLL_NAME}") file(RELATIVE_PATH DST_DIR ${SRC_DIR} ${CMAKE_CURRENT_SOURCE_DIR}) install(FILES ${DLL_PATH} DESTINATION ${DST_DIR} COMPONENT "PY")
Note: Typically you can construct CMake variable via libarary's CMake variables rather than hard-coding the
DLL_PATH
as done above. -
Project Root
CMakeLists.txt
defines general configurations (such as finding dependent libraries and setting up tests) of the build project. There are a couple things could be configured here to improve the CMake/Setuptools co-operation. -
Set default install path to be
dist
so CMake by default installs to the samedist
directory location as setuptools:if (CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT) set (CMAKE_INSTALL_PREFIX "${CMAKE_SOURCE_DIR}/dist" CACHE PATH "default install path" FORCE ) endif()
setup()
Arguments
cmaketools.setup()
call wraps setuptools.setup()
so to initialize CMakeBuilder
and auto-generate setuptools.setup()
arguments. As such, it accepts most of setuptools.setup() arguments with additional arguments to configure CMake build. Also, it may overwrite user-provided setuptools arguments in order to integrate CMake.
List of New Arguments for CMake
Keyword | Type | Description |
---|---|---|
cmake_path |
str | path to cmake command (default auto-detected) |
src_dir |
str | Source directory (default "src") |
ext_module_dirs |
str[] | List of source directories defining external modules |
ext_module_hint |
str | Regex pattern to auto-detect external module directories |
test_dir |
str | Unit test directory (default "tests") |
test_submodules |
str[] | List of git submodules only used for testing |
has_package_data |
bool | Set False if project has no package_data (default True) |
skip_configure |
bool | Set True to configure cmake externally (default False) |
config |
str | Default CMake build type (default "Release") |
generator |
str | Default CMake --G argument |
platform |
str | Default CMake --platform argument |
toolset |
str | Default CMake --toolset argument |
parallel |
int > 0 | Default CMake --parallel argument |
configure_opts |
str[] | List of other default option arguments for CMake configure |
build_opts |
str[] | List of other default option arguments for CMake build |
install_opts |
str[] | List of other default option arguments for CMake install |
Overriden setuptools arguments
cmdclass
(partial override, overridesegg_info
,build_py
,build_ext
,sdist
, andinstall_data
commands)data_files
ext_modules
package_dir
package_data
packages
build_ext
Command Options for cmaketools
-based setup.py
The build_ext
command options are completely changed to accomodate CMake command-line options. Here is the output of python setup.py --help build_ext
Common commands: (see '--help-commands' for more)
setup.py build will build the package underneath 'build/'
setup.py install will install the package
Global options:
--verbose (-v) run verbosely (default)
--quiet (-q) run quietly (turns verbosity off)
--dry-run (-n) don't actually do anything
--help (-h) show detailed help message
--no-user-cfg ignore pydistutils.cfg in your home directory
Options for 'build_ext' command:
--cmake-path Name/path of the CMake executable to use, overriding
default auto-detection.
--build-lib (-b) directory for compiled extension modules
--inplace (-i) ignore build-lib and put compiled extensions into the
source directory alongside your pure Python modules
--force (-f) forcibly build everything (delete existing
CMakeCache.txt)
--cache (-C) Pre-load a CMake script to populate the cache.
--define (-D) Create or update a CMake CACHE entry (separated by
';')
--undef (-U) Remove matching entries from CMake CACHE.
--generator (-G) Specify a build system generator.
--toolset (-T) Toolset specification for the generator, if supported.
--platform (-A) Specify platform name if supported by generator.
--Wno-dev Suppress developer warnings.
--Wdev Enable developer warnings.
--Werror Make specified warnings into errors: dev or
deprecated.
--Wno-error Make specified warnings not errors.
--Wdeprecated Enable deprecated functionality warnings.
--Wno-deprecated Suppress deprecated functionality warnings.
--log-level Set the log level to one of: ERROR, WARNING, NOTICE,
STATUS, VERBOSE, DEBUG, TRACE
--log-context Enable the message() command outputting context
attached to each message.
--debug-trycompile Do not delete the try_compile() build tree. Only
useful on one try_compile() at a time.
--debug-output Put cmake in a debug mode.
--debug-find Put cmake find commands in a debug mode.
--trace Put cmake in trace mode.
--trace-expand Put cmake in trace mode with variables expanded.
--trace-format Put cmake in trace mode and sets the trace output
format.
--trace-source Put cmake in trace mode, but output only lines of a
specified file.
--trace-redirect Put cmake in trace mode and redirect trace output to a
file instead of stderr.
--warn-uninitialized Specify a build system generator.
--warn-unused-vars Warn about unused variables.
--no-warn-unused-cli Don’t warn about command line options.
--check-system-vars Find problems with variable usage in system files.
--parallel (-j) The maximum number of concurrent processes to use when
building.
--config For multi-configuration tools, choose this
configuration.
--clean-first Build target clean first, then build.
--verbose (-v) Enable verbose output - if supported - including the
build commands to be executed.
--strip Strip before installing.
--help-generator list available compilers
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cmaketools-0.1.5-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 137e427692b2954195f33889ef38cfa90da02b309552449b4a69bc4d6c6b3eaf |
|
MD5 | b7eabd355513e68d4e355f2ae892a244 |
|
BLAKE2b-256 | 1ad64d3e3adea0cb30d99402c02ad8e43fa6b3b46a9fe6f97f1f635518e9e850 |