Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Python wrappers for generating CTest and submitting to CDash without a CMake build system

Project description

PyCTest

Travis AppVeyor
Build Status Build status

PyCTest Documentation (readthedocs)

GitHub

  • git clone https://github.com/jrmadsen/pyctest.git

PyPi

  • pip install pyctest

Anaconda Cloud

  • conda install -c jrmadsen pyctest
Name Version Platforms Downloads
Conda Recipe Anaconda-Server Badge Anaconda-Server Badge Anaconda-Server Badge

conda-forge

  • conda install -c conda-forge pyctest
Name Version Platforms Downloads
Conda Recipe Conda Version Conda Platforms Conda Downloads

Benefits

  • Integration into continuous integration systems (e.g. Travis, AppVeyor, CircleCI, etc.) and pushing to CDash dashboard will combine all the results in one place
    • The warnings and errors are enumerated in CDash (no more parsing stdout logs for errors)
  • Easily create platform-independent testing
  • No need to migrate build system to CMake -- just specify pyctest.BUILD_COMMAND

General Setup

  • Add PyCTest to conda environment: conda install pyctest
  • Write a driver Python script, e.g. pyctest-runner.py, in top of the source code tree

Example for Python project

The following is an example for a Python code with a compiled C extension that uses nosetests for unit-testing:

#!/usr/bin/env python

import os, sys, platform
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(),binary_dir=os.getcwd(), vcs_type="git")
args = parser.parse_args()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.BUILD_COMMAND = "python setup.py build_ext --inplace"

test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["nosetests"])

pyctest.run()

Example for autotools project

#!/usr/bin/env python

import os, sys, platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

parser = helpers.ArgumentParser("ProjectName", source_dir=os.getcwd(), binary_dir=os.getcwd(),
                                vcs_type="git")
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()

# CONFIGURE_COMMAND can only run one command so if autogen is required, just execute it here
cmd = pyctest.command(["./autogen.sh"])
cmd.SetWorkingDirectory(pyctest.SOURCE_DIRECTORY)
cmd.SetErrorQuiet(False)
cmd.Execute()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.CONFIGURE_COMMAND = "./configure"
pyctest.BUILD_COMMAND = "make -j{}".format(mp.cpu_count())

test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["./run-testing.sh"])

pyctest.run()

Example for CMake project

#!/usr/bin/env python

import os
import sys
import platform
import multiprocessing as mp
import pyctest.pyctest as pyctest
import pyctest.helpers as helpers

binary_dir = os.path.join(os.getcwd(), "build-ProjectName")
parser = helpers.ArgumentParser("ProjectName", os.getcwd(), binary_dir)
parser.add_argument("-n", "--build", type=str, required=True, help="Build name for identification")
args = parser.parse_args()

pyctest.BUILD_NAME = "{}".format(args.build)
pyctest.UPDATE_COMMAND = "git"
pyctest.CONFIGURE_COMMAND = "cmake {}".format(pyctest.SOURCE_DIRECTORY)
pyctest.BUILD_COMMAND = "cmake --build {} --target all -- -j{}".format(pyctest.BINARY_DIRECTORY, mp.cpu_count())

test = pyctest.test()
test.SetName("unittest")
# insert the command to run the tests for project
test.SetCommand(["./run-testing.sh"])

pyctest.run()

Python Modules

  • import pyctest -- global package
  • import pyctest.pyctest -- CTest module
  • import pyctest.pycmake -- CMake module
  • import pyctest.helpers -- Helpers module
    • includes command line arguments (argparse) for PyCTest

Direct Access to CMake/CTest/CPack Executables

  • python -m pyctest.cmake <ARGS> == cmake <ARGS>
  • python -m pyctest.ctest <ARGS> == ctest <ARGS>
  • python -m pyctest.cpack <ARGS> == cpack <ARGS>

Following Python code:

from pyctest.ctest import CTest
from pyctest.cmake import CMake
from pyctest.cpack import CPack

CMake({"CMAKE_BUILD_TYPE":"Release"}, os.getcwd(), "-G", "Ninja")
CTest("--build-and-test", os.getcwd(), "-VV")
CPack("-G", "TGZ")

is equivalent to the following shell commands:

$ cmake -DCMAKE_BUILD_TYPE=Release ${PWD} -G Ninja
$ ctest --build-and-test ${PWD} -VV
$ cpack -G TGZ

Standard Configuration Variables

  • pyctest.PROJECT_NAME
  • pyctest.SOURCE_DIRECTORY
  • pyctest.BINARY_DIRECTORY
  • pyctest.SITE
  • pyctest.BUILD_NAME
  • pyctest.TRIGGER
  • pyctest.CHECKOUT_COMMAND
  • pyctest.BUILD_COMMAND
  • pyctest.MODEL
  • pyctest.CUSTOM_COVERAGE_EXCLUDE
  • pyctest.CUSTOM_MAXIMUM_NUMBER_OF_ERRORS
  • pyctest.CUSTOM_MAXIMUM_NUMBER_OF_WARNINGS
  • pyctest.CUSTOM_MAXIMUM_PASSED_TEST_OUTPUT_SIZE

Setting Arbitrary Variables

    pyctest.set("CTEST_TOKEN_FILE", "${CMAKE_CURRENT_LIST_DIR}/.ctest-token")

Generating a Test

    test = pyctest.test()
    test.SetName("nosetests")
    test.SetCommand(["nosetests", "test", "--cover-xml", "--cover-xml-file=coverage.xml"])
    # set directory to run test
    test.SetProperty("WORKING_DIRECTORY", pyctest.BINARY_DIRECTORY)
    test.SetProperty("RUN_SERIAL", "ON")
    test.SetProperty("ENVIRONMENT", "OMP_NUM_THREADS=1")

Examples

CDash Integration Example

Results from running the TomoPy example can be found at the TomoPy CDash Testing Dashboard @ NERSC

  • Python code with C extensions without CMake build system
  • The build logs from "python setup.py install" are registered in the "Build" section
  • The nosetests test command + other are wrapped into CTests

Testing Example

PyCTest can be used to simple execute tests and submit to a dashboard without any configuration, build, etc. steps

#!/usr/bin/env python

import os
import sys
import shutil
import argparse
import platform
import traceback

import pyctest.pyctest as pyctest
import pyctest.pycmake as pycmake
import pyctest.helpers as helpers

if __name__ == "__main__":

    directory = os.path.join(os.getcwd(), "pycm-test")

    # these are required
    pyctest.PROJECT_NAME = "PyCTest"
    pyctest.SOURCE_DIRECTORY = directory
    pyctest.BINARY_DIRECTORY = directory

    args = helpers.ArgumentParser(pyctest.PROJECT_NAME,
                                  pyctest.SOURCE_DIRECTORY,
                                  pyctest.BINARY_DIRECTORY).parse_args()

    # set explicitly
    pyctest.MODEL = "Continuous"
    pyctest.SITE = platform.node()

    # create a Test object
    test = pyctest.test()
    test.SetName("list_directory")
    test.SetCommand(["ls", directory])
    test.SetProperty("WORKING_DIRECTORY", os.getcwd())

    # create a second test
    # previous test is already stored by PyCTest
    test = pyctest.test()
    test.SetName("hostname")
    test.SetCommand(["hostname"])
    test.SetProperty("TIMEOUT", "10")

    # generate the CTestConfig.cmake and CTestCustom.cmake
    pyctest.generate_config(directory)

    # generate the CTestTestfile.cmake file
    pyctest.generate_test_file(directory)

    # run CTest -- e.g. ctest -VV ${PWD}/pycm-test
    pyctest.run(pyctest.ARGUMENTS, directory)
CTest arguments (default): '-V -DSTAGES=Start;Update;Configure;Build;Test;Coverage;MemCheck -S Stages.cmake -j1'
Writing CTest test file: "/Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestTestfile.cmake"...
Generating test "list_directory"...
Generating test "hostname"...
-- STAGES = Start;Update;Configure;Build;Test;Coverage;MemCheck
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Running CTEST_START stage...
Run dashboard with model Continuous
   Source directory: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
   Build directory: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
   Track: Continuous
   Reading ctest configuration file: /Users/jrmadsen/devel/c++/pyctest-master/pycm-test/CTestConfig.cmake
   Site: JRM-macOS-DOE.local
   Build name: [Darwin macOS 10.13.6 x86_64] [Python 3.6.7]
   Use Continuous tag: 20181129-2118
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_UPDATE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_CONFIGURE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_BUILD stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Running CTEST_TEST stage...
Test project /Users/jrmadsen/devel/c++/pyctest-master/pycm-test
    Start 1: list_directory
1/2 Test #1: list_directory ...................   Passed    0.00 sec
    Start 2: hostname
2/2 Test #2: hostname .........................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_COVERAGE stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_MEMCHECK stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Skipping CTEST_SUBMIT stage...
-- [[Darwin macOS 10.13.6 x86_64] [Python 3.6.7]] Finished Continuous Stages (Start;Update;Configure;Build;Test;Coverage;MemCheck)

Command Line Interface

PyCTest args: ['--help']
  CTest args: []
  CMake args: []
usage: pyctest-runner.py [-h] [-F] [-S] [-A] [-j <INT>] [-m <TYPE>] [-b <NAME>] [-i <PATH>] [-o <PATH>] [-M <TYPE>] [-H <SITE>] [-P <EXE>] [-N <NAME>]
                         [-C <PATH> [<PATH> ...]] [-L <LABEL> [<LABEL> ...]] [-T [<TYPE> [<TYPE> ...]]] [-fc <EXE>] [-cc <EXE>] [-cxx <EXE>]
                         [--pyctest-token <TOKEN>] [--pyctest-token-file <FILE>] [--pyctest-vcs-type <TYPE>] [--pyctest-build-type <TYPE>]
                         [--pyctest-trigger <TYPE>] [--pyctest-use-launchers <BOOL>] [--pyctest-ctest-args [<ARG> [<ARG> ...]]] [--pyctest-cdash-version <STR>]
                         [--pyctest-submit-retry-count <INT>] [--pyctest-submit-retry-delay <INT>] [--pyctest-curl-options <LIST>]
                         [--pyctest-drop-location <STR>] [--pyctest-drop-method <STR>] [--pyctest-drop-site <STR>] [--pyctest-drop-site-password <STR>]
                         [--pyctest-drop-site-user <STR>] [--pyctest-nightly-start-time <STR>] [--pyctest-update-version-only]

PyCTest argparse. Arguments after first '--' are passed directly to CTest, arguments after second '--' are passed directly to CMake

optional arguments:
  -h, --help                                                                   show this help message and exit
  -F, --pyctest-clean-first                                                    Remove standard PyCTest files and binary directory (if not source directory)
  -S, --pyctest-submit                                                         Enable submission to dashboard
  -A, --pyctest-append                                                         Append to last invocation of CTest
  -j <INT>, --pyctest-jobs <INT>                                               number of concurrent jobs
  -m <TYPE>, --pyctest-mode <TYPE>                                             Run specific stage. Choices:
                                                                               [Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit,Stages]
  -b <NAME>, --pyctest-build-name <NAME>                                       Build name for identification
  -i <PATH>, --pyctest-source-dir <PATH>                                       Source directory
  -o <PATH>, --pyctest-binary-dir <PATH>                                       Binary/build/working directory
  -M <TYPE>, --pyctest-model <TYPE>                                            CTest submission model (track). Choices: [Nightly,Continuous,Experimental]
  -H <SITE>, --pyctest-site <SITE>                                             CTest submission site
  -P <EXE>, --pyctest-python-exe <EXE>                                         Python executable to use. This can be absolue, relative, or CMake path
  -N <NAME>, --pyctest-project-name <NAME>                                     Name of project using PyCTest
  -C <PATH> [<PATH> ...], --pyctest-cleanup <PATH> [<PATH> ...]                Remove standard PyCTest files and binary directory (if not source directory) and
                                                                               exit
  -L <LABEL> [<LABEL> ...], --pyctest-subproject-labels <LABEL> [<LABEL> ...]  Add labels for subproject
  -T [<TYPE> [<TYPE> ...]], --pyctest-stages [<TYPE> [<TYPE> ...]]             Run multiple stages. Choices:
                                                                               [Start,Update,Configure,Build,Test,Coverage,MemCheck,Submit]
  -fc <EXE>, --pyctest-fortran-compiler <EXE>                                  Specify Fortan compiler (if needed)
  -cc <EXE>, --pyctest-c-compiler <EXE>                                        Specify the C compiler (if needed)
  -cxx <EXE>, --pyctest-cxx-compiler <EXE>                                     Specify C++ compiler (if needed)
  --pyctest-token <TOKEN>                                                      CTest site token for submission
  --pyctest-token-file <FILE>                                                  Path to file for CTest site token for submission
  --pyctest-vcs-type <TYPE>                                                    Set to enable revision ID discovery during the Update stage. Choices:
                                                                               [bzr,cvs,git,hg,p4,svn]
  --pyctest-build-type <TYPE>                                                  Specify CMAKE_BUILD_TYPE (if using CMake). Choices:
                                                                               [Release,RelWithDebInfo,Debug,MinSizeRel]
  --pyctest-trigger <TYPE>                                                     DEPRECATED
  --pyctest-use-launchers <BOOL>                                               Enable launchers
  --pyctest-ctest-args [<ARG> [<ARG> ...]]                                     CTest arguments
  --pyctest-cdash-version <STR>                                                Set CTest variable: 'CTEST_CDASH_VERSION'
  --pyctest-submit-retry-count <INT>                                           Set CTest variable: 'CTEST_SUBMIT_RETRY_COUNT'
  --pyctest-submit-retry-delay <INT>                                           Set CTest variable: 'CTEST_SUBMIT_RETRY_DELAY'
  --pyctest-curl-options <LIST>                                                Set CTest variable: 'CTEST_CURL_OPTIONS'
  --pyctest-drop-location <STR>                                                Set CTest variable: 'CTEST_DROP_LOCATION'
  --pyctest-drop-method <STR>                                                  Set CTest variable: 'CTEST_DROP_METHOD'
  --pyctest-drop-site <STR>                                                    Set CTest variable: 'CTEST_DROP_SITE'
  --pyctest-drop-site-password <STR>                                           Set CTest variable: 'CTEST_DROP_SITE_PASSWORD'
  --pyctest-drop-site-user <STR>                                               Set CTest variable: 'CTEST_DROP_SITE_USER'
  --pyctest-nightly-start-time <STR>                                           Set CTest variable: 'CTEST_NIGHTLY_START_TIME'
  --pyctest-update-version-only                                                Specify that you want the version control update command to only discover the
                                                                               current version that is checked out, and not to update to a different version

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pyctest, version 0.0.12
Filename, size File type Python version Upload date Hashes
Filename, size pyctest-0.0.12.tar.gz (9.4 MB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page