setprogramoptions 0.5.0a3

Program options configuration file reader using ConfigParserEnhanced.

SetProgramOptions

The SetProgramOptions package extends ConfigParserEnhanced to enable the processing of .ini files that specify command line program options.

As a subclass of ConfigParserEnhanced, SetProgramOptions supports all the operations that ConfigParserEnhanced supports and adds some of its own. The following table notes the new operations that SetProgramOptions adds:

Operation	Format	Defined By
use	use <section>	ConfigParserEnhanced
opt-set	opt-set Param1 [Param2..ParamN] [: <VALUE>]	SetProgramOptions
opt-remove	opt-remove Param [SUBSTR]	SetProgramOptions

Generally speaking, when we parse a section from a .ini file, the following steps are taken:

1. Parse the section, resolving any use operations to fully parse the DAG generated by the section and its dependents.
2. This generates a list of actions that are specified in the configuration file. Any remove operations encountered will execute their removal search during parsing to ensure they only remove matches that are previously defined.
3. Once parsed, a generator can be invoked to process the actions-list and generate the requested back-end format. Currently, SetProgramOptions only generates output for bash scripts but this can easily be extended to support other formats such as Windows batch or Powershell files, or other kinds of shell commands. Subclasses can also be extended to include their own generator types as well.

Supported Operations

use

The use operation is inherited from ConfigParserEnhanced. Please see its documentation on this command and its use.

opt-set

Sets a generic command line style option.

The format of this is opt-set Param1 [Param2] [Param3] ... [ParamN] : [VALUE]

In a bash context, this operation attempts to generate an option for some command that will be executed. SetProgramOptions will concactenate the Params together and then append =VALUE if a VALUE field is present. For example, opt-set Foo Bar : Baz will become FooBar=Baz.

opt-remove

Removes existing entries that have been processed up to the point the opt-remove is encountered that match a pattern.

The format of this is opt-remove Param [SUBSTR]

When a remove is encountered, SetProgramOptions will search through all processed options and will delete any that contain any Param-i that matches Param. By default the parameters much be an exact match of Param, but if the optional SUBSTR parameter is provided then SetProgramOptions will treat Param as a substring and will remove all existing options if any parameter contains Param.

SetProgramOptions Config Files

A .ini file that can be processed by SetProgramOptions can be formatted like this:

[COMMAND_LS]
opt-set ls

This is perhaps the most simple thing we could do. Using gen_option_list('COMMAND_LS', generator="bash") this operation would generate the command ls when processed.

A more complex section which creates a CMake command call might look like this:

[COMMAND_CMAKE]
opt-set cmake
opt-set -G : Ninja
opt-set -D CMAKE_CXX_FLAGS : "-O3"

and this would generate the command cmake -G=Ninja -DCMAKE_CXX_FLAGS="-O3" when processed for bash output.

Variable Expansion within VALUE fields

Variables can be added to the VALUE fields in handled instructions, but they have their own format that must be used:

${VARNAME|VARTYPE}

• VARNAME is the variable name that you might expect for a bash style environment variable that might be defined like this: export VARNAME=VALUE.
• VARTYPE is the type of the variable that is being declared. For SetProgramOptions the only recognized type is ENV which defines environment variables. Subclasses such as SetProgramOptionsCMake define their own types.

We do not provide a default type for this because we wish it to be explicit that this is a pseudo-type and do not want it to be confused with some specific variable type since that meaning can change depending on the kind of generator being used. For example, ${VARNAME} is an environment variable within a bash context but in a CMake fragment file it would be an internal CMake variable and $ENV{VARNAME} would be an environment variable. By not providing a default we force type consideration to be made explicitly during the creation of the .ini file.

SetProgramOptions Examples

Example 1

example-01.ini

[BASH_VERSION]
opt-set bash
opt-set --version

[LS_COMMAND]
opt-set ls

[LS_LIST_TIME_REVERSED]
opt-set "-l -t -r"

[LS_CUSTOM_TIME_STYLE]
opt-set --time-style : "+%Y-%m-%d %H:%M:%S"

[MY_LS_COMMAND]
use LS_COMMAND
use LS_LIST_TIME_REVERSED
use LS_CUSTOM_TIME_STYLE

example-01.py

#!/usr/bin/env python3
import setprogramoptions

filename = "example-01.ini"
section  = "MY_LS_COMMAND"

# Create SetProgramOptions instance
popts = setprogramoptions.SetProgramOptions(filename)

# Parse section
popts.parse_section(section)

# Generate the list of bash options for the command
bash_options = popts.gen_option_list(section, generator="bash")

# Print out the commands
print(" ".join(bash_options))

generates the output:

ls -l -t -r --time-style="+%Y-%m-%d %H:%M:%S"

Example 2

We can utilize the use operation to create a more complex configuration file that provides some sort of common sections and then point-of-use sections that would generate customized configurations for a particular use:

[CMAKE_COMMAND]
opt-set cmake
opt-set -G : Ninja

[CMAKE_OPTIONS_COMMON]
opt-set -D CMAKE_CXX_FLAGS : "-fopenmp"

[CMAKE_OPTIONS_APPLICATION]
opt-set -D MYAPP_FLAG1 : "foo"
opt-set -D MYAPP_FLAG2 : "bar"

[APPLICATION_PATH_TO_SOURCE]
opt-set /path/to/source/.

[APPLICATION_CMAKE_PROFILE_01]
use CMAKE_COMMAND
use CMAKE_OPTIONS_COMMON
use CMAKE_OPTIONS_APPLICATION
use APPLICATION_PATH_TO_SOURCE

[APPLICATION_CMAKE_PROFILE_02]
use APPLICATION_PROFILE_01
opt-remove MYAPP_FLAG2

This example follows a pattern that larger projects might wish to use when there are many configurations that may be getting tested. Here, we set up some common option groups and then create aggregation sections that will include the other sections to compose a full command line.

Using this .ini file, if we generate bash output for section APPLICATION_CMAKE_PROFILE_01 the resulting command generated would be: cmake -G=Ninja -DCMAKE_CXX_FLAGS="-fopenmp" -DMYAPP_FLAG1="foo" -DMYAPP_FLAG2="bar" /path/to/source/.

Alternatively, we can generate bash output for section APPLICATION_CMAKE_PROFILE_02 which first clones APPLICATION_CMAKE_PROFILE_01 and then removes all entries containing the parameter MYAPP_FLAG2 using the opt-remove operation. This will result in a generated comand cmake -G=Ninja -DCMAKE_CXX_FLAGS="-fopenmp" -DMYAPP_FLAG1="foo" /path/to/source/..

This example shows how the opt-remove operation will fully remove occurrences that contain that substring from the list of actions.

This example shows some of the capabilities that SetProgramOptions provides for managing many build configurations within a single .ini file.

SetProgramOptionsCMake

SetProgramOptionsCMake is a subclass of SetProgramOptions that adds additional operations and generators to handle processing CMake options:

• Adds opt-set-cmake-var.
• Adds cmake_fragment generator.
• Adds CMAKE type to variables.

New operations defined in SetProgramOptionsCMake:

Operation	Format	Defined By
opt-set-cmake-var	opt-set-cmake-var VARNAME [TYPE] [FORCE] [PARENT_SCOPE]: VALUE	SetProgramOptionsCMake

Supported Operations

opt-set-cmake-var

This adds a CMake variable program option. These have a special syntax in bash that looks like -DVARNAME:TYPE=VALUE where the :TYPE is an optional parameter. If the type is left out then CMake assumes the value is a STRING.

We may not wish to generate bash only output though. For CMake files, we might wish to generate a cmake fragment file which is basically a snippet of CMake that can be loaded during a CMake call using the -S option: cmake -S cmake_fragment.cmake. The syntax within a CMake fragment file is the same as in a CMake script itself.

If the back-end generator is creating a CMake fragment file, the set command generated will use [CMake set syntax]. This looks something like set(<variable> <value>) but can also contain additional options. These extra options can be provided in the opt-set-cmake-var operation in the .ini file:

• FORCE -
  • By default, a set() operation does not overwrite entries in a CMake file. This can be added to force the value to be saved.
  • This is only applicable to generating cmake fragment files.
• PARENT_SCOPE - If provided, this option instructs CMake to set the variable in the scope that is above the current scope.
  • This is only applicable to generating cmake fragment files.
• TYPE - Specifies the TYPE the variable can be.
  • This is a positional argument and must always come after VARNAME.
  • Valid options for this are STRING (default), BOOL, PATH, INTERNAL, FILEPATH.
  • Adding a TYPE option implies that the CACHE and docstring parameters will be added to a set() command in a CMake fragment file according to the syntax: set(<variable> <value> CACHE <type> <docstring> [FORCE]) as illustrated on the CMake set() documentation.
  • This is applicable to both cmake fragment and bash generation.

SetProgramOptionsCMake Config Files

Here is an example of what a .ini file may look like using the CMake operations provided by this class:

[SECTION_A]
opt-set cmake
opt-set-cmake-var MYVARIABLENAME  : VALUE
opt-set-cmake-var MYVARIABLENAME2 PARENT_SCOPE : VALUE

Handling CMake Variables

A CMake variable in this context would be an internal variable that is known to CMake. Because this is not a variable that would be known outside of the context of .cmake files, this kind of variable is only applicable when generating CMake fragment files.

It is necessary to provide a CMake variant for variable expansions because the CMake syntax for variables is different than that used by Bash, and CMake fragments have a specialized syntax for environment variables as well. In CMake fragment files:

• environment variables are written as $ENV{VARNAME}
• internal CMake variables are written as: ${VARNAME}

We saw variables in SetProgramOptions follow the syntax: ${VARNAME|ENV} where ENV specifies the kind of variable we're declaring. We extend this in SetProgramOptionsCMake by adding a ${VARNAME|CMAKE} variation which indicates that the variable is expected to be an internal cmake variable and is more suited towards being used within a CMake fragment file since it has no meaning at the command line.

You can still use a CMake variable expansion entry when generating bash output but there is a catch. The variable must be resolvable to something that is not a CMake variable through its transitive closure. This is achieved by caching the last known value of a variable as we process a .ini file and provided that the value ultimately resolves to either a string or an environment variable we can still use it. If it cannot be resolved to something that isn't a CMake variable then an exception should be generated.

For example, if we have a .ini file that sets up CMAKE_CXX_FLAGS to include -O0 in a common section like this:

[COMMON]
opt-set-cmake-var CMAKE_CXX_FLAGS STRING FORCE: "-O0"

and then we have a later section that adds an OpenMP flag to it like this:

[ADD_OPENMP]
use COMMON
opt-set-cmake-var CMAKE_CXX_FLAGS STRING FORCE: "${CMAKE_CXX_FLAGS|CMAKE} -fopenmp"

This is valid since ${CMAKE_CXX_FLAGS|CMAKE} will get replaced with -O0 so the resulting CMAKE_CXX_FLAGS variable would be set to -O0 -fopenmp after processing. If we generate bash output for the ADD_OPENMP section we'll get a -D option that looks like -DCMAKE_CXX_FLAGS:STRING="-O0 -fopenmp".

But what if we have a .ini file with a CMake variable that can't be resolved to something that is not a CMake flag, such as:

[COMMON]
opt-set-cmake-var FOO : ${SOME_CMAKE_VAR|CMAKE}

If we tried to process this and write out the resulting script using the bash generator an exception should be raised citing that we don't know what to do with that unresolved CMake variable. This would be the equivalent to a bash option -DFOO=<SOME_CMAKE_VAR> and bash can't handle that because it has no idea what it should put in that cmake var field.

Note: if the same CMake option is provided in multiple lines they will all be included in the generated output. In that case, the behaviour will match what will occur if one called cmake directly with the same option multiple times. In that case, the last one wins since all -D options are treated as though they both have FORCE and CACHE flags set.

SetProgramOptionsCMake Examples

Example

This example shows a configuration file that can be used to generate build files using Ninja or Makefile. In the .ini file we set up some common sections that contain the arguments and then the point-of-use sections ( MYPROJ_CONFIGURATION_NINJA and MYPROJ_CONFIGURATION_MAKEFILES ) can compose their command lines by importing the argument definition sections via use.

example-02.ini

#
# example-02.ini
#
[CMAKE_COMMAND]
opt-set cmake

[CMAKE_GENERATOR_NINJA]
opt-set -G : Ninja

[CMAKE_GENERATOR_MAKEFILES]
opt-set -G : "Unix Makefiles"

[MYPROJ_OPTIONS]
opt-set-cmake-var  MYPROJ_CXX_FLAGS       STRING       : "-O0 -fopenmp"
opt-set-cmake-var  MYPROJ_ENABLE_OPTION_A BOOL   FORCE : ON
opt-set-cmake-var  MYPROJ_ENABLE_OPTION_B BOOL         : ON

[MYPROJ_SOURCE_DIR]
opt-set /path/to/source/dir

[MYPROJ_CONFIGURATION_NINJA]
use CMAKE_COMMAND
use CMAKE_GENERATOR_NINJA
use MYPROJ_OPTIONS
use MYPROJ_SOURCE_DIR

[MYPROJ_CONFIGURATION_MAKEFILES]
use CMAKE_COMMAND
use CMAKE_GENERATOR_MAKEFILES
use MYPROJ_OPTIONS
use MYPROJ_SOURCE_DIR

example-02.py

This python code shows generating a bash script and a CMake fragment of the configuration specified in the .ini file.

#!/usr/bin/env python3
# -*- mode: python; py-indent-offset: 4; py-continuation-offset: 4 -*-
from pathlib import Path
import setprogramoptions

print(80*"-")
print(f"- {Path(__file__).name}")
print(80*"-")

filename = "example-02.ini"
popts = setprogramoptions.SetProgramOptionsCMake(filename)

section = "MYPROJ_CONFIGURATION_NINJA"
popts.parse_section(section)

# Generate BASH output
print("")
print("Bash output")
print("-----------")
bash_options = popts.gen_option_list(section, generator="bash")
print(" \\\n   ".join(bash_options))

# Generate a CMake Fragment
print("")
print("CMake fragment output")
print("---------------------")
cmake_options = popts.gen_option_list(section, generator="cmake_fragment")
print("\n".join(cmake_options))

print("\nDone")

Output

Using the Ninja specialization from the above code, we generate the following output:

$ python3 example-02.py
--------------------------------------------------------------------------------
- example-02.py
--------------------------------------------------------------------------------

**Bash output**
cmake \
   -G=Ninja \
   -DMYPROJ_CXX_FLAGS:STRING="-O0 -fopenmp" \
   -DMYPROJ_ENABLE_OPTION_A:BOOL=ON \
   -DMYPROJ_ENABLE_OPTION_B:BOOL=ON \
   /path/to/source/dir

CMake fragment output
---------------------
set(MYPROJ_CXX_FLAGS "-O0 -fopenmp" CACHE STRING "from .ini configuration")
set(MYPROJ_ENABLE_OPTION_A ON CACHE BOOL "from .ini configuration" FORCE)
set(MYPROJ_ENABLE_OPTION_B ON CACHE BOOL "from .ini configuration")

Done