pit-rift 0.1.4

Parametrized Integration Testing framework Tool

RIFT

RIFT stands for - paRametrized Integration testing Framework Tool. It's an integration test framework that focuses its attention to the parametrization and standardization of test procedure.

Installation

Using mamba or conda (recommended)

mamba create -n riftenv
mamba activate riftenv
mamba install pit-rift

Using pip

pip install pit-rift

Usage

How to create a test | Documentation

rift <path_to_test_case> [-v VERBOSE_LEVEL] [-w WORK_DIR] [-i] [-h]

Positional Argument

• test_case: Path to the test case (in ini format) to exec.

Optional Arguments

• -v VERBOSE_LEVEL: Optional. Verbose level from 1 (CRITICAL) to 5 (DEBUG). Default is 4 (INFO).
• -w WORK_DIR: To change the test working directory. Default is ini_test_case_YYYYMMDDTHHmmss.
• -i: If the test case doesn't exist, create it
• -h, --help: Show this help message and exit.

Usage Requirements

If we indicate with <my_test_case> a test case to execute, the following requirements must be satisfied:

1. The command rift must be executed in a directory which contains tests
2. The test implementation must be located in: tests/<procedure_name>, where <procedure_name> is the value of <my_test_case>[EXEC]test_procedure

It's a good practice to save all test cases in a directory called test_cases

How to create a test

To create a test case my_test_case, run the commands:

rift test_cases/my_test_case.ini  -i

After the execution, your working directory will appear like this:

CWD/
├── my_test_case.ini_20240105T165240
│ ├── log
│ │ ├── my_test_case.ini
│ │ ├── pid
│ │ └── test.log
│ ├── out
│ └── work
├── my_test_case.ini
└── tests
    └── my_test_case
        └── main.sh

Expected output:

(base) [am09320@juno CMCC_local]$ rift test_cases/my_test_case.ini  -i
[2024-01-05T17:08:30Z] - INFO - mock.create_test: Init test case: test_cases/my_test_case.ini
[2024-01-05T17:08:30Z] - INFO - mock.init_test_procedure: Mkdir procedure: /work/opa/am09320/dev/DevOpsEnv/CMCC_local/tests/mock_test
[2024-01-05T17:08:30Z] - INFO - mock.init_test_procedure: Copy /work/opa/am09320/dev/DevOpsEnv/dev_refactoring/conf/main.sh to /work/opa/am09320/dev/DevOpsEnv/CMCC_local/tests/mock_test
[2024-01-05T17:08:30Z] - INFO - mock.init_test_case: Mkdir test_cases
[2024-01-05T17:08:30Z] - INFO - mock.init_test_case: Copy /work/opa/am09320/dev/DevOpsEnv/dev_refactoring/conf/test_case.ini as test_cases/my_test_case.ini
[2024-01-05T17:08:30Z] - INFO - rift.main: Loading test case: test_cases/my_test_case.ini
...
[2024-01-05T17:08:30Z] - INFO - scheduler.exec: EXECUTION START
INFO Loading modules

INFO Starting test case
Hello World!
Variable declared into test_case: This is a test
[2024-01-05T17:08:30Z] - INFO - scheduler.exec: EXECUTION_DONE

Once the test has been created, the parameter -i can be omitted

Long runs

If the running time of a test is too long (due to the running time or the waiting time of a job), it's possible to exec rift in background using nohup:

nohup rift test_cases/my_test_case.ini &> my_test_case.log &

---

Authors

• Antonio mariani - [e-mail]

Contributors

• Massimiliano Drudi - [e-mail]

---

References

• What is DevOps?, Web article, https://opensource.com/resources/devops?src=devops_resource_menu1
• Is there a reproducibility crisis in Science?, Nature Video, 2016, https://www.nature.com/articles/d41586-019-00067-3
• Keyes DE, McInnes LC, Woodward C, et al. Multiphysics simulations: Challenges and opportunities. The International Journal of High Performance Computing Applications. 2013;27(1):4-83. https://doi.org/10.1177/1094342012468181
• Theorists and experimentalists must join forces, Nature Editorial, 2021, https://www.nature.com/articles/s43588-021-00082-3
• But is the code (re)usable?, Nature Editorial, 2021, https://www.nature.com/articles/s43588-021-00109-9
• Zebula Sampedro, Aaron Holt, and Thomas Hauser. 2018. Continuous Integration and Delivery for HPC: Using Singularity and Jenkins. In Proceedings of the Practice and Experience on Advanced Research Computing (PEARC '18). Association for Computing Machinery, New York, NY, USA, Article 6, 1–6. https://doi.org/10.1145/3219104.3219147
• Dong H. Ahn, Allison H. Baker, Michael Bentley, Ian Briggs, Ganesh Gopalakrishnan, Dorit M. Hammerling, Ignacio Laguna, Gregory L. Lee, Daniel J. Milroy, and Mariana Vertenstein. 2021. Keeping science on keel when software moves. Commun. ACM 64, 2 (February 2021), 66–74. https://doi.org/10.1145/3382037
• National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.
• Geyer, B., Ludwig, T. & von Storch, H. Limits of reproducibility and hydrodynamic noise in atmospheric regional modelling. Commun Earth Environ 2, 17 (2021). https://doi.org/10.1038/s43247-020-00085-4

RIFT Framework Documentation

Introduction

The RIFT framework, puts its focus in the procedure re-usability, providing the necessary execution information via parameters.

It's composed by:

• Test case: It's a configuration file in .ini format which contains the necessary information for executing a test. Each test case, represents a well specific scenario that must be tested using a general procedure.
• Test procedure: Parameterized code that runs the test with the parameters and environment provided by a test case

Test case

Template

Here is a template that can be used as base to build your own test case:

[Environment]

MY_VAR = value1
...
MY_VARX = Another value

[Exec]
test_procedure = mock_test_procedure
;write as module1 module2 ... moduleX -> automatic loaded before to call the test procedure
modules =
;if defined, the conda environment is automatic loaded before to call the test procedure
conda_env =

;the git sections are optionals - it is possible to declare as much section as you want
[GIT_REPO1]
name = REPO1
url = SSH_GIT_URL
; branch or tag to download - mandatory
branch = GIT_BRANCH

[GIT_REPOX]
;cloned as CMCC_local/<name>
name = REPOX
url = SSH_GIT_URL
; branch or tag to download - mandatory
branch = GIT_BRANCH

Example

Here is a real example of a working test case (from quick-start example):

[Environment]

MY_VAR = This is a test
MY_VAR2 = Another custom variable


[Exec]

test_procedure = mock_test_procedure
# no modules to load
modules =
# no conda env to load
conda_env =

Section Environment

In the environment section:

• declaring all variables needed to exec the test
• the variable name is case-sensitive: VAR != var
• all variables will be automatically exported by the test framework before to run the test procedure
• no limit in the number of variables that ca be declared
• a variable can refer to another variable declared previously into the same file using the bash syntax:

var1 = MY_VALUE
;set var2 as MY_VALUE_extend
var2 = ${var1}_extended

Section Exec

In this section, is reported some information needed for the test execution:

• test_procedure: name of the directory in CMCC_local/tests which contains the main.sh to start the test
• modules: list of module name that will be loaded using te command module load before to run the test procedure (ignored if empty)
• conda_env: name of the conda env to load before to run the test procedure (ignored if empty)

Test Procedure

Definition

A test procedure is a collection of parametrized procedures which are in charge to exec the test. It's composed by:

• main.sh: it's the starting point called by the test framework
• any other executable script necessary for the test execution: the test procedure path will be added to the PATH, this means that it is possible to use into the test case any script that the test needs

Global Environment

Here is a list of global variables exported by the test framework that can be used in the test implementation:

• SOURCE_DIR: The parent directory of CMCC_local
• TEST_DIR: The test procedure dir: CMCC_local/tests/<test_case>
• LOG_DIR: <test_case>_YYYYMMDDTHHmmss/log
• WORK_DIR: <test_case>_YYYYMMDDTHHmmss/work
• DATA_DIR: <test_case>_YYYYMMDDTHHmmss/data
• OUT_DIR: <test_case>_YYYYMMDDTHHmmss/out
• STATUS_DIR: <test_case>_YYYYMMDDTHHmmss/status

Global Function

The bash libraries introduced with old versions of rift, are still available, but they will be removed in the next releases. The libraries will print a warning message to alert the users.

Currently, the test framework, provides two type of libraries:

• Logging
• Submission

Logging

It is a bash implementation of the python logging library [1]:

• logger::debug
• logger::info
• logger::warning
• logger::error
• logger::critical

For each function, the printed format is:

[YYYY-MM-DDTHH:mm:ssZ] - <LEVEL> - <FUNC_NAME>: <MSG>

where LEVEL is: DEBUG, INFO, WARNING, ERROR, CRITICAL

Submission

A function is available to submit a job to LSF or SLURM with a general interface.

Usage: exec_task <cmd> [OPTIONS]

OPTIONS
    -m, --mem <memory>          Sets a memory limit for all the processes that belong to the job.
    -r, --runtime <runtime>     Sets the runtime limit of the job.
    -q, --queue                 Submits the job to specified queue.
    -j, --jobname <jobname>     Assigns the specified name to the job
    -w, --wait                  Submits a job and waits for the job to complete. Sends job status messages to the terminal
    -l, --log <log>             Path of the directory where to write LSF/SLURM log files
    -h, --help                  Show this help message and exit

As an alternative, it is possible to pass some parameter directly from variables declared into the test case:

• SUBMIT_MEM_LIMIT: Sets a memory limit for all the processes that belong to the job.
• SUBMIT_RUN_TIME: Sets the runtime limit of the job.
• SUBMIT_QUEUE: Submit the job to specified queue.
• SUBMIT_LOG_DIR: Path of the directory where to write LSF/SLURM log files

Test Execution

If we indicate with <cmcc_local> the path of CMCC_local which contains the test, to exec a test <my_test_case>exec the commands:

cd <cmcc_local>
rift test_cases/<my_test_case>.ini

Reference

R1.

Logging facility for Python, direct link