Change analyzer

• Change analyzer
  • About
    • Main Data features
    • Data Collection
    • Data Reconstruction
    • Data Analysis
    • Data Validation
  • Setup and usage in Windows environment
    • 1. Download required 3rd party softwares:
    • 2. Enable developer mode in Windows
    • 3. Create virtual environment
    • 4. Create configuration file
    • 5. Run the software
    • 6. Compare results
  • Available commands
    • 1. ca-run
    • 2. ca-compare
  • Configuration files
  • Acknowledgement

About

Change-Analyzer (CA in short) is a framework built utilizing ML techniques, leveraging OpenAI Gym library and LudwigAI framework. CA allows product teams to get feedback regarding their software product, aka SUT (System Under Test).

Main Data features

Essentially, is built around the following main Data features:

• Data Collection, done through Automated Exploratory Testing
• Data Reconstruction, done through Automated Regression Testing
• Data Analysis, done through Change Detection
• Data Validation, done through Verdict Generation

Data Collection

After installation and configuration (see sections below) Data Collection can be started. An agent is exploring the SUT without prior knowledge of it, recording testing sequence and state of the application. Once the execution is completed, the Data is saved in a csv file, available in a dedicated folder, from recordings. Currently, the Data consists of the following information regarding the steps of the executed sequence:

• Timestamp: an Integer with the execution time in epoch
• SequenceId: a String with a unique ID to identify the sequence
• PageSourceBefore: a String with the page source (before step execution)
• PageSourceAfter: a String with the page source (after step execution)
• ActionToPerform: a String with the action that will be performed next
• ActionImageBefore: a numpy.ndarray with the image of SUT (before step execution)
• ActionImageAfter: a numpy.ndarray with the image of SUT (after step execution)

Data Reconstruction

Once several sequences exist, they can be used for Data Reconstruction. An agent is rerunning the specified existing sequence, recording a new state of the application. Data collected has the same structure as described in the above section.

Data Analysis

In this phase, two sequences are analyzed. The sequences are the same, from steps perspective, but are executed against different SUT versions (deltas).
A report is created, to showcase the differences between the two sequences, if any. The goal is to provide a visual tool for change detection.
The report will contain the following info:

• The info regarding which is the expected sequence and which is the actual sequence
• A visual indicator for each step if it was considered passed or failed
• A description of the executed steps step
• A short info to support the step status
• Expected image of the SUT, from expected sequence
• Actual image of the SUT, from actual sequence
• Actual and expected images may contain highlighted elements, if they were detected to be part of a change

Data Validation

Using ML techniques, in this phase, the tool should be able to say if the detected changes are expected behavior of faults in the SUT. Data Validation will be cover in more details later, because currently is in planning phase.

Setup and usage in Windows environment

1. Download required 3rd party softwares:

• Download Selenium standalone server.
  • Version 3.141.59 confirmed as working.
  • Put .jar file to place where you can easily find it.
• FOR DESKTOP APP TESTING: Download Windows Application Driver.
  • Version 1.2.1 confirmed as working.
  • Install downloaded .msi file.
• FOR WEBSITE TESTING: Download Chrome driver.
  • Version 94.0.4606.61 confirmed as working.
  • Put binary file to place where you can easily find it.

2. Enable developer mode in Windows

• Windows logo -> developer settings -> Enable
  • Official documentation if needed.

3. Create virtual environment

3.1. Virtualenv option

Requires Python 3.7 or newer installed and added to PATH variable.

• Run install.bat located in source code root directory.
  • This script will install necessary dependencies defined in setup.py.

3.2. Conda option

If preferred, it is also possible to use a Conda environment.

• Open an Anaconda terminal and go to project's main folder
• Run conda env create -f requirements.yml
• The above command will create the conda environment change_analyzer including all the needed dependencies
• In order to use the Conda environment, you can either attach it to a project within your favorite IDE, or activate it on command line with conda activate change_analyzer

4. Create configuration file

It's mandatory to tell change-analyzer how to execute testing. For this reason .ini configuration is needed. Create configure now before running the software.

• More details can be found in later part of this README.

5. Run the software

Requires Java 8 installed and added to PATH variable.

• Start Selenium standalone server:
  • Open CMD and change directory to Selenium directory (downloaded in step 1).
  • java -jar selenium-server-standalone-[version].jar
    • Version needs to be change accordingly.
  • Minimize CMD and let it run on background.
• FOR DESKTOP APP TESTING: Start Windows Application Driver
  • Run WinAppDriver.exe
    • Default location C:\Program Files (x86)\Windows Application Driver
  • Minimize CMD and let it run on background.
• FOR WEBSITE TESTING:
  • Run chromedriver.exe
  • Minimize CMD and let it run on background.
• Run the software:
  • Open CMD and change directory to source code root.
  • Activate virtual environment: .venv\Scripts\activate
  • Generate new report: ca-run --config [configuration filename].ini
• Results appear in recordings directory.

6. Compare results

Requires two generated test results.

• Open CMD and change directory to source code root.
• Activate virtual environment: .venv\Scripts\activate
• Compare results: ca-compare --sequence1_folder [Test folder 1] --sequence2_folder [Test folder 2]
  • Replace [Test folder 1] with real results folder name (example 2021_10_05-14_15_00).
  • Test results are generated to recordings directory.
  • Argument paths starts from recordings directory.

Available commands

1. ca-run

It is used to explore a given application and generate new test results.

Parameters

• --config (mandatory) - Defines which .ini file to use. Currently, Windows and Web platforms are supported.
• --csv_folder (optional) - it triggers the Replay mode. Replay will run the sequence found within the csv_folder.
• --steps(optional) - it represents the number of total steps desired for the sequence to have (default is 10)
• --strategy(optional) - it defines the agent strategy, either random or rl (reinforcement learning)
• --model_dir (optional) - it specifies the name of the pretrained model to use from pretrained_models folder

Results

Results can be found within recordings folder in a dedicated sub-folder.

Command examples

1a. ca-run --config win.ini --steps 5 --strategy rl --model_dir model1

CA will run using the config file win.ini and will aim to have a sequence of 5 steps. It will also use reinforcement learning as strategy, with the pretrained model model1.

1b. ca-run --config web.ini --steps 15

CA will run using the config file web.ini and will aim to have a sequence of 15 steps. In this case, the strategy to get the next action is random.

2. ca-compare

It is used to compare two test results of the same test sequence, applied to two versions of the same SUT.

Parameters

• --sequence1_folder (optional) - recordings folder with the expected test results
• --sequence2_folder (optional) - recordings folder with the actual test results

Note: If both sequences are missing, CA will attempt to find the last two test results

Results

Results can be found within sequence1_folder folder in a dedicated sub-folder.

Command examples

2a. ca-compare --sequence1_folder 2022_08_26-12_14_34 --sequence2_folder 2022_08_26-12_24_45

CA will compare the test results from defined sequence1_folder (expected) and sequence2_folder (actual).

Configuration files

This project uses .ini files as configure files to change software's behavior. User must define own configuration files. Ini file must define following values:

  [driver]
  command_executor=
  platform=
  app=

Example .ini file using WinAppDriver and Windows platform:

  [driver]
  command_executor=http://127.0.0.1:4723
  platform=win
  app=/path/to/test/application.exe

Example .ini file using SeleniumDriver and Web platform:

  [driver]
  command_executor=http://127.0.0.1:4444/wd/hub
  platform=web
  url=https://learn.ivves.eu/

Port of command_executor needs to match the driver's (ex. WinAppDriver/chromedriver) port.

Acknowledgement

The package was developed by F-Secure Corporation in scope of IVVES project. This work was labelled by ITEA3 and funded by local authorities under grant agreement 'ITEA-2019-18022-IVVES'.