Package used to run tests using Appsurify
Project description
Appsurify CI Script
Index
Available Arguments
Required Arguments
argument | options |
---|---|
apikey | Apikey from appsurify |
url | Url of the appsurify instance i.e. "https://dummy.appsurify.com" do not end with a / |
project | Name of the project |
testsuite | Name of the testsuite |
Recommended Arguments
argument | options |
---|---|
runtemplate | Options - all tests (defaults to fail on all failures, unless overwritten), no tests (will fail on any failures, unless overwritten), prioritized tests with unassigned (will only fail on new or reopened defects, unless overwritten), prioritized tests without unassigned (will only fail on new or reopened defects, unless overwritten). See below for details |
testtemplate | Options - mvn, cucumber mvn, sahi testrunner, sahi ant, testim, mocha, pytest, rspec, robotframework, cyprus, mstest, vstest, katalon, opentest. See below for usage details |
testtemplatearg1-4 | Additional argumentsfor the specified test template |
Parameter Details - testtemplate
Click to expand for test template parameter details/options
Maven
Maven
Parameter value - "mvn"
The following values are set when this testtemplate is selected
- testseparator=","
- addtestsuitename="true"
- testsuitesnameseparator="%23"
- startrunspecific="mvn -Dtest="
- endrunspecific="test"
- startrunall="mvn test"
- report="./target/surefire-reports/"
- reporttype="directory"
Cucumber Mvn
Maven
Parameter value - "cucumber mvn"
The following values are set when this testtemplate is selected
- testseparator=" "
- startrunspecific="mvn test -Dcucumber.options=""
- endrunspecific="" "
- postfixtest="$'"
- prefixtest="--name '^"
- startrunall="mvn test"
- report="./target/surefire-reports/"
- reporttype="directory"
- deletereports="true"
Testim
Testim
Parameter value - "testim"
The following values are set when this testtemplate is selected
- testseparator=" --name '"
- reporttype="file"
- report="test-results.xml"
- startrunspecific="testim --report-file test-results.xml --name '"
- postfixtest="'"
- startrunall="testim --report-file test-results.xml"
Rspec
Rspec
Parameter value - "rspec"
Required Config
The following must be installedhttps://github.com/sj26/rspec_junit_formatter The following values are set when this testtemplate is selected
- testseparator=" "
- startrunspecific="rspec --format RspecJunitFormatter --out rspec.xml "
- prefixtest = "-e '"
- postfixtest="'"
- startrunall="rspec --format RspecJunitFormatter --out rspec.xml"
- reporttype="file"
- report="rspec.xml"
Sahi Ant
Sahi Ant
Parameter value - "sahi ant"
Required Config
Ensure junit report is set in the ant file - https://sahipro.com/docs/using-sahi/playback-desktop.html#Playback%20via%20ANT
- testtemplatearg1 - The report location set as per above
- testtemplatearg2 - The ant file to run all tests
- testtemplatearg3 - The ant file to the specific set of tests
The following values are set when this testtemplate is selected
- testseparator=","
- addtestsuitename="true"
- testsuitesnameseparator="%23"
- generatefile="sahi"
- startrunall="ant -f "+testtemplatearg2
- startrunspecific="ant -f "testtemplatearg3
- report = testtemplatearg1
Sahi Testrunner
Sahi Testrunner
Parameter value - "sahi testrunner"
Required Config
Set the Sahi runner to create a junit report - https://sahipro.com/docs/using-sahi/sahi-reports.html The following values must be supplied
- testtemplatearg1 - The report location set as per above
- testtemplatearg2 - The command to run all tests
The following values are set when this testtemplate is selected
- testseparator=","
- addtestsuitename="true"
- testsuitesnameseparator="%23"
- generatefile="sahi"
- startrunspecific="testrunner temp.dd.csv"
- startrunall="testrunner " + testtemplatearg2
- report=testtemplatearg1
Robot Framework
Robot Framework
Parameter value - "robotframework"
Required Config
- testtemplatearg1 - The execution command used to start robotframework tests i.e. java -jar robotframework.jar
- testtemplatearg2 - The location of your tests
- testtemplatearg3 - Report location
The following values are set when this testtemplate is selected
- testseparator=" --test '"
- postfixtest="'"
- reporttype="file"
- report=testtemplatearg3
- startrunall=testtemplatearg1+" -x "+testtemplatearg3+" "
- endrunall=testtemplatearg2
- startrunspecific=testtemplatearg1+" -x "+testtemplatearg3+" "
- endrunall=testtemplatearg2
Mocha
Mocha
Parameter value - "mocha"
Required Config
Note the following plugin must be installed to generate the junit report file - https://www.npmjs.com/package/mocha-junit-reporter
The following values are set when this testtemplate is selected
- testseparator="|"
- reporttype="file"
- report="test-results.xml"
- startrunspecific="mocha test --reporter mocha-junit-reporter -g "
- postfixtest="$"
- prefixtest="^"
- startrunall="mocha test --reporter mocha-junit-reporter "
Pytest
Pytest
Parameter value - "pytest"
The following values are set when this testtemplate is selected
- testseparator=" or "
- reporttype="file"
- report="test-results.xml"
- startrunspecific="python -m pytest --junitxml=test-results.xml -k '"
- endrunspecific="'"
- startrunall="python -m pytest --junitxml=test-results.xml"
Cyprus
Cyprus
Parameter value - "cyprus"
Required Config
The following cyprus addon must be installed in order to specify the tests - https://github.com/bahmutov/cypress-select-tests
The following values are set when this testtemplate is selected
- testseparator="|"
- reporttype="file"
- report="results.xml"
- startrunspecific="cypress run --reporter junit --reporter-options mochaFile=result.xml grep="
- postfixtest="'"
- prefixtest="'"
- startrunall="cypress run --reporter junit --reporter-options mochaFile=result.xml"
Mstest
Mstest
Parameter value - "mstest"
The following values are set when this testtemplate is selected
- testseparator=","
- reporttype="file"
- startrunspecific="mstest /resultsfile:'" + testtemplatearg1 + "' /testcontainer:'" + testtemplatearg2 + "'" + "/tests:"
- postfixtest="'"
- prefixtest="'"
- startrunall="mstest /resultsfile:'" + testtemplatearg1 + "' /testcontainer:'" + testtemplatearg2 + "'"
- report=testtemplatearg1
- importtype="trx"
VSTest
VSTest
Parameter value - "vstest"
The following values are set when this testtemplate is selected
- testseparator=","
- reporttype="file"
- startrunspecific="vstest.console.exe /resultsfile:'" + testtemplatearg1 + "' /testcontainer:'" + testtemplatearg2 + "'" + "/tests:"
- postfixtest="'"
- prefixtest="'"
- startrunall="vstest.console.exe /resultsfile:'" + testtemplatearg1 + "' /testcontainer:'" + testtemplatearg2 + "'"
- report=testtemplatearg1
- importtype="trx"
Katalon
Katalon
Parameter value - "katalon"
Required Config
- testtemplatearg1 - Report location
- testtemplatearg2 - Absolute path to project file
- testtemplatearg3 - Relative path from project to a created test suite with all tests added to it
- testtemplatearg4 - API Key
The following values are set when this testtemplate is selected
- testseparator=","
- reporttype="file"
- report = testtemplatearg1
- head_tail = os.path.split(testtemplatearg1)
- report_folder = head_tail[0]
- report_file = head_tail[1]
- head_tail = os.path.split(testtemplatearg3)
- startrunspecific="katalonc -noSplash -runMode=console -projectPath='" + testtemplatearg2 + "' -testSuitePath='" + "'" + os.path.join(head_tail[0], "temp.ts") + "' -apiKey='" + testtemplatearg4 +"' -reportFolder='" + report_folder + " -reportFileName='" + report_file + "'"
- startrunall="katalonc -noSplash -runMode=console -projectPath='" + testtemplatearg2 + "' -testSuitePath='" + "'" + testtemplatearg3 + "' -apiKey='" + testtemplatearg4 +"' -reportFolder='" + report_folder + " -reportFileName='" + report_file + "'"
- generatefile="katalon"
Opentest
Opentest
Parameter value - "opentest"
Required Config
- testtemplatearg1 - Report location
- testtemplatearg2 - Template of template with no tests - this template will be copied as temp.yaml and tests to be executed will be appended to this template
- testtemplatearg3 - Template to run with all tests
The following values are set when this testtemplate is selected
- testseparator=",,"
- reporttype="file"
- report = testtemplatearg1
- full_path = os.path.realpath(source)
- destination = os.path.join(os.path.dirname(full_path),"temp.yaml")
- startrunspecific="opentest session create --out '"+testtemplatearg1+ "' --template '" + destination + "' "
- startrunall="opentest session create --out '"+testtemplatearg1+ "' --template '" + testtemplatearg3 + "' "
- generatefile="opentest"
Click to expand for additional test template configuration options
Test Template Customization
In order to customize the way tests are run you can use the following parameters
argument | options |
---|---|
startrunpostfix | appended to the startrun command |
endrunprefix | prepended to the endrun command |
endrunpostfix | appended to the endrun command |
Editing these will change the way tests are executed to the following: startrun + startrunpostfix + testlist + endrunprefix + endrun + endrunpostfix
Parameter Details - teststorun
Option | Details |
---|---|
all tests | Will run all tests. Will fail on any failure. Recommended for nightly or weekly test executions |
no tests | Will run no tests. Will fail on any failure. Recommended when starting with Appsurify to just upload data |
prioritized tests with unassigned | Will run just high, medium and unassigned tests. Will only fail the build on new/reopened defects. Recommended for your most frequent run type i.e. per commit, nightly |
prioritized tests without unassigned | Will run just high and medium tests. Will only fail the build on new/reopened defects. Recommended for long term use |
Click to expand for further details
We recommend using the runtemplate parameter instead of using teststorun. But for additional control on which tests to run you may change the teststorun field. When doing so we recommend initially running all your tests and just pushing the results to Appsurify. To do this choose either "all" and use this script to run the tests, or choose none and use this script to just upload the results.
Once you have either selected the coverage of the tests or uploaded a number of test reults for Appsurify to learn from select Appsurify recommended and choose from the options where - high = most likely to fail, covers the exact change, medium = less likely to fail, covers the functional area or dependencies, low = very unlikely to fail, does not cover the change, unassigned = tests where Appsurify is unsure whether the coverage of these tests, ready defects = tests which have a defect associated with them which is ready for testing, open defects = tests which have a defect associated with them which is open.
Example Usage
python RunTestsWithAppsurify.py --url "https://dummy.appsurify.com" --apikey "apikeyvalue" --project "Test" --testsuite "Test" --runtemplate "prioritized tests with unassigned" --testtemplate "mvn"
Additional Arguments - For Customization
Click to expand for details on additional arguments, these will be necessary if you want to use Appsurify in a way not available in the templates above
Test Selection Arguments
argument | default | options/details |
---|---|---|
teststorun | "all" | Defaults to "all" if runtemplate is not set. Options include - high, medium, low, unassigned, ready, open, none. Details below |
maxtests | 10000000 | the maximum number of tests to run in each test execution set |
runfrequency | "multiple" | #options 'single' or 'multiple' determines if the prioritized tests should be for a single commit or mulltiple i.e. since the last test run |
Build Failure Arguments
argument | default | options/details |
---|---|---|
fail | newdefects, reopeneddefects | options newdefects, reopeneddefects, flakybrokentests, newflaky, reopenedflaky, failedtests, brokentests |
rerun | "false" | whether failed tests should be rerun |
maxrerun | 3 | the number of times failed tests should be rerun |
failfast | "false" | whether after each set of test runs to determine if the build has failed (tests will still be rerun if this has been selected) |
runtemplate | None | Options - all tests (defaults to fail on all failures, unless overwritten), no tests (will fail on any failures, unless overwritten), prioritized tests with unassigned (will only fail on new or reopened defects, unless overwritten), prioritized tests without unassigned (will only fail on new or reopened defects, unless overwritten) |
Test Command Arguments
argument | default | options/details |
---|---|---|
testtemplate | None | Options - mvn, sahi testrunner, sahi ant, testim, mocha, pytest, rspec, robotframework. See below for usage details |
testtemplatearg1 | None | Additional argument for the specified test template |
testtemplatearg2 | None | Additional argument for the specified test template |
testtemplatearg3 | None | Additional argument for the specified test template |
startrunall | None | Required if testtemplate is not set and runtemplate is all tests or teststorun is set to all. Command to start the test run to execute all tests. Details below |
endrunall | None | Command to end the test run to execute all tests. Details below |
startrunspecific | None | Required if testtemplate is not set and teststorun is not all or none or testtemplate is not all tests or no tests. Command to start the test run to execute specific tests. Details below |
endrunspecific | None | Command to end the test run to execute specific tests |
Test Format Arguments
argument | default | options/details |
---|---|---|
testseparator | "" | #string or character used to separate tests when creating the command to run a specific set of tests |
postfixtest | "" | #string to postpend to each test when creating the command to run a specific set of tests |
prefixtest | "" | string to prepend to each test when creating the command to run a specific set of tests |
addclassname | "false" | whether to include the classname when creating the command to run a specific set of tests |
classnameseparator | " " | string or character to separate classname and testname when creating the command to run a specific set of tests |
addtestsuitename | "false" | whether to include the testsuitename when creating the command to run a specific set of tests |
testsuitenameseparator | " " | string or character to separate testsuitename and classname/testname when creating the command to run a specific set of tests |
Report Arguments
argument | default | options/details |
---|---|---|
report | Required if testtemplate is not passed through as an argument. Location of the reports generated by the test run. | |
deletereports | "false" | options true or false, BE CAREFUL THIS WILL DELETE THE SPECIFIC FILE OR ALL XML FILES IN THE DIRECTORY |
importtype | "junit" | #options 'junit', 'nunit' or 'trx' |
reporttype | "directory" | default directory must end with a /, will look for all importtype files in that directory |
Commit Arguments
argument | default | options/details |
---|---|---|
branch | 'git branch | grep \* | cut -d ' ' -f2' | The branch from which you would like to select the commits to get priorized tests for. Will default to the value of 'git branch | grep \* | cut -d ' ' -f2' if not specified |
commit | git log -1 --pretty="%H | commitId that this test run is for, if not supplied the script will use git log -1 --pretty="%H |
Test Execution Details
Click to expand for test execution details
Execution Command
Note the execution arguments will automatically be set when using the testtemplate.
When executing all tests the following command is created: startrunall+startrunpostfix+endrunprefix+endrunall+endrunpostfix. To execute all tests via maven the command 'mvn test' needs to be run. To do this startrunall could be set to 'mvn test' or startrunall could be set to 'mvn ' and endrunall set to 'test'. Please note spaces are required otherwise the command will not be properly formatted.
When executing specific tests the following command is created startrunspecific+startrunpostfix+specifictestlist+endrunprefix+endrunspecific+endrunpostfix To execute specific tests via maven the command 'mvn -Dtests= test'. To do this startrunspecific needs to be set to 'mvn -Dtests=' and endrunspecific set to ' test'. For details on how specifictestlist is created see below.
Specific Tests
To create the list of specific tests the ci script queries Appsurify for a list of tests. For each test they may have an associated testsuite name and classname as often found in the xml junit report. See below for an example. The list of tests returned by the api may include the testsuitename and/or classname dependant on whether addtestsuitename, addclassname, testssuitenameseparator and classnameseparator are set. If for example addtestsuitename="true", testsuitenameseparator="%23, addclassname="true", classnameseparator=%40 (note values need to be encoded via percent encoding https://en.wikipedia.org/wiki/Percent-encoding) the tests would be formatted as follows: testsuitename#classname@testname.
To create the list the arguments will be conbined in the following way: prefixtest+test+postfixtest If there are multiple tests the arguments would be combined as follows: prefixtest+test+postfixtest+testseparator+prefixtest+test+postfixtest
Combining the above to run a specific set of tests using mvn the following arguments would need to be set: addtestsuitname="true", testsuiteseparator="%23", testseparator=",", startrunspecific='mvn -Dtests=' and endrunspecific =' test' which would cause the following command to be created "mvn -Dtest=testsuite#testname,testsuite#testname test
If however the tests needed to be in the format of 'run tests -test=test1 -test=test2 -test=test3' then the following arguments would be required: startrunspecific="run tests " and prefixtest="-test="
<?xml version="1.0" encoding="UTF-8" ?>
<testsuites>
<testsuite errors="0" failures="1" hostname="DESKTOP-16O0SVF" id="0" name="TestRun" package="junit" skipped="0" tests="1" time="0.16" timestamp="2018-06-07T06:22:50">
<testcase classname="junit.TestRun" name="PassToFail" time="0.008">
<failure message="failreason1" type="junit.framework.AssertionFailedError"> fails
</failure>
</testcase>
</testsuite>
</testsuites>
Examples
Maven
python RunTestsWithAppsurify.py --url "https://dummy.appsurify.com" --apikey "apikeyvalue" --project "Test" --testsuite "Test" --report "report --runtemplate "prioritized tests with unassigned" --testtemplate "mvn"
Deprecated examples
Click to expand for deprecated examples
Java All Tests
RunTestsWithAppsurify.sh --url "https://dummy.appsurify.com" --apikey "apikeyvalue" --project "Test" --testsuite "Test" --report "report" --teststorun "all" --startrun "mvn test"
Java High Tests
RunTestsWithAppsurify.sh --url "https://dummy.appsurify.com" --apikey "apikeyvalue" --project "Test" --testsuite "Test" --report "report" --teststorun "high,medium" --startrun "mvn -Dtest=" --fullname "true" --fullnameseparator "#" --endrun " test" --testseparator ","
Rspec all tests
https://github.com/sj26/rspec_junit_formatter must be installed Startrun="rspec --format RspecJunitFormatter --out rspec.xml"
Rspec specific tests
RunTestsWithAppsurify.sh --url "https://dummy.appsurify.com" --apikey "apikeyvalue" --project "Test" --testsuite "Test" --report "rspec.xml" --teststorun "high,medium" --startrun "rspec --format RspecJunitFormatter --out rspec.xml -e '" --endrun = "'" --testseparator "' -e '" https://relishapp.com/rspec/rspec-core/docs/command-line/example-option
Change Log
0.0.23
- Fix for webdriver io creating invalud Junit
- Fix for large trx files
- Improvement for specflow removing retry
0.0.22
- Include additional filter for Azure
0.0.21
- Change report to work for partial paths in Windows
0.0.19
- Update for Cypress
0.0.18
- Added recursive search for reports
0.0.17
- Add integration with Bitrise using --bitrise flag saves tests to env variable TESTS_TO_RUN
0.0.16
- Update for pipe output
0.0.12
- Update for pipe output
0.0.9
- Update for trainer
0.0.7 (08/02/2021)
- Update for Azure to run tests via saved variable
0.0.5 (08/02/2021)
- Default rerun to false
0.0.4 (08/02/2021)
- Add try catch to environment variable creation
0.0.3 (08/02/2021)
- Third Release
- Fix failfast tests
0.0.2 (07/02/2021)
- Second Release
0.0.1 (07/02/2021)
- First Release
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.