Tools for serialising test results to SQL database
Project description
TestArchiver
TestArchiver is a tool for archiving your test results to a SQL database.
And Epimetheus is the tool for browsing the results you archived.
Testing framework support
Framework | Status | Fixture test status | Parser option |
---|---|---|---|
Robot Framework | Supported | Done | robot |
Mocha | Supported | Done | mocha-junit |
JUnit | Experimental | Missing | junit |
xUnit | Experimental | Missing | xunit |
MSTest | Experimental | Missing | mstest |
pytest | Supported | Done | pytest-junit |
Experimental status here means that there is a parser that can take in e.g. generic JUnit formatted output but there is no specific test set or any extensive testing or active development for the parser.
Contributions for output parsers or listeners for different testing frameworks are appreciated. Contributing simply a fixture test set (that can be used to generate output files for developing a specific parser) is extremely helpful for any new framework.
Installation
pip3 install testarchiver
Supported databases
SQLite
SQLite default database for the archiver and is mainly useful for testing and demo purposes. Sqlite3 driver is part of the python standard library so there are no additional dependencies for trying out the archiver.
PostgreSQL
PostgreSQL is the currently supported database for real projects. For example Epimetheus service uses a PosrgreSQL database. For accessing PostgreSQL databases the script uses psycopg2 module: pip install psycopg2-binary
(comes with pip install)
Basic usage
The output files from different testing frameworks can be parsed into a database using test_archiver/output_parser.py
script.
testarchiver --database test_archive.db output.xml
Assuming that output.xml
is a output file generated by Robot Framework (the default parser option), this will create a SQLite database file named test_archive.db
that contains the results.
For list of other options: testarchiver --help
positional arguments:
output_files list of test output files to parse in to the test
archive
optional arguments:
-h, --help show this help message and exit
--config CONFIG_FILE path to JSON config file containing database
credentials
--dbengine DB_ENGINE Database engine, postgresql or sqlite (default)
--database DATABASE database name
--host HOST databse host name
--user USER database user
--pw PASSWORD, --password PASSWORD
database password
--port PORT database port (default: 5432)
--dont-require-ssl Disable the default behavior to require ssl from the
target database.
--allow-minor-schema-updates
Allow TestArchiver to perform MINOR (backwards
compatible) schema updates the test archive
--allow-major-schema-updates
Allow TestArchiver to perform MAJOR (backwards
incompatible) schema updates the test archive
--format {robot,robotframework,xunit,junit,mocha-junit,pytest-junit,mstest}
output format (default: robotframework)
--repository REPOSITORY
The repository of the test cases. Used to
differentiate between test with same name in different
projects.
--team TEAM Team name for the test series
--series SERIES Name of the test series (and optionally build number
'SERIES_NAME#BUILD_NUM' or build id
'SERIES_NAME#BUILD_ID')
--metadata NAME:VALUE
Adds given metadata to the testrun. Expected format:
'NAME:VALUE'
--change-engine-url CHANGE_ENGINE_URL
Starts a listener that feeds results to ChangeEngine
Data model
Useful metadata
There are meta data that are useful to add with the results. Some testing frameworks allow adding metadata to your test results and for those frameworks (e.g. Robot Framework) it is recommended to add that metadata already to the tests so the same information is also available in the results. Additional metadata can be added when parsing the results using the --metadata
option. Metadata given during the parsing is linked to the top level test suite.
--metadata NAME:VALUE
Test series and teams
In the data model, each test result file is represented as single test run. These test runs are linked and organized into builds in in different result series. Depending on the situation the series can be e.g. CI build jobs or different branches. By default if no series is specified the results are linked to a default series with autoincrementing build numbers. Different test runs (from different testing frameworks or parallel executions) that belong together can be organized into the same build. Different test series are additionally organized by team. Series name and build number/id are separated by #
.
Some examples using the --series
and --team
options of testarchiver
--series ${JENKINS_JOB_NAME}#${BUILD_NUMBER}
--series "UI tests"#<commit hash>
--series ${CURRENT_BRANCH}#${BUILD_ID} --team Team-A
--series manually_run
Each build will have a build number in the series. If the build number is specified then that number is used. If the build number/id is omitted then the build number will be checked from the previous build in that series and incremented. If the build number/id is not a number it is considered a build identifier string. If that id is new to the series the build number is incremented just as if it no build number was specified. If the same build id is found in the same test series then the results are added under that same previously archived build.
If the tests are executed in a CI environment the build number/id is an excellent way to link the archived results to the actual builds.
The series can also be indicated using metadata. Any metadata with name prefixed with series
are interpreted as series information. This is especially useful when using listeners. For example when using Robot Framework metadata --metadata team:A-Team --metadata series:JENKINS_JOB_NAME#BUILD_NUMBER
Release notes
-
2.0.0 (TBD)
- Distribution as pip package
pip3 install testarchiver
and it installs:testarchiver
script (aka: output_parser.py)testarchive_schematool
script (aka: database.py)diff2change_context_list.py
scripttest_archiver
module
- Functionality for managing schema updates
- TestArchiver version has to match with the schema version
- TestArchiver can apply schema updates when explicitly allowed
- Updates are divided to major and minor updates
- Minor updates are backwards compatible for applications reading the database
- Major updates are backwards incompatible for applications reading the database
- Major schema update #1:
- Adds
schema_updates
table for recording schema updates - Adds
schema_version
column totest_run
table to make schema incompatible with old versions TestArchiver - Adds index for log messages for query performance
- Adds
- Renamed
output_parser.py
cli option--change_engine_url
to--change-engine-url
- Record an execution path for test cases, suites and log messages.
- The path explains the position of the item in its test run.
- E.g.
s1-s2-t3
means the third test in the second subsuite of the top suite.
- Minor schema update #2:
- Adds
execution_path
column totest_result
,suite_result
andlog_message
tables
- Adds
- Renamed Robot Framework listener
ArchiverListener.py
asArchiverRobotListener.py
for clarity
- Distribution as pip package
-
1.2.0 (2020-08-18)
- Important database integrity fix when using sqlite database
- Record test criticality (Robot Framework specific)
- Redesign of configurations management
- Allows using both config file and command line arguments uniformly
- CLI arguments override options set in config file
-
1.1.3 (2020-06-09)
- Performance fix for the schema existance check
- Improved error messages:
- Error when trying to archive results that have already been archived
- Error when psycopg2 module is not found
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for testarchiver-2.0.0rc237346140.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 12a9bf50b48c3129d3b812ccff6a28050f526ed8cfac59ecdb300b3f4ef91924 |
|
MD5 | 236889db7cfbe59ac098f100baa039dc |
|
BLAKE2b-256 | 543deb89938a21511297ae65c360fcacb529ec155bb7698ad8ce21f6ae8ea3cc |
Hashes for testarchiver-2.0.0rc237346140-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bd63b613dc16222c9df849eaf6bb23a51feda8a95813e11c2e839c70a5ce7bc6 |
|
MD5 | 7ff542cb175b927f856b278718ac8be3 |
|
BLAKE2b-256 | 6e7c1677dd62fbfe730bf52bbb2720f6391ebc3d2ee06bad6567df9395d7d5ab |