robotframework-dashboard 0.5.0

Output processor and dashboard generator for Robot Framework output files

Robot Framework Dashboard

Table of Contents

• Overview
• Installation
• Command Line Features
• Dashboard HTML Features
• Version Details
• Usage
• Examples
• Custom Database Class
• Contributing
• License

Overview

The Robot Framework Dashboard is a tool for Robot Framework that provides insight of your test results across multiple runs. The tool makes use of the built in Robot Framework Result Visitor API to analyse output.xml files, stores these in a simple sqlite3 database and finally creates a HTML dashboard that makes use of Chart.js and Datatables for the graphs and tables and makes use of Bootstrap for styling.

If you need help, have suggestions or want to discuss anything, feel free to contact through the Slack Channel or through a GitHub issue

Installation

1. Install Robot Framework 6.0 or higher (if not already installed):
2. Install Robot Framework Dashboard

pip install robotframework-dashboard

Command Line Features

When running the tool the following steps will be executed

1. Database preparation creates robot_results.db or specified database if it does not yet exist and connects to it
2. Optionally reads test execution result from 1 or more output XML file(s) and uploads to the database
3. Optionally lists all runs currently available in the database, ordered by the time they were entered
4. Optionally removes run(s) specified by run_date or index of being added
5. Optionally generates a dashboard HTML file that can be used to look at the stored results

Dashboard HTML Features

The Robot Framework Dashboard HTML will provide information across your runs across all levels. Currently available features:

• Global filters: run, run tag, run date
• Runs
  • Statistics: pass, fail, skip
  • Duration: total elapsed time
  • Overview: table overview of runs
• Suites
• • Filter: suite
  • Statistics: pass, fail, skip
  • Duration: total elapsed time
  • Most failed: top 10 failing suites
  • Overview: table overview of suites
• Tests
  • Filter: suite, test
  • Statistics: pass, fail, skip
  • Duration: total elapsed time
  • Most failed: top 10 failing tests
  • Fail messages: top 10 fail messages
  • Overview: table overview of tests
• Keywords
  • Filter: keywords
  • Keyword statistics: pass, fail, skip
  • Keyword times run
  • Keyword total duration
  • Keyword average duration
  • Keyword min duration
  • Keyword max duration
  • Overview: table overview of keywords

Version Details

• Tested robot versions: 6.0, 6.0.1, 6.0.2, 6.1, 6.1.1, 7.0, 7.0.1, 7.1, 7.1.1
• When running robotdashboard with robot 6.x installed, output files that are generated by robot 7.x will have no duration fields when processed, this is caused by changes in the Result Visitor model
• When running robotdashboard with robot 7.x installed every output generated by any version above should be processed correctly

Usage

Run robotdashboard through the command line to initialize your database and generate the report:

robotdashboard -o output.xml

For detailed command line options take a look at:

robotdashboard -h

or

robotdashboard --help

Advanced examples:

• Generate the report without adding new outputs

robotdashboard

• Add tags to your output.xmls

robotdashboard -o output1.xml:tag1 -o reports/output2.xml:tag1:tag2

• Add all *output*.xml in an entire folder tree to the database and provide all these runs with tags

robotdashboard -f ./reports
robotdashboard -f ../../some_folder/sub_folder/logs
robotdashboard -f C:/nightly_runs:tag1:tag2:tag3

• Do not list runs and/or do not create the dashboard HTML

robotdashboard -l false -g false

• Custom path/file name for your dashboard HTML

robotdashboard -n results/result_robot_dashboard.html

• Custom database path/file location

robotdashboard -d result_data/robot_result_database.db

• Remove data from the database, can be based on run number (index) or run start time

robotdashboard -r 0
robotdashboard -r -1 -r "2024-10-17 15:05:04.563559"

• Add a custom dashboard html title

robotdashboard -t "My Cool Title"

• Show milliseconds in the graphs in the dashboard html (setting excludemilliseconds to False as it defaults to True)

robotdashboard -e False

• Make use of a custom DatabaseProcessor class see also the Custom Database Class section for examples and more details of the requirements.

robotdashboard -c ./path/to/custom_class.py
robotdashboard --databaseclass mysql.py

Examples

Here are some examples of generated files/output:

• See robot_dashboard.html in the Example Folder in GitHub -> Download and open in a browser to use it!
• See robot_results.db in the Example Folder in GitHub -> Download and use any tool to check the tables or use it as a base for using the "robotdashboard" command line interface
• Example Command Line Output (below)

This is an example after running robotdashboard for the first time. 3 outputs are added, stored and processed in the dashboard HTML.

Input:

robotdashboard -o output1.xml -o output2.xml -o output3.xml

Results:

======================================================================================
 ____   ___  ____   ___ _____ ____    _    ____  _   _ ____   ___    _    ____  ____  
|  _ \ / _ \| __ ) / _ |_   _|  _ \  / \  / ___|| | | | __ ) / _ \  / \  |  _ \|  _ \
| |_) | | | |  _ \| | | || | | | | |/ _ \ \___ \| |_| |  _ \| | | |/ _ \ | |_) | | | |
|  _ <| |_| | |_) | |_| || | | |_| / ___ \ ___) |  _  | |_) | |_| / ___ \|  _ <| |_| |
|_| \_\___/|____/ \___/ |_| |____/_/   \_|____/|_| |_|____/ \___/_/   \_|_| \_|____/

======================================================================================
 1. Database preparation
  created database connection: 'robot_results.db'
======================================================================================
 2. Processing output XML(s)
  Processing output XML 'output1.xml'
  Processed output XML 'output1.xml' in 0.04 seconds
  Processing output XML 'output2.xml'
  Processed output XML 'output2.xml' in 0.05 seconds
  Processing output XML 'output3.xml'
  Processed output XML 'output3.xml' in 0.05 seconds
======================================================================================
 3. Listing all available runs in the database
  Run 0   | 2024-10-17 15:05:04.563559 | RobotFramework-Project1
  Run 1   | 2024-10-18 16:43:12.772757 | Robotframework-Project2
  Run 2   | 2024-10-21 10:54:25.663094 | Robotframework-Project1
======================================================================================
 4. Removing runs from the database
  skipping step
======================================================================================
 5. Creating dashboard HTML
  created dashboard 'robot_dashboard_20241021-150726.html' in 0.01 seconds

Custom Database Class

See the python files in the Example Folder in GitHub for some examples of custom database class implementations.

Available examples

Currently available database type examples:

• template.py (completely empty only requirements are filled)
• mysql.py (for a mysql database connection)
• ...

If you have made an implementation that is not yet an example please feel free to add it through a pull request or submit it in an issue. This way you can help other people implement their own solutions!!

Custom Class Requirements

• File: The filename can be anything as long as it is a python file
• Class: The you should name the class DatabaseProcessor
• Methods:
  • __init__(self, database_path: Path):, should handle the connection to the database and possible creation of the tables if necessary
  • close_database(self):, should close the database connection
  • insert_output_data(self, output_data: dict, tags: list):, should be able to handle the output_data dict and the run tags that are provided. Look at the example implementations for the content of output_data and tags.
  • get_data(self):, should retrieve all data (runs/suites/tests/keywords) from all tables in the form of a dictionary containing the 4 data types. In which each data type is a list of dicts with entries. Example: {'runs': [{"run_start": "2024-10-13 22:33:19", "full_name": "Robotframework-Dashboard", "name": "Robotframework-Dashboard", "total": 6, "passed": 4, "failed": 1, "skipped": 1, "elapsed_s": "6.313", "start_time": "2024-10-13 22:33:19.673821", "tags": ""}, {"run_start"...}]}
  • list_runs(self):, should print out the runs in the database with useful identiefiers. This function can be empty as long as it exists. It is purely for the command line overview.
  • remove_runs(self, remove_runs):, should be able to handle either indexes to be removed, or run_starts that are provided which can then be used to delete the data in the database.
• Do not use relative imports! This will not work on runtime!

Contributing

Contributions are welcome! If you encounter any issues, have suggestions for improvements, or would like to add new features, feel free to open an issue or submit a pull request.

License

This project is licensed under the MIT License.

Note: This project is not officially affiliated with or endorsed by robotframework.