robotframework-stopwatch
Robot Framework listener extension for agile perfomance testing.
Evaluate test case execution against its statistical average run time & user-configured timedelta in.
Why?
While Robot Framework does provide an ability to setup time limits (e.g. timeouts) as a means of failing tests/suites depending on their time of execution, such approach is somewhat set in stone. Meaning that you have to manually inspect runtimes of your tests and decide upon much needed time limits. When you have single master suite, or your testing production is not quite large - this works alright. But when your number of tests is 300+, managing each timeout individually is rather straining. Add to that multiple layers of configurations such as different environments, testing tools, or downright infrastructure inconsistencies that do occur and you end up quite strained with the idea of reliable perfomance testing. This extension is an attempt to provide a reliable tool for flexible, integrated perfomance testing.
Overview
As mentioned above, robotframework-stopwatch is not quite a library, but extension of built-in Robot Framework Listener.
When first integrated, the StopwatchListener will initialise an empty logfile (by default: ./stopwatch_log.json) to store required run data for every individual test. After first run, taking into consideration the test succeeded, average runtime for this test will be set to this run time of execution. Afterwards, each succesful test runtime will be used to update the last average runtime value for the test. The user-preconfigured timedelta is then used to evaluate all future test runs status. Failed tests, whether due to timedelta exceeded or natural test fails, have no effect upon test average runtime. Extension also provides a log data divider environment to catalog same tests based on different setups.
When test with established average runtime fails due to exceeded time limit, test is marked as failed with a following message output:
Stopwatch! Test runtime exceeded the acceptable time limit.
Installation
pip install robotframework-stopwatch
Usage
robotframework-stopwatch relies on a single base class StopwatchListener, that comes with following attributes:
-
logfile (str): The path to the log file. By default:./stopwatch_log.json -
environment (str, optional)The marker/divider used to separate test run data across different configurations. -
timedelta (str, optional): The time delta value used for evaluating test runtimes. Defaults to"10s". Time formats include:"5s","2 min","1 hour".
There are two ways to implement robotframework-stopwatch.
Listener approach
When custom listener is already in-use, or robotframework-stopwatch is needed on a global level (every suite, every test), create or modify the listener:
from robot.stopwatch import StopwatchListener
class CustomListener(StopwatchListener):
def __init__(self, logfile, environment, timedelta):
This won't interfere with your custon listener functionality.
Command-line approach
When there is a need to set separate timedelta for different suites/tests, or setting up a listener is not required, it is possible to pass StopwatchListener and its command line arguments directly from terminal:
robot --listener robot.stopwatch.StopwatchListener:"logfile":"environment":"timedelta" some_test.robot
Concurecy
As of now, robotframework-stopwatch supports concurent read/write, making it possible to share a logfile data among multiple agents, instances, test runners. To incorporate concurent logging do rely on the most suitable tactics of shared file storage for your organisation (e.g. Azure Shared Storage).
Plans
- Interface for setting up test run data log in a remote database.
- Robust
timedeltatime formats. - Robust
logfilehandling.
Metadata
Release files for robotframework-stopwatch 0.2.10
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| robotframework_stopwatch-0.2.10.tar.gz | 17.3 kB | Without provenance | Details for robotframework_stopwatch-0.2.10.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| robotframework_stopwatch-0.2.10-py3-none-any.whl | Python 3 | none | any | Details for robotframework_stopwatch-0.2.10-py3-none-any.whl |
Total release size: 34.8 kB
Release files / robotframework_stopwatch-0.2.10.tar.gz
| Download URL | robotframework_stopwatch-0.2.10.tar.gz |
|---|---|
| Size | 17.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
d6a4258b5fe0b9aaf200bd739c362571839340177af5fb0fc7359bc4263e478f
|
|
BLAKE2b-256 checksum How to use checksums |
00ff258ee7243c4e5bee82f1f93b94ee42cdacee6fcbf53a83a8210f75f25e5f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.6
|
Release files / robotframework_stopwatch-0.2.10-py3-none-any.whl
| Download URL | robotframework_stopwatch-0.2.10-py3-none-any.whl |
|---|---|
| Size | 17.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
793e1803fef7acbb722de21f743d45135c80980acfa1dff9adbcd2abb589e6a8
|
|
BLAKE2b-256 checksum How to use checksums |
a1963cace8fe68f0ef86bb5bc2082dfbe99dd665cbd3398796e8ccf8f1ac2451
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.6
|
Security for robotframework-stopwatch 0.2.10
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.