Skip to main content

robotframework-stopwatch

Version

Robot Framework listener extension for agile perfomance testing.

Evaluate test case execution against its statistical average run time & user-configured timedelta in.

Why?

While Robot Framework does provide an ability to setup time limits (e.g. timeouts) as a means of failing tests/suites depending on their time of execution, such approach is somewhat set in stone. Meaning that you have to manually inspect runtimes of your tests and decide upon much needed time limits. When you have single master suite, or your testing production is not quite large - this works alright. But when your number of tests is 300+, managing each timeout individually is rather straining. Add to that multiple layers of configurations such as different environments, testing tools, or downright infrastructure inconsistencies that do occur and you end up quite strained with the idea of reliable perfomance testing. This extension is an attempt to provide a reliable tool for flexible, integrated perfomance testing.

Overview

As mentioned above, robotframework-stopwatch is not quite a library, but extension of built-in Robot Framework Listener.

When first integrated, the StopwatchListener will initialise an empty logfile (by default: ./stopwatch_log.json) to store required run data for every individual test. After first run, taking into consideration the test succeeded, average runtime for this test will be set to this run time of execution. Afterwards, each succesful test runtime will be used to update the last average runtime value for the test. The user-preconfigured timedelta is then used to evaluate all future test runs status. Failed tests, whether due to timedelta exceeded or natural test fails, have no effect upon test average runtime. Extension also provides a log data divider environment to catalog same tests based on different setups.

When test with established average runtime fails due to exceeded time limit, test is marked as failed with a following message output:

Stopwatch! Test runtime exceeded the acceptable time limit.

Installation

pip install robotframework-stopwatch

Usage

robotframework-stopwatch relies on a single base class StopwatchListener, that comes with following attributes:

  • logfile (str): The path to the log file. By default: ./stopwatch_log.json

  • environment (str, optional) The marker/divider used to separate test run data across different configurations.

  • timedelta (str, optional): The time delta value used for evaluating test runtimes. Defaults to "10s". Time formats include: "5s", "2 min", "1 hour".

There are two ways to implement robotframework-stopwatch.

Listener approach

When custom listener is already in-use, or robotframework-stopwatch is needed on a global level (every suite, every test), create or modify the listener:

from robot.stopwatch import StopwatchListener

class CustomListener(StopwatchListener):
    def __init__(self, logfile, environment, timedelta):

This won't interfere with your custon listener functionality.

Command-line approach

When there is a need to set separate timedelta for different suites/tests, or setting up a listener is not required, it is possible to pass StopwatchListener and its command line arguments directly from terminal:

robot --listener robot.stopwatch.StopwatchListener:"logfile":"environment":"timedelta" some_test.robot

Concurecy

As of now, robotframework-stopwatch supports concurent read/write, making it possible to share a logfile data among multiple agents, instances, test runners. To incorporate concurent logging do rely on the most suitable tactics of shared file storage for your organisation (e.g. Azure Shared Storage).

Plans

  1. Interface for setting up test run data log in a remote database.
  2. Robust timedelta time formats.
  3. Robust logfile handling.

Metadata

Release files for robotframework-stopwatch 0.2.10

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for robotframework-stopwatch 0.2.10
File Size Uploaded Provenance Actions
robotframework_stopwatch-0.2.10.tar.gz 17.3 kB Without provenance Details for robotframework_stopwatch-0.2.10.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for robotframework-stopwatch 0.2.10
File Interpreter ABI Platform Actions
robotframework_stopwatch-0.2.10-py3-none-any.whl Python 3 none any Details for robotframework_stopwatch-0.2.10-py3-none-any.whl

Total release size: 34.8 kB

Release files / robotframework_stopwatch-0.2.10.tar.gz

Download URL robotframework_stopwatch-0.2.10.tar.gz
Size 17.3 kB
Tags Source
SHA-256 checksum
How to use checksums
d6a4258b5fe0b9aaf200bd739c362571839340177af5fb0fc7359bc4263e478f
BLAKE2b-256 checksum
How to use checksums
00ff258ee7243c4e5bee82f1f93b94ee42cdacee6fcbf53a83a8210f75f25e5f
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.12.6

Release files / robotframework_stopwatch-0.2.10-py3-none-any.whl

Download URL robotframework_stopwatch-0.2.10-py3-none-any.whl
Size 17.6 kB
Tags Python 3
SHA-256 checksum
How to use checksums
793e1803fef7acbb722de21f743d45135c80980acfa1dff9adbcd2abb589e6a8
BLAKE2b-256 checksum
How to use checksums
a1963cace8fe68f0ef86bb5bc2082dfbe99dd665cbd3398796e8ccf8f1ac2451
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.12.6

Security for robotframework-stopwatch 0.2.10

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.2.10 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page