Adds the ability to retry flaky tests in CI environments
pytest-retry is a plugin for Pytest which adds the ability to retry flaky tests, thereby improving the consistency of the test suite results.
pytest-retry is designed for the latest versions of Python and Pytest. Python 3.9+ and pytest 7.0.0 are required.
Use pip to install pytest-retry:
$ pip install pytest-retry
There are two main ways to use pytest-retry:
1. Command line
Run Pytest with the command line argument --retries in order to retry every test in the event of a failure. The following example will retry each failed up to two times before proceeding to the next test:
$ python -m pytest --retries 2
An optional delay can be specified using the --retry-delay argument. This will insert a fixed delay (in seconds) between each attempt when a test fails. This can be useful if the test failures are due to intermittent environment issues which clear up after a few seconds
$ python -m pytest --retries 2 --retry-delay 5
2. Pytest flaky mark
Mark individual tests as 'flaky' to retry them when they fail. If no command line arguments are passed, only the marked tests will be retried. The default values are 1 retry attempt with a 0-second delay
The number of times each test will be retried and/or the delay can be manually specified as well
Finally, there is both a command line option and flaky mark argument for the test
timing method, which can either be
overwrite (default) or
cumulative timing, the duration of each test attempt is summed for the overall test
duration reported at the end. The default behavior simply uses the timing for
the final attempt.
If you're not sure which to use, stick with the default
overwrite method. This
generally plays nicer with time-based test splitting algorithms and will result in
more even splits.
The flaky mark will override any command line options if passed when running Pytest.
Things to consider
Currently, failing test fixtures are not retried. In the future, flaky test setup may be retried, although given the undesirability of flaky tests in general, flaky setup should be avoided at all costs. Any failures during teardown will immediately halt further attempts so that they can be addressed immediately. Make sure your teardowns always work reliably regardless of the number of retries when using this plugin
When a flaky test is retried, the plugin runs teardown steps for the test as if it had passed. This is to ensure that any partial state created by the test is cleaned up before the next attempt so that subsequent attempts do not conflict with one another. Class and module fixtures are included in this teardown with the assumption that false test failures should be a rare occurrence and the performance hit from re-running these potentially expensive fixtures is worth it to ensure clean initial test state. With feedback, the option to not re-run class and module fixtures may be added, but in general, these types of fixtures should be avoided for known flaky tests.
Flaky tests are not sustainable. This plugin is designed as an easy short-term solution while a permanent fix is implemented. Use the reports generated by this plugin to identify issues with the tests or testing environment and resolve them.
pytest-retry intercepts the standard Pytest report flow in order to retry tests and update the reports as required. When a test is retried at least once, an R is printed to the live test output and the counter of retried tests is incremented by 1. After the test session has completed, an additional report is generated below the standard output which lists all of the tests which were retried, along with the exceptions that occurred during each failed attempt.
collected 1 item
test_retry_passes_after_temporary_test_failure.py R. [100%]
======================= the following tests were retried =======================
test_eventually_passes failed on attempt 1! Retrying!
Traceback (most recent call last):
File "tests/test_example.py", line 4, in test_eventually_passes
assert len(a) > 1
AssertionError: assert 1 > 1
+ where 1 = len()
=========================== end of test retry report ===========================
========================= 1 passed, 1 retried in 0.01s =========================
Tests which have been retried but eventually pass are counted as both retried and passed, and tests which have been retried but eventually fail are counted as both retried and failed. Skipped, xfailed, and xpassed tests are never retried.
Three pytest stash keys are available to import from the pytest_retry plugin:
duration_key. These keys are used by the plugin
to store the number of attempts each item has undergone, whether the test passed or
failed, and the total duration from setup to teardown, respectively. (If any stage of
setup, call, or teardown fails, a test is considered failed overall). These stash keys
can be used to retrieve these reports for use in your own hooks or plugins.
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for pytest_retry-1.2.0-py3-none-any.whl