pytest plugin to re-run tests to eliminate flaky failures
- 10.0 (2021-05-26)
- 9.1.1 (2020-09-29)
- 9.1 (2020-08-26)
- 9.0 (2020-03-18)
- 8.0 (2019-11-18)
- 7.0 (2019-03-28)
- 6.0 (2019-01-08)
- 5.0 (2018-11-06)
- 4.2 (2018-10-04)
- 4.1 (2018-05-23)
- 4.0 (2017-12-23)
- 3.1 (2017-08-29)
- 3.0 (2017-08-17)
- 2.2 (2017-06-23)
- 2.1.0 (2016-11-01)
- 2.0.1 (2016-08-10)
- 2.0.0 (2016-04-06)
- 1.0.2 (2016-03-29)
- 1.0.1 (2016-02-02)
- 1.0.0 (2016-02-02)
pytest-rerunfailures is a plugin for pytest that re-runs tests to eliminate intermittent failures.
You will need the following prerequisites in order to use pytest-rerunfailures:
- Python 3.6, up to 3.9, or PyPy3
- pytest 5.3 or newer
This package is currently tested against the last 5 minor pytest releases. In case you work with an older version of pytest you should consider updating or use one of the earlier versions of this package.
To install pytest-rerunfailures:
$ pip install pytest-rerunfailures
To re-run all test failures, use the --reruns command line option with the maximum number of times you’d like the tests to run:
$ pytest --reruns 5
Failed fixture or setup_class will also be re-executed.
To add a delay time between re-runs use the --reruns-delay command line option with the amount of seconds that you would like wait before the next test re-run is launched:
$ pytest --reruns 5 --reruns-delay 1
To re-run only those failures that match a certain list of expressions, use the --only-rerun flag and pass it a regular expression. For example, the following would only rerun those errors that match AssertionError:
$ pytest --reruns 5 --only-rerun AssertionError
Passing the flag multiple times accumulates the arguments, so the following would only rerun those errors that match AssertionError or ValueError:
$ pytest --reruns 5 --only-rerun AssertionError --only-rerun ValueError
To mark individual tests as flaky, and have them automatically re-run when they fail, add the flaky mark with the maximum number of times you’d like the test to run:
@pytest.mark.flaky(reruns=5) def test_example(): import random assert random.choice([True, False])
Note that when teardown fails, two reports are generated for the case, one for the test case and the other for the teardown error.
You can also specify the re-run delay time in the marker:
@pytest.mark.flaky(reruns=5, reruns_delay=2) def test_example(): import random assert random.choice([True, False])
You can also specify an optional condition in the re-run marker:
@pytest.mark.flaky(reruns=5, condition=sys.platform.startswith("win32")) def test_example(): import random assert random.choice([True, False])
Note that the test will re-run for any condition that is truthy.
Here’s an example of the output provided by the plugin when run with --reruns 2 and -r aR:
test_report.py RRF ================================== FAILURES ================================== __________________________________ test_fail _________________________________ def test_fail(): > assert False E assert False test_report.py:9: AssertionError ============================ rerun test summary info ========================= RERUN test_report.py::test_fail RERUN test_report.py::test_fail ============================ short test summary info ========================= FAIL test_report.py::test_fail ======================= 1 failed, 2 rerun in 0.02 seconds ====================
Note that output will show all re-runs. Tests that fail on all the re-runs will be marked as failed.
- This plugin may not be used with class, module, and package level fixtures.
- This plugin is not compatible with pytest-xdist’s –looponfail flag.
- This plugin is not compatible with the core –pdb flag.
Test execution count can be retrieved from the execution_count attribute in test item’s object. Example:
@hookimpl(tryfirst=True) def pytest_runtest_makereport(item, call): print(item.execution_count)
- Drop support for Python 3.5.
- Drop support for pytest < 5.3.
- Check for the resultlog by feature and not by version as pytest master does not provide a consistent version.
- Ignore --result-log command line option when used together with pytest >= 6.1.0, as it was removed there. This is a quick fix, use an older version of pytest, if you want to keep this feature for now. (Thanks to @ntessore for the PR)
- Support up to pytest 6.1.0.
- Add a new flag --only-rerun to allow for users to rerun only certain errors.
- Drop support for pytest version 4.4, 4.5 and 4.6.
- Drop support for Python 2.7.
- Add support for pytest 5.4.
- Add support for Python 3.8.
- Drop support for pytest version 3.10, 4.0, 4.1, 4.2 and 4.3
- Drop support for Python 3.4.
- Add support for pytest version 4.4, 4.5, 4.6, 5.0, 5.1 and 5.2.
- Drop support for pytest version 3.8 and 3.9.
- Add support for pytest version 4.2 and 4.3.
- Drop support for pytest version 3.6 and 3.7.
- Add support for pytest version 4.0 and 4.1.
- Drop support for pytest versions < 3.6 to reduce the maintenance burden.
- Add support up to pytest version 3.10. Thus supporting the newest 5 pytest releases.
- Add support for Python 3.7.
- Fix issue can occur when used together with pytest-flake8 (#73)
- Add support for pytest 3.6 by using Node.get_closest_marker() (Thanks to @The-Compiler for the PR).
- Added option to add a delay time between test re-runs (Thanks to @Kanguros for the PR).
- Added support for pytest >= 3.3.
- Drop support for pytest < 2.8.7.
- Add support for Python 3.6.
- Add support for pytest 2.9 up to 3.2
- Drop support for Python 2.6 and 3.3.
- Drop support for pytest < 2.7.
- Ensure that other plugins can run after this one, in case of a global setting --rerun=0. (Thanks to @sublee for the PR)
- Add default value of reruns=1 if pytest.mark.flaky() is called without arguments.
- Also offer a distribution as universal wheel. (Thanks to @tltx for the PR)
- Prepare CLI options to pytest 3.0, to avoid a deprecation warning.
- Fix error due to missing CHANGES.rst when creating the source distribution by adding a MANIFEST.in.
- Drop support for Python 3.2, since supporting it became too much of a hassle. (Reason: Virtualenv 14+ / PIP 8+ do not support Python 3.2 anymore.)
- Add support for –resultlog option by parsing reruns accordingly. (#28)
- Improve package description and include CHANGELOG into description.
- Rewrite to use newer API of pytest >= 2.3.0
- Improve support for pytest-xdist by only logging the final result. (Logging intermediate results will finish the test rather rerunning it.)
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size pytest_rerunfailures-10.0-py3-none-any.whl (9.0 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size pytest-rerunfailures-10.0.tar.gz (12.7 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for pytest_rerunfailures-10.0-py3-none-any.whl
Hashes for pytest-rerunfailures-10.0.tar.gz