pytest plugin to re-run tests to eliminate flaky failures
- 10.2 (2021-09-17)
- 10.1 (2021-07-02)
- 10.0 (2021-05-26)
- 9.1.1 (2020-09-29)
- 9.1 (2020-08-26)
- 9.0 (2020-03-18)
- 8.0 (2019-11-18)
- 7.0 (2019-03-28)
- 6.0 (2019-01-08)
- 5.0 (2018-11-06)
- 4.2 (2018-10-04)
- 4.1 (2018-05-23)
- 4.0 (2017-12-23)
- 3.1 (2017-08-29)
- 3.0 (2017-08-17)
- 2.2 (2017-06-23)
- 2.1.0 (2016-11-01)
- 2.0.1 (2016-08-10)
- 2.0.0 (2016-04-06)
- 1.0.2 (2016-03-29)
- 1.0.1 (2016-02-02)
- 1.0.0 (2016-02-02)
pytest-rerunfailures is a plugin for pytest that re-runs tests to eliminate intermittent failures.
You will need the following prerequisites in order to use pytest-rerunfailures:
- Python 3.6, up to 3.10, or PyPy3
- pytest 5.3 or newer
This plugin can recover from a hard crash with the following optional prerequisites:
- pytest-xdist 2.3.0 or newer
This package is currently tested against the last 5 minor pytest releases. In case you work with an older version of pytest you should consider updating or use one of the earlier versions of this package.
To install pytest-rerunfailures:
$ pip install pytest-rerunfailures
If one or more tests trigger a hard crash (for example: segfault), this plugin will ordinarily be unable to rerun the test. However, if a compatible version of pytest-xdist is installed, and the tests are run within pytest-xdist using the -n flag, this plugin will be able to rerun crashed tests, assuming the workers and controller are on the same LAN (this assumption is valid for almost all cases because most of the time the workers and controller are on the same computer). If this assumption is not the case, then this functionality may not operate.
To re-run all test failures, use the --reruns command line option with the maximum number of times you’d like the tests to run:
$ pytest --reruns 5
Failed fixture or setup_class will also be re-executed.
To add a delay time between re-runs use the --reruns-delay command line option with the amount of seconds that you would like wait before the next test re-run is launched:
$ pytest --reruns 5 --reruns-delay 1
To re-run only those failures that match a certain list of expressions, use the --only-rerun flag and pass it a regular expression. For example, the following would only rerun those errors that match AssertionError:
$ pytest --reruns 5 --only-rerun AssertionError
Passing the flag multiple times accumulates the arguments, so the following would only rerun those errors that match AssertionError or ValueError:
$ pytest --reruns 5 --only-rerun AssertionError --only-rerun ValueError
To mark individual tests as flaky, and have them automatically re-run when they fail, add the flaky mark with the maximum number of times you’d like the test to run:
@pytest.mark.flaky(reruns=5) def test_example(): import random assert random.choice([True, False])
Note that when teardown fails, two reports are generated for the case, one for the test case and the other for the teardown error.
You can also specify the re-run delay time in the marker:
@pytest.mark.flaky(reruns=5, reruns_delay=2) def test_example(): import random assert random.choice([True, False])
You can also specify an optional condition in the re-run marker:
@pytest.mark.flaky(reruns=5, condition=sys.platform.startswith("win32")) def test_example(): import random assert random.choice([True, False])
You can use @pytest.mark.flaky(condition) similarly as @pytest.mark.skipif(condition), see pytest-mark-skipif
@pytest.mark.flaky(reruns=2,condition="sys.platform.startswith('win32')") def test_example(): import random assert random.choice([True, False]) # totally same as the above @pytest.mark.flaky(reruns=2,condition=sys.platform.startswith("win32")) def test_example(): import random assert random.choice([True, False])
Note that the test will re-run for any condition that is truthy.
Here’s an example of the output provided by the plugin when run with --reruns 2 and -r aR:
test_report.py RRF ================================== FAILURES ================================== __________________________________ test_fail _________________________________ def test_fail(): > assert False E assert False test_report.py:9: AssertionError ============================ rerun test summary info ========================= RERUN test_report.py::test_fail RERUN test_report.py::test_fail ============================ short test summary info ========================= FAIL test_report.py::test_fail ======================= 1 failed, 2 rerun in 0.02 seconds ====================
Note that output will show all re-runs. Tests that fail on all the re-runs will be marked as failed.
- This plugin may not be used with class, module, and package level fixtures.
- This plugin is not compatible with pytest-xdist’s –looponfail flag.
- This plugin is not compatible with the core –pdb flag.
Test execution count can be retrieved from the execution_count attribute in test item’s object. Example:
@hookimpl(tryfirst=True) def pytest_runtest_makereport(item, call): print(item.execution_count)
- Allow recovery from crashed tests with pytest-xdist.
- Add support for Python 3.10 (as of Python 3.10.rc2). (Thanks to @hugovk for the PR.)
- Drop support for Python 3.5.
- Drop support for pytest < 5.3.
- Check for the resultlog by feature and not by version as pytest master does not provide a consistent version.
- Ignore --result-log command line option when used together with pytest >= 6.1.0, as it was removed there. This is a quick fix, use an older version of pytest, if you want to keep this feature for now. (Thanks to @ntessore for the PR)
- Support up to pytest 6.1.0.
- Add a new flag --only-rerun to allow for users to rerun only certain errors.
- Drop support for pytest version 4.4, 4.5 and 4.6.
- Drop support for Python 2.7.
- Add support for pytest 5.4.
- Add support for Python 3.8.
- Drop support for pytest version 3.10, 4.0, 4.1, 4.2 and 4.3
- Drop support for Python 3.4.
- Add support for pytest version 4.4, 4.5, 4.6, 5.0, 5.1 and 5.2.
- Drop support for pytest version 3.8 and 3.9.
- Add support for pytest version 4.2 and 4.3.
- Drop support for pytest version 3.6 and 3.7.
- Add support for pytest version 4.0 and 4.1.
- Drop support for pytest versions < 3.6 to reduce the maintenance burden.
- Add support up to pytest version 3.10. Thus supporting the newest 5 pytest releases.
- Add support for Python 3.7.
- Fix issue can occur when used together with pytest-flake8 (#73)
- Add support for pytest 3.6 by using Node.get_closest_marker() (Thanks to @The-Compiler for the PR).
- Added option to add a delay time between test re-runs (Thanks to @Kanguros for the PR).
- Added support for pytest >= 3.3.
- Drop support for pytest < 2.8.7.
- Add support for Python 3.6.
- Add support for pytest 2.9 up to 3.2
- Drop support for Python 2.6 and 3.3.
- Drop support for pytest < 2.7.
- Ensure that other plugins can run after this one, in case of a global setting --rerun=0. (Thanks to @sublee for the PR)
- Add default value of reruns=1 if pytest.mark.flaky() is called without arguments.
- Also offer a distribution as universal wheel. (Thanks to @tltx for the PR)
- Prepare CLI options to pytest 3.0, to avoid a deprecation warning.
- Fix error due to missing CHANGES.rst when creating the source distribution by adding a MANIFEST.in.
- Drop support for Python 3.2, since supporting it became too much of a hassle. (Reason: Virtualenv 14+ / PIP 8+ do not support Python 3.2 anymore.)
- Add support for –resultlog option by parsing reruns accordingly. (#28)
- Improve package description and include CHANGELOG into description.
- Rewrite to use newer API of pytest >= 2.3.0
- Improve support for pytest-xdist by only logging the final result. (Logging intermediate results will finish the test rather rerunning it.)
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size pytest_rerunfailures-10.2-py3-none-any.whl (11.3 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
|Filename, size pytest-rerunfailures-10.2.tar.gz (15.9 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for pytest_rerunfailures-10.2-py3-none-any.whl
Hashes for pytest-rerunfailures-10.2.tar.gz