Skip to main content

pytest plugin to re-run tests to eliminate flaky failures

Project description

pytest-rerunfailures
====================

pytest-rerunfailures is a plugin for `py.test <http://pytest.org>`_ that
re-runs tests to eliminate intermittent failures.

.. image:: https://img.shields.io/badge/license-MPL%202.0-blue.svg
:target: https://github.com/pytest-dev/pytest-rerunfailures/blob/master/LICENSE
:alt: License
.. image:: https://img.shields.io/pypi/v/pytest-rerunfailures.svg
:target: https://pypi.python.org/pypi/pytest-rerunfailures/
:alt: PyPI
.. image:: https://img.shields.io/travis/pytest-dev/pytest-rerunfailures.svg
:target: https://travis-ci.org/pytest-dev/pytest-rerunfailures/
:alt: Travis

Requirements
------------

You will need the following prerequisites in order to use pytest-rerunfailures:

- Python 2.7, 3.4, 3.5, 3.6, 3.7, PyPy, or PyPy3
- pytest 3.8 or newer

This package is currently tested against the last 5 minor pytest releases. In
case you work with an older version of pytest you should consider updating or
use one of the earlier versions of this package.

Installation
------------

To install pytest-rerunfailures:

.. code-block:: bash

$ pip install pytest-rerunfailures

Re-run all failures
-------------------

To re-run all test failures, use the ``--reruns`` command line option with the
maximum number of times you'd like the tests to run:

.. code-block:: bash

$ pytest --reruns 5

Failed fixture or setup_class will also be re-executed.

To add a delay time between re-runs use the ``--reruns-delay`` command line
option with the amount of seconds that you would like wait before the next
test re-run is launched:

.. code-block:: bash

$ pytest --reruns 5 --reruns-delay 1

Re-run individual failures
--------------------------

To mark individual tests as flaky, and have them automatically re-run when they
fail, add the ``flaky`` mark with the maximum number of times you'd like the
test to run:

.. code-block:: python

@pytest.mark.flaky(reruns=5)
def test_example():
import random
assert random.choice([True, False])

Note that when teardown fails, two reports are generated for the case, one for
the test case and the other for the teardown error.

You can also specify the re-run delay time in the marker:

.. code-block:: python

@pytest.mark.flaky(reruns=5, reruns_delay=2)
def test_example():
import random
assert random.choice([True, False])

Output
------

Here's an example of the output provided by the plugin when run with
``--reruns 2`` and ``-r aR``::

test_report.py RRF

================================== FAILURES ==================================
__________________________________ test_fail _________________________________

def test_fail():
> assert False
E assert False

test_report.py:9: AssertionError
============================ rerun test summary info =========================
RERUN test_report.py::test_fail
RERUN test_report.py::test_fail
============================ short test summary info =========================
FAIL test_report.py::test_fail
======================= 1 failed, 2 rerun in 0.02 seconds ====================

Note that output will show all re-runs. Tests that fail on all the re-runs will
be marked as failed.

Compatibility
-------------

* This plugin may *not* be used with class, module, and package level fixtures.
* This plugin is *not* compatible with pytest-xdist's --looponfail flag.
* This plugin is *not* compatible with the core --pdb flag.

Resources
---------

- `Issue Tracker <http://github.com/pytest-dev/pytest-rerunfailures/issues>`_
- `Code <http://github.com/pytest-dev/pytest-rerunfailures/>`_

Development
-----------

* Test execution count can be retrieved from the ``execution_count`` attribute in test ``item``'s object. Example:

.. code-block:: python

@hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
print(item.execution_count)


Changelog
---------

6.0 (2019-01-08)
================

Backwards incompatible changes
++++++++++++++++++++++++++++++

- Drop support for pytest version 3.6 and 3.7.

Features
++++++++

- Add support for pytest version 4.0 and 4.1.

Bug fixes
+++++++++

- Fixed #77 regression issue introduced in 4.2 related to the ``rerun``
attribute on the test report.
(`#77 <https://github.com/pytest-dev/pytest-rerunfailures/issues/77>`_)
(Thanks to `@RibeiroAna`_ for the PR).

.. _@RibeiroAna: https://github.com/RibeiroAna


5.0 (2018-11-06)
================

- Drop support for pytest versions < 3.6 to reduce the maintenance burden.

- Add support up to pytest version 3.10. Thus supporting the newest 5 pytest
releases.

- Add support for Python 3.7.

- Fix issue can occur when used together with `pytest-flake8`
(`#73 <https://github.com/pytest-dev/pytest-rerunfailures/issues/73>`_)


4.2 (2018-10-04)
================

- Fixed #64 issue related to ``setup_class`` and ``fixture`` executions on rerun (Thanks to
`@OlegKuzovkov`_ for the PR).

- Added new ``execution_count`` attribute to reflect the number of test case executions according to #67 issue.
(Thanks to `@OlegKuzovkov`_ for the PR).

.. _@OlegKuzovkov: https://github.com/OlegKuzovkov


4.1 (2018-05-23)
================

- Add support for pytest 3.6 by using ``Node.get_closest_marker()`` (Thanks to
`@The-Compiler`_ for the PR).

.. _@The-Compiler: https://github.com/The-Compiler

4.0 (2017-12-23)
================

- Added option to add a delay time between test re-runs (Thanks to `@Kanguros`_
for the PR).

- Added support for pytest >= 3.3.

- Drop support for pytest < 2.8.7.

.. _@Kanguros: https://github.com/Kanguros


3.1 (2017-08-29)
================

- Restored compatibility with pytest-xdist. (Thanks to `@davehunt`_ for the PR)

.. _@davehunt: https://github.com/davehunt


3.0 (2017-08-17)
================

- Add support for Python 3.6.

- Add support for pytest 2.9 up to 3.2

- Drop support for Python 2.6 and 3.3.

- Drop support for pytest < 2.7.


2.2 (2017-06-23)
================

- Ensure that other plugins can run after this one, in case of a global setting
``--rerun=0``. (Thanks to `@sublee`_ for the PR)

.. _@sublee: https://github.com/sublee

2.1.0 (2016-11-01)
==================

- Add default value of ``reruns=1`` if ``pytest.mark.flaky()`` is called
without arguments.

- Also offer a distribution as universal wheel. (Thanks to `@tltx`_ for the PR)

.. _@tltx: https://github.com/tltx


2.0.1 (2016-08-10)
==================

- Prepare CLI options to pytest 3.0, to avoid a deprecation warning.

- Fix error due to missing CHANGES.rst when creating the source distribution
by adding a MANIFEST.in.


2.0.0 (2016-04-06)
==================

- Drop support for Python 3.2, since supporting it became too much of a hassle.
(Reason: Virtualenv 14+ / PIP 8+ do not support Python 3.2 anymore.)


1.0.2 (2016-03-29)
==================

- Add support for `--resultlog` option by parsing reruns accordingly. (#28)


1.0.1 (2016-02-02)
==================

- Improve package description and include CHANGELOG into description.


1.0.0 (2016-02-02)
==================

- Rewrite to use newer API of pytest >= 2.3.0

- Improve support for pytest-xdist by only logging the final result.
(Logging intermediate results will finish the test rather rerunning it.)

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pytest_rerunfailures-6.0-py2.py3-none-any.whl (10.8 kB) Copy SHA256 hash SHA256 Wheel py2.py3
pytest-rerunfailures-6.0.tar.gz (12.3 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page