Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

Saves previous test runs and allow re-execute previous pytest runs to reproduce crashes or flaky tests

Project description

http://img.shields.io/pypi/v/pytest-replay.svg https://anaconda.org/conda-forge/pytest-replay/badges/version.svg See Build Status on Travis CI See Build Status on AppVeyor https://img.shields.io/pypi/pyversions/pytest-replay.svg https://img.shields.io/badge/code%20style-black-000000.svg

Saves previous test runs and allow re-execute previous pytest runs to reproduce crashes or flaky tests


This Pytest plugin was generated with Cookiecutter along with @hackebrot’s Cookiecutter-pytest-plugin template.

Features

This plugin helps to reproduce random or flaky behavior when running tests with xdist. pytest-xdist executes tests in a non-predictable order, making it hard to reproduce a behavior seen in CI locally because there’s no convenient way to track which test executed in which worker.

This plugin records the executed node ids by each worker in the directory given by --replay-record-dir=<dir> flag, and a --replay=<file> can be used to re-run the tests from a previous run. For example:

$ pytest -n auto --replay-record-dir=build/tests/replay

This will generate files with the node id of each test executed by each worker, for example worker gw1 will generate a file .pytest-replay-gw1.txt with contents like this:

test_foo.py::test[1]
test_foo.py::test[3]
test_foo.py::test[5]
test_foo.py::test[7]
test_foo.py::test[8]

If there is a crash or a flaky failure in the tests of the worker gw1, one can take that file from the CI server and execute the tests in the same order with:

$ pytest --replay=.pytest-replay-gw1.txt

Hopefully this will make it easier to reproduce the problem and fix it.

FAQ

  1. pytest has its own cache, why use a different mechanism?

    The internal cache saves its data using json, which is not suitable in the advent of a crash because the file will not be readable.

  2. Shouldn’t the ability of selecting tests from a file be part of the pytest core?

    Sure, but let’s try to use this a bit as a separate plugin before proposing its inclusion into the core.

Installation

You can install pytest-replay via pip from PyPI:

$ pip install pytest-replay

Or with conda:

$ conda install -c conda-forge pytest-replay

Contributing

Contributions are very welcome.

Tests can be run with tox if you are using a native Python installation.

To run tests with conda, first create a virtual environment and execute tests from there (conda with Python 3.5+ in the root environment):

$ python -m venv .env
$ .env\scripts\activate
$ pip install -e . pytest-xdist
$ pytest tests

Releases

Follow these steps to make a new release:

  1. Create a new branch release-X.Y.Z from master;
  2. Update CHANGELOG.rst;
  3. Open a PR;
  4. After it is green and approved, push a new tag in the format X.Y.Z;

Travis will deploy to PyPI automatically.

Afterwards, update the recipe in conda-forge/pytest-replay-feedstock.

License

Distributed under the terms of the MIT license.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pytest-replay, version 1.1.0
Filename, size File type Python version Upload date Hashes
Filename, size pytest_replay-1.1.0-py3-none-any.whl (5.4 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size pytest-replay-1.1.0.tar.gz (9.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page