Skip to main content

Runs tests multiple times to expose flakiness.

Project description

Build Status

Runs tests multiple times to expose flakiness.


This Pytest plugin was generated with Cookiecutter along with @hackebrot’s Cookiecutter-pytest-plugin template.

Features

  • When enabled it will ‘multiply’ your tests so that they get run multiple times without having to restart pytest. This helps you find flakiness in your tests.
  • You can limit your flake runs to a particular timeout value.

Installation

Install with setup.py:

python setup.py install

For best flake-finding combine with pytest-xdist:

pip install pytest-xdist

Usage

Flake Finding

Enable plugin for tests:

py.test --flake-finder

This will run every test the default, 50, times. Every test is run independently and you can even use xdist to send tests to multiple processes.

To configure the number of runs:

py.test --flake-finder --flake-runs=runs

To find flakes in one test or a couple of tests you can use pytest’s built in test selectiong.

Finding flakes in one test:

py.test -k test_maybe_flaky --flake-finder

When used with xdist the flake finder can expose many timing related flakes.

Timing Out

When using flake-finder as part of a CI run it might be useful to limit the amount of time it will run.

Running with timeout:

py.test --flake-finder --flake-max-minutes=minutes

Tests started after the timeout will be skipped.

Contributing

Contributions are very welcome. Tests can be run with tox, please ensure the coverage at least stays the same before you submit a pull request.

Issues

If you encounter any problems, please file an issue along with a detailed description.

Project details


Release history Release notifications

This version
History Node

0.1.1

History Node

0.1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
pytest-flakefinder-0.1.1.tar.gz (6.4 kB) Copy SHA256 hash SHA256 Source None Jan 11, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page