Skip to main content

pytest plugin help to find coupled tests

Project description


Build Status Cov PyPI PyPI - Python Version PyPI - Wheel

Pytest plugin which help to find coupled tests.

Sometimes we have coupled tests which depend from ordering

For example:

  • PASSES tests/exmaple/ tests/exmaple/ tests/exmaple/
  • FAILED tests/exmaple/ tests/exmaple/ tests/exmaple/

In this case pretty simple to detect coupled tests, but if we have >=1k tests which called before it will hard



pip install pytest-sherlock

how to use:

pytest tests/exmaple/ tests/exmaple/ tests/exmaple/ --flaky-test="test_read_params" -vv

Plugin didn't run all tests, it try to find some possible guilty test and will run first

Try to find coupled tests in [3-4] steps
__________________________ Step [1 of 4]: __________________________

tests/exmaple/ PASSED                                                                                                      [ 20%]
tests/exmaple/ PASSED                                                                                                      [ 40%]
tests/exmaple/ PASSED                                                                                                           [ 60%]
tests/exmaple/ PASSED                                                                                                            [ 80%]
tests/exmaple/ FAILED                                                                                                             [100%]
__________________________ Step [2 of 4]: __________________________

tests/exmaple/ PASSED                                                                                                      [ 33%]
tests/exmaple/ PASSED                                                                                                      [ 66%]
tests/exmaple/ FAILED                                                                                                             [100%]
__________________________ Step [3 of 4]: __________________________

tests/exmaple/ PASSED                                                                                                      [ 50%]
tests/exmaple/ PASSED                                                                                                              [100%]
__________________________ Step [4 of 4]: __________________________

tests/exmaple/ PASSED                                                                                                      [ 50%]
tests/exmaple/ FAILED                                                                                                             [100%]

============================== FAILURES ==============================
__________________________ test_read_params __________________________

Found coupled tests:

Common fixtures:

How to reproduce:
pytest -l -vv tests/exmaple/ tests/exmaple/

AssertionError: assert 13 == 2
 +  where 13 = <built-in method get of dict object at 0x102664280>('b')
 +    where <built-in method get of dict object at 0x102664280> = {'a': 1, 'b': 13, 'c': 3}.get

tests/exmaple/ AssertionError
=================== 1 failed, 9 passed in 0.08 seconds ===================


I have a couple ideas, how to improve finder coupled tests:

  • use AST for detect common peace of code (variables, functions, etc...)

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pytest-sherlock, version 0.2.2
Filename, size File type Python version Upload date Hashes
Filename, size pytest_sherlock-0.2.2-py2-none-any.whl (8.9 kB) File type Wheel Python version py2 Upload date Hashes View
Filename, size pytest-sherlock-0.2.2.tar.gz (7.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page