Skip to main content

pytest plugin help to find coupled tests

Project description

pytest-sherlock

Build Status Cov PyPI PyPI - Python Version PyPI - Wheel

Pytest plugin which help to find coupled tests.

Sometimes we have coupled tests which depend from ordering

For example:

  • PASSES tests/exmaple/test_all_read.py tests/exmaple/test_b_modify.py tests/exmaple/test_c_delete.py
  • FAILED tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py

In this case pretty simple to detect coupled tests, but if we have >=1k tests which called before it will hard

Content:

Install

pip install pytest-sherlock

how to use:

pytest tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py --flaky-test="test_read_params" -vv

Plugin didn't run all tests, it try to find some possible guilty test and will run first

======================= test session starts =======================
collected 10 items                                                                                                                                                                                  
Try to find coupled tests
Step #1:
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED                                                                                                                               [ 20%]
tests/exmaple/test_c_delete.py::test_delete_random_param PASSED                                                                                                                               [ 40%]
tests/exmaple/test_c_delete.py::test_deleted_passed PASSED                                                                                                                                    [ 60%]
tests/exmaple/test_c_delete.py::test_do_not_delete PASSED                                                                                                                                     [ 80%]
tests/exmaple/test_all_read.py::test_read_params COUPLED                                                                                                                                      [100%]
Step #2:
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED                                                                                                                               [ 33%]
tests/exmaple/test_c_delete.py::test_delete_random_param PASSED                                                                                                                               [ 66%]
tests/exmaple/test_all_read.py::test_read_params COUPLED                                                                                                                                      [100%]
Step #3:
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED                                                                                                                               [ 50%]
tests/exmaple/test_all_read.py::test_read_params COUPLED                                                                                                                                      [100%]

======================== found coupled in 0.08 seconds ========================
found coupled tests: 
         - tests/exmaple/test_b_modify.py::test_modify_random_param
         - tests/exmaple/test_all_read.py::test_read_params

pytest -l -vv tests/exmaple/test_b_modify.py::test_modify_random_param tests/exmaple/test_all_read.py::test_read_params
_____________________________ test_read_params __________________________________

config = {'a': 1, 'b': 13, 'c': 3}, param = 'b'

    def test_read_params(config, param):
>       assert config.get(param) == 2
E       AssertionError: assert 13 == 2
E        +  where 13 = <built-in method get of dict object at 0x107c464b0>('b')
E        +    where <built-in method get of dict object at 0x107c464b0> = {'a': 1, 'b': 13, 'c': 3}.get

tests/exmaple/test_all_read.py:8: AssertionError
======================= 7 passed, 3 coupled in 0.08 seconds ======================= 

TODO

I have a couple ideas, how to improve finder coupled tests:

  • use AST for detect common peace of code (variables, functions, etc...)
  • Also need to add tests for it =)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pytest-sherlock, version 0.1.6
Filename, size File type Python version Upload date Hashes
Filename, size pytest_sherlock-0.1.6-py2-none-any.whl (7.8 kB) File type Wheel Python version py2 Upload date Hashes View
Filename, size pytest-sherlock-0.1.6.tar.gz (6.1 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page