pytest plugin help to find coupled tests
Project description
pytest-sherlock
Pytest plugin which help to find coupled tests.
Sometimes we have coupled tests which depend from ordering
For example:
- PASSES
tests/exmaple/test_all_read.py tests/exmaple/test_b_modify.py tests/exmaple/test_c_delete.py
- FAILED
tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py
In this case pretty simple to detect coupled tests, but if we have >=1k tests which called before it will hard
Content:
Install
pip install pytest-sherlock
how to use:
pytest tests/exmaple/test_c_delete.py tests/exmaple/test_b_modify.py tests/exmaple/test_all_read.py --flaky-test="test_read_params" -vv
Plugin didn't run all tests, it try to find some possible guilty test and will run first
======================= test session starts =======================
collected 10 items
Try to find coupled tests
Step #1:
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 20%]
tests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 40%]
tests/exmaple/test_c_delete.py::test_deleted_passed PASSED [ 60%]
tests/exmaple/test_c_delete.py::test_do_not_delete PASSED [ 80%]
tests/exmaple/test_all_read.py::test_read_params COUPLED [100%]
Step #2:
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 33%]
tests/exmaple/test_c_delete.py::test_delete_random_param PASSED [ 66%]
tests/exmaple/test_all_read.py::test_read_params COUPLED [100%]
Step #3:
tests/exmaple/test_b_modify.py::test_modify_random_param PASSED [ 50%]
tests/exmaple/test_all_read.py::test_read_params COUPLED [100%]
======================== found coupled in 0.08 seconds ========================
found coupled tests:
- tests/exmaple/test_b_modify.py::test_modify_random_param
- tests/exmaple/test_all_read.py::test_read_params
pytest -l -vv tests/exmaple/test_b_modify.py::test_modify_random_param tests/exmaple/test_all_read.py::test_read_params
_____________________________ test_read_params __________________________________
config = {'a': 1, 'b': 13, 'c': 3}, param = 'b'
def test_read_params(config, param):
> assert config.get(param) == 2
E AssertionError: assert 13 == 2
E + where 13 = <built-in method get of dict object at 0x107c464b0>('b')
E + where <built-in method get of dict object at 0x107c464b0> = {'a': 1, 'b': 13, 'c': 3}.get
tests/exmaple/test_all_read.py:8: AssertionError
======================= 7 passed, 3 coupled in 0.08 seconds =======================
TODO
I have a couple ideas, how to improve finder coupled tests:
- use AST for detect common peace of code (variables, functions, etc...)
- Also need to add tests for it =)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pytest-sherlock-0.1.13.tar.gz
(6.1 kB
view hashes)
Built Distribution
Close
Hashes for pytest_sherlock-0.1.13-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0f9ef31871f1d24df7c701fef082fb12ebafa491f15eefbda8fd435e3f3940c6 |
|
MD5 | 760c5d03f384f150d745a70818f0b442 |
|
BLAKE2b-256 | 7cdf6e9512501a8129f7a683a71b9b6b9bba9e2a5325adf66e4bdfd6cae683f7 |