Skip to main content

a tool to detect test pollution

Project description

build status pre-commit.ci status

detect-test-pollution

a tool to detect test pollution

installation

pip install detect-test-pollution

what is test pollution?

video about test pollution

test pollution is where a test fails due to the side-effects of some other test in the test suite.

it usually appears as a "test flake" something where the test fails mysteriously but passes when run by itself.

a simple example of this is the following python code:

k = 1

def test_k():
    assert k == 1

def test_k2():
    global k

    k = 2
    assert k == 2

now this example is a little bit silly, you probably wouldn't write code this poorly but helps us demonstrate the problem here.

when run normally -- these tests pass:

$ pytest -q t.py
..                                                                       [100%]
2 passed in 0.00s

but, if the tests were run in some other order (due to something like pytest-randomly or pytest-xdist) then the pollution would be apparent:

$ pytest -q t.py::test_k2 t.py::test_k
.F                                                                       [100%]
=================================== FAILURES ===================================
____________________________________ test_k ____________________________________

    def test_k():
>       assert k == 1
E       assert 2 == 1

t.py:4: AssertionError
=========================== short test summary info ============================
FAILED t.py::test_k - assert 2 == 1
1 failed, 1 passed in 0.03s

often this flake happens in a codebase with hundreds or thousands of tests and it's difficult to track down which test is causing the global side-effects.

that's where this tool comes in handy! it helps you find the pair of tests which error when run in order.

usage

video about using detect-test-pollution

once you have identified a failing test, you'll be able to feed it into detect-test-pollution to find the causal test.

the basic mode is to run:

detect-test-pollution \
    --failing-test test.py::test_id_here \
    --tests ./tests

where test.py::test_id_here is the identifier of the failing test and ./tests is the directory where your testsuite lives.

if you've already narrowed down the list of testids further than that, you can specify a --testids-file instead of --tests to speed up discovery:

detect-test-pollution \
    --failing-test test.py::test_id_here \
    --testids-file ./testids

you can usually get a list of testids via pytest --collect-only -q (though you'll need to strip some unrelated lines at the end, such as timing and warning info).

then detect-test-pollution will bisect the list of tests to find the failing one. here's an example bisection from a bug in pytest

$ detect-test-pollution --tests ./testing --failing-test testing/io/test_terminalwriter.py::test_should_do_markup_FORCE_COLOR
discovering all tests...
-> discovered 3140 tests!
ensuring test passes by itself...
-> OK!
ensuring test fails with test group...
-> OK!
running step 1:
- 3139 tests remaining (about 12 steps)
running step 2:
- 1570 tests remaining (about 11 steps)
running step 3:
- 785 tests remaining (about 10 steps)
running step 4:
- 393 tests remaining (about 9 steps)
running step 5:
- 197 tests remaining (about 8 steps)
running step 6:
- 99 tests remaining (about 7 steps)
running step 7:
- 50 tests remaining (about 6 steps)
running step 8:
- 25 tests remaining (about 5 steps)
running step 9:
- 12 tests remaining (about 4 steps)
running step 10:
- 6 tests remaining (about 3 steps)
running step 11:
- 3 tests remaining (about 2 steps)
double checking we found it...
-> the polluting test is: testing/test_terminal.py::TestTerminal::test_report_teststatus_explicit_markup

fuzzing

detect-test-pollution can also be used to "fuzz" out failing tests.

it does this by shuffling the test ids and running the testsuite until it fails.

here's an example execution on a silly testsuite:

$ detect-test-pollution --fuzz --tests t.py
discovering all tests...
-> discovered 1002 tests!
run 1...
-> OK!
run 2...
-> found failing test!
try `detect-test-pollution --failing-test t.py::test_k --tests t.py`!

afterwards you can use the normal mode of detect-test-pollution to find the failing pair.

supported test runners

at the moment only pytest is supported -- though in theory the tool could be adapted to support other python test runners, or even other languages.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

detect_test_pollution-1.2.0.tar.gz (6.8 kB view hashes)

Uploaded Source

Built Distribution

detect_test_pollution-1.2.0-py2.py3-none-any.whl (7.3 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page