pytest plugin for regression tests
pytest-regtest is a pytest-plugin for implementing regression tests. Compared to functional testing a regression test does not test if software produces correct results, instead a regression test checks if software behaves the same way as it did before introduced changes.
More about regression testing at https://en.wikipedia.org/wiki/Regression_testing. Regression testing is a common technique to get started when refactoring legacy code lacking a test suite.
pytest-regtest allows capturing selected output which then can be compared to the captured output from former runs.
To install and activate this plugin execute:
$ pip install pytest-regtest
pytest-regtest plugin provides a fixture named regtest which can be used as a file handle for recording data:
from __future__ import print_function def test_squares_up_to_ten(regtest): result = [i*i for i in range(10)] # one way to record output: print(result, file=regtest) # alternative method to record output: regtest.write("done")
If you run this test script with pytest the first time there is no recorded output for this test function so far and thus the test will fail with a message including a diff:
$ py.test ... regression test output differences for test_demo.py::test_squares_up_to_ten: > --- current > +++ tobe > @@ -1,2 +1 @@ > -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81] > -done > +
The output tells us what the current output is, and that the “tobe” output is still empty.
For accepting this output, we run pytest with the –reset-regtest flag:
$ py.test --regtest-reset
Now the next execution of py.test will succeed:
Now we break the test by modifying the code under test to compute the first eleven square numbers:
from __future__ import print_function def test_squares_up_to_ten(regtest): result = [i*i for i in range(11)] # changed ! # one way to record output: print(result, file=regtest) # alternative method to record output: regtest.write("done")
The next run of pytest delivers a nice diff of the current and expected output from this test function:
$ py.test ... > --- current > +++ tobe > @@ -1,2 +1,2 @@ > -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100] > +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81] > done
The recorded output was written to files in the subfolder _regtest_outputs next to your test script(s). You might keep this folder under version control.
Another way to record output is the regtest_redirect fixture:
def test_squares_up_to_ten(regtest_redirect): result = [i*i for i in range(10)] with regtest_redirect(): print result
You can reset recorded output of files and functions individually as:
$ py.test --regtest-reset tests/test_00.py $ py.test --regtest-reset tests/test_00.py::test_squares_up_to_ten
To supress the diff and only see the stats use:
$ py.test --regtest-nodiff
To see recorded output during test execution run:
$ py.test --regtest-tee -s
If you develop on mixed platforms it might be usefull to ignore white spaces at the end of the lines when comparing output. This can be achieved by specifying:
$ py.test --regtest-ignore-line-endings