pytest-regtest 1.0.14

pytest plugin for regression tests

pytest-regtest is a pytest-plugin for implementing regression tests. Compared to functional testing a regression test does not test if software produces correct results, instead a regression test checks if software behaves the same way as it did before introduced changes.

More about regression testing at https://en.wikipedia.org/wiki/Regression_testing. Regression testing is a common technique to get started when refactoring legacy code lacking a test suite.

pytest-regtest allows capturing selected output which then can be compared to the captured output from former runs.

To install and activate this plugin execute:

$ pip install pytest-regtest

pytest-regtest plugin provides a fixture named regtest which can be used as a file handle for recording data:

from __future__ import print_function

def test_squares_up_to_ten(regtest):

    result = [i*i for i in range(10)]

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")

If you run this test script with pytest the first time there is no recorded output for this test function so far and thus the test will fail with a message including a diff:

$ py.test
...

regression test output differences for test_demo.py::test_squares_up_to_ten:

>   --- current
>   +++ tobe
>   @@ -1,2 +1 @@
>   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>   -done
>   +

The output tells us what the current output is, and that the “tobe” output is still empty.

For accepting this output, we run pytest with the –reset-regtest flag:

$ py.test --regtest-reset

Now the next execution of py.test will succeed:

$ py.test

Now we break the test by modifying the code under test to compute the first eleven square numbers:

from __future__ import print_function

def test_squares_up_to_ten(regtest):

    result = [i*i for i in range(11)]  # changed !

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")

The next run of pytest delivers a nice diff of the current and expected output from this test function:

$ py.test

...
>   --- current
>   +++ tobe
>   @@ -1,2 +1,2 @@
>   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
>   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>    done

The recorded output was written to files in the subfolder _regtest_outputs next to your test script(s). You might keep this folder under version control.

Other features

Another way to record output is the regtest_redirect fixture:

def test_squares_up_to_ten(regtest_redirect):

    result = [i*i for i in range(10)]

    with regtest_redirect():
        print result

You can reset recorded output of files and functions individually as:

$ py.test --regtest-reset tests/test_00.py
$ py.test --regtest-reset tests/test_00.py::test_squares_up_to_ten

To supress the diff and only see the stats use:

$ py.test --regtest-nodiff

To see recorded output during test execution run:

$ py.test --regtest-tee -s

If you develop on mixed platforms it might be usefull to ignore white spaces at the end of the lines when comparing output. This can be achieved by specifying:

$ py.test --regtest-ignore-line-endings