pytest-regtest 2.1.0

"pytest plugin for snapshot regression testing"

Home

About

pytest-regtest is a plugin for pytest to implement regression testing.

Unlike functional testing, regression testing testing does not test whether the software produces the correct results, but whether it behaves as it did before changes were introduced.

More specifically, pytest-regtest provides snapshot testing, which implements regression testing by recording the textual output of a test function and comparing this recorded output to a reference output.

Regression testing is a common technique to implement basic testing before refactoring legacy code that lacks a test suite.

Snapshot testing can also be used to implement tests for complex outcomes, such as recording textual database dumps or the results of a scientific analysis routine.

Installation

To install and activate this plugin execute:

$ pip install pytest-regtest

Basic Usage

Write a test

The pytest-regtest plugin provides multiple fixtures. To record output, use the fixture regtest that works like a file handle:

def test_squares(regtest):

    result = [i*i for i in range(10)]

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")

You can also use the regtest_all fixture. This enables all output to stdout to be recorded in a test function.

Run the test

If you run this test script with pytest the first time there is no recorded output for this test function so far and thus the test will fail with a message including a diff:

$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.11.4, pytest-7.4.3, pluggy-1.3.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.0.3
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output not recorded yet for test_squares.py::test_squares:

[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
done
---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.02s ===============================

This is a diff of the current output is to a previously recorded output tobe. Since we did not record output yet, the diff contains no lines marked +.

Reset the test

To record the current output, we run pytest with the --reset-regtest flag:

$ pytest -v --regtest-reset test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.11.4, pytest-7.4.3, pluggy-1.3.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.0.3
collecting ... collected 1 item

test_squares.py::test_squares RESET                                      [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
the following output files have been reset:
  _regtest_outputs/test_squares.test_squares.out
============================== 1 passed in 0.00s ===============================

You can also see from the output that the recorded output is in the _regtest_outputs folder which in the same folder as the test script. Don't forget to commit this folder to your version control system!

Run the test again

When we run the test again, it succeeds:

$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.11.4, pytest-7.4.3, pluggy-1.3.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.0.3
collecting ... collected 1 item

test_squares.py::test_squares PASSED                                     [100%]

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 0
============================== 1 passed in 0.00s ===============================

Break the test

Let us break the test by changing the test function to compute 11 instead of 10 square numbers:

def test_squares(regtest):

    result = [i*i for i in range(11)]

    # one way to record output:
    print(result, file=regtest)

    # alternative method to record output:
    regtest.write("done")

The next run of pytest delivers a nice diff of the current and expected output from this test function:

$ pytest -v test_squares.py
============================= test session starts ==============================
platform darwin -- Python 3.11.4, pytest-7.4.3, pluggy-1.3.0 -- ...
cachedir: .pytest_cache
rootdir: ...
plugins: regtest-2.0.3
collecting ... collected 1 item

test_squares.py::test_squares FAILED                                     [100%]

=================================== FAILURES ===================================
_________________________________ test_squares _________________________________

regression test output differences for test_squares.py::test_squares:
(recorded output from _regtest_outputs/test_squares.test_squares.out)

>   --- current
>   +++ expected
>   @@ -1,2 +1,2 @@
>   -[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100]
>   +[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
>    done

---------------------------- pytest-regtest report -----------------------------
total number of failed regression tests: 1
=========================== short test summary info ============================
FAILED test_squares.py::test_squares
============================== 1 failed in 0.02s ===============================

Other features

Using the regtest fixture as context manager

The regtest fixture also works as a context manager to capture all output from the wrapped code block:

def test_squares(regtest):

    result = [i*i for i in range(10)]

    with regtest:
        print(result)

The regtest_all fixture

The regtest_all fixture leads to recording of all output to stdout in a test function.

def test_all(regtest_all):
    print("this line will be recorded.")
    print("and this line also.")

Reset individual tests

You can reset recorded output of files and functions individually as:

$ py.test --regtest-reset test_demo.py
$ py.test --regtest-reset test_demo.py::test_squares

Suppress diff for failed tests

To hide the diff and just show the number of lines changed, use:

$ py.test --regtest-nodiff ...

Show all recorded output

For complex diffs it helps to see the full recorded output also. To enable this use:

$ py.test --regtest-tee...

Line endings

Per default pytest-regtest ignores different line endings in the output. In case you want to disable this feature, use the -regtest-consider-line-endings flag.

Clean indeterministic output before recording

Output can contain data which is changing from test run to test run, e.g. paths created with the tmpdir fixture, hexadecimal object ids or timestamps.

Per default the plugin helps to make output more deterministic by:

• replacing all temporary folder in the output with <tmpdir...> or similar markers, depending on the origin of the temporary folder (tempfile module, tmpdir fixture, ...)
• replacing hexadecimal numbers 0x... of arbitary length by the fixed string 0x?????????.

You can also implement your own cleanup routines as described below.

Register own cleanup functions

You can register own converters in conftest.py:

import re
import pytest_regtest

@pytest_regtest.register_converter_pre
def remove_password_lines(txt):
    '''modify recorded output BEFORE the default fixes
    like temp folders or hex object ids are applied'''

    # remove lines with passwords:
    lines = txt.splitlines(keepends=True)
    lines = [l for l in lines if "password is" not in l]
    return "".join(lines)

@pytest_regtest.register_converter_post
def fix_time_measurements(txt):
    '''modify recorded output AFTER the default fixes
    like temp folders or hex object ids are applied'''

    # fix time measurements:
    return re.sub(
        "\d+(\.\d+)? seconds",
        "<SECONDS> seconds",
        txt
    )

If you register multiple converters they will be applied in the order of registration.

In case your routines replace, improve or conflict with the standard cleanup converters, you can use the flag --regtest-disable-stdconv to disable the default cleanup procedure.

Command line options summary

These are all supported command line options:

$ pytest --help
...

regression test plugin:
  --regtest-reset       do not run regtest but record current output
  --regtest-tee         print recorded results to console too
  --regtest-consider-line-endings
                        do not strip whitespaces at end of recorded lines
  --regtest-nodiff      do not show diff output for failed regresson tests
  --regtest-disable-stdconv
                        do not apply standard output converters to clean up
                        indeterministic output
...