pytest-pinned 0.1.1

A simple pytest plugin for pinning tests

pytest-pinned

A simple pytest plugin for writing pinning tests.

Pinning tests or snapshot tests or characterization tests are meant to describe (characterize) the actual behavior of an existing piece of software, and therefore protect existing behavior against unintended changes via automated testing.

This type of testing can come in handy for several reasons:

• Legacy codebase with poor or non-existing test coverage
• As broad integration tests where more specific testing may prove difficult

In my case, the main usecase is legacy and non-legacy codebases aimed for scientific computing. The implementation of, usually quite complex, scientific models are hard to test for specific behaviours, especially when they are used for research purposes. Pinning tests allow for more confident refactoring of the implementation of such models while tracing how results and predicitons changes.

pytest-pinned keeps all expected results from pinning tests in a single JSON file resulting in only a single file needing to be added to the VCS repository and diffs are also contained to this single file. The use of JSON for serialization of the expected results however imposes some restrictions on the datatypes that can be used, see the JSON type conversion table from the standard library.

Requirements

pytest-pinned has no external dependencies except for pytest itself.

Installation

You can install pytest-pinned via pip from PyPI:

$ pip install pytest-pinned

Usage

pytest-pinned expose a single pytest fixture (pinned). pinned will keep track of what test it is used in, supports usage with the standard assert statement and allows for multiple asserts in the same test.

Syntax

Simple pinning test sample:

def test_simple(pinned):
    assert(10.0 == pinned)

pytest-pinned also supports approximate comparisons using pytest.approx. See last assert statement in example below for syntax. pinned accepts the same keyword arguments as pytest.approx.

More elaborate example:

def test_elaborate(pinned):
    assert(10.0 == pinned)
    assert([1,2,3] == pinned)
    assert({'a': 1, 'b': 2} == pinned)
    assert(5.2983746239134 == pinned(rel=0.00001, abs=0.001))

Expected results

If pytest-pinned cannot find any expected results for a comparison it will fail the test and ask teh user to write new expected results.

To rewrite the expected results "from scratch", use:

$ pytest --pinned-rewrite

To update the expected results for only some tests, use:

$ pytest tests/sample_test.py::specific_test --pinned-update

To change the path where pytest-pinned stores (and loads) the expected results, use:

$ pytest --pinned-path path/to/expected/results.json

License

Distributed under the terms of the MIT license, pytest-pinned is free and open source software

Issues

If you encounter any problems, please file an issue along with a detailed description.

Contributing

Contributions are very welcome.