snaptol

Snapshot testing for numerical results with built in optional tolerance parameters.

snaptol is a pytest plugin for snapshot testing that comfortably handles tolerance with floating-point data. It is designed for numerical, scientific, and simulation-based code, where results may often not be exactly deterministic and can depend on randomised initial conditions.

Motivation

Consider an optimisation algorithm that may have a highly complex functional landscape or involve controlled randomness. For example, Monte Carlo simulations, eigensolvers, or iterative solvers. A test may look like,

import numpy as np

def test_something(snaptolshot):
    rng = np.random.default_rng(seed=123)
    random_initial_conditions = rng.normal(loc=0.0, scale=1e-6, size=100)
    result = very_complex_computation(random_initial_conditions)

    assert snaptolshot(rtol=1e-5, atol=1e-8) == result

snaptol allows you to easily snapshot this result and provide tolerance parameters for future comparisons. The plugin handles the storage and retrieval of data with JSON serialisation, and has support for NumPy arrays and its testing functions.

Installation

Use the package manager pip to install snaptol.

pip install snaptol

Usage

Basic usage

In a test file, add the fixture, snaptolshot,

def test_something(snaptolshot):
    result = compute_something()
    assert snaptolshot == result

When pytest runs, it will compare the result to the snapshot stored in file. Note that, on first passes, this will fail as the snapshot file does not exist. A --snaptol-update pass will need to be run, which is explained below.

Using numerical tolerance

You can specify tolerances directly,

def test_something(snaptolshot):
    result = compute_something()
    assert snaptolshot(rtol=1e-05, atol=1e-08) == result

Or equivalently, use the match method if you prefer,

def test_something(snaptolshot):
    result = compute_something()
    assert snaptolshot.match(rtol=1e-05, atol=1e-08) == result

Multiple snapshots per test

Tests can include multiple asserts against the snaptolshot fixture, each one gets its own separate snapshot file, and can use different tolerances:

def test_somethings(snaptolshot):
    result1, result2 = compute_somethings()
    assert snaptolshot(rtol=1e-5, atol=1e-5) == result1
    assert snaptolshot(rtol=1e-2, atol=1e-3) == result2

You can give an optional name to the snapshots with ["name"]:

def test_somethings(snaptolshot):
    result1, result2 = compute_somethings()
    assert snaptolshot["first"] == result1
    assert snaptolshot["second"].assert_allclose(result2, rtol=1e-2, atol=1e-3)

Updating snapshots

Initial run (or intentional changes)

When creating snapshot tests for the first time, or when the expected result in a test has intentionally changed, run,

pytest --snaptol-update

This writes new snapshots instead of comparing values.

Important behaviour

The --snaptol-update flag will only update tests failed in the previous run. Use the --snaptol-update-all flag to enforce an update on all tests.

There should exist only one snapshot comparison per test. If the setup data is needed to be shared between different snapshot comparisons, then consider factoring it into a fixture or helper function.

NumPy support

snaptol is able to snapshot NumPy arrays and also integrates directly with the following testing utilities,

• assert_allclose
• assert_array_almost_equal_nulp
• assert_array_max_ulp
• assert_array_equal
• assert_equal
• assert_string_equal

For example,

import numpy as np

def test_numpy_allclose(snaptolshot):
    result = np.arange(100, dtype=float)
    snaptolshot.assert_allclose(result, rtol=1e-05, atol=1e-08)

Where any *args or **kwargs are passed directly to the corresponding NumPy function. The defaults of parameters are set to match the default values of the corresponding NumPy function.

Cache

snaptol provides an optional caching feature that can be used during update runs. This is particularly useful for testing suites that call expensive or time-consuming functions.

Results of tests that have failed on the previous run are stored in a cache directory on disk. To enable caching, add the following option to an update run,

pytest --snaptol-update --snaptol-use-cache

When enabled, cached data is used in place of re-running the test.

Snapshot location

By default, snapshot files are stored alongside the corresponding tests in a directory called __snapshots__. You can customise the name of this directory through the --snaptol-dirname option:

$ pytest --snaptol-dirname=testdata

or in a pyproject.toml/pytest.toml file:

[tool.pytest]
addopts = ["--snaptol-dirname=testdata"]

You can also customise the snapshot location for individual tests via the test fixture:

def test_something(snaptolshot):
    snaptolshot.set_snaptolshot_dir("testdata")

    result = compute_something()
    assert snaptolshot == result

For both of these methods, relative paths are relative to the test file. You can also use absolute paths.

Command line options

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate.

License

MIT