pytest-cleanslate: work around or find test state pollution

by Juan Altmayer Pizzorno and Emery Berger at UMass Amherst's PLASMA lab.

About

pytest-cleanslate is a small plugin for the pytest test framework which, as the name implies, helps give each test module a "clean slate" to execute. Plugins such as pytest-forked and pytest-isolate allow you to execute tests in separate processes, working around in-memory test "state pollution" resulting from their execution, but do not protect against pollution caused by top-level code in test modules; this is what pytest-cleanslate remedies.

This module also includes cleanslate-reduce, a tool for finding where the state pollution is occurring. Invoked on a test suite with a failing test, cleanslate-reduce looks for a smaller set of test modules and functions that still lead to the test failure.

How to use

After pip install pytest-cleanslate, simply add --cleanslate to your pytest command line (or configuration options).

Interaction with other plugins

Running with --cleanslate also makes use of pytest-forked, i.e., it is as though you installed that plugin and passed in --forked to execute all tests in separate processes.

It also subverts somewhat pytest's mode of operation in that it postpones collecting test items within test modules (i.e., within Python test files) until the test execution phase. While we have attempted to stay as compatible with other plugins as possible, it is likely to not work in some combinations (such as, for example, pytest-xdist). Feel free to open an issue if you come across a case where it doesn't work.

Requirements

Python 3.8+, Linux or MacOS.