pytest plugin to abort tests after a timeout
This is a plugin which will terminate tests after a certain timeout. When doing so you will get a stack dump of all threads running at the time. This is useful when e.g. running tests under a continuous integration (CI) server.
Note that while by default on POSIX systems py.test will continue to execute the tests after a test has timed out this is not always possible. Often the only sure way to interrupt a hanging test is by terminating the entire process. This in turn will result in no JUnit XML output being created etc. But the plugin will ensure you will have the debugging output on stderr nevertheless. See below for detailed information on the timeout methods and their side-effects.
The pytest-timeout plugin has been tested on python 2.6 or higher, including 3.X.
Install is as simple as e.g.:
pip install pytest-timeout
Now you can run tests using a timeout, in seconds, after which they will be terminated:
Alternatively you can mark individual tests as having a timeout:
@pytest.mark.timeout(300) def test_foo(): pass
By default the plugin will not time out any tests, you must specify a valid timeout for the plugin to interrupt long-running tests. A timeout is always specified as an integer number of seconds and can be defined in a number of ways, from low to high priority:
You can set a global timeout in the py.test configuration file using the timeout option. E.g.:
[pytest] timeout = 300
The PYTEST_TIMEOUT environment variable sets a global timeout overriding a possible value in the configuration file.
The --timeout command line option sets a global timeout overriding both the environment variable and configuration option.
Using the timeout marker on test items you can specify timeouts on a per-item basis:
@pytest.mark.timeout(300) def test_foo(): pass
Setting a timeout to 0 seconds disables the timeout, so if you have a global timeout set you can still disable the timeout by using the mark.
Interrupting tests which hang is not always as simple and can be platform dependent. Furthermore some methods of terminating a test might conflict with the code under test itself. The pytest-timeout plugin tries to pick the most suitable method based on your platform, but occasionally you may need to specify a specific timeout method explicitly.
If a timeout method does not work your safest bet is to use the thread method.
This is the surest and most portable method. It is also the default on systems not supporting the signal method. For each test item the pytest-timeout plugin starts a timer thread which will terminate the whole process after the specified timeout. When a test item finishes this timer thread is cancelled and the test run continues.
The downsides of this method are that there is a relatively large overhead for running each test and that test runs are not completed. This means that other py.test features, like e.g. JUnit XML output or fixture teardown, will not function normally. The second issue might be alleviated by using the --boxed option of the pytest-xdist plugin.
The benefit of this method is that it will always work. Furthermore it will still provide you debugging information by printing the stacks of all the threads in the application to stderr.
If the system supports the SIGALRM signal the signal method will be used by default. This method schedules an alarm when the test item starts and cancels it when it finishes. If the alarm expires during the test the signal handler will dump the stack of any other threads running to stderr and use pytest.fail() to interrupt the test.
The benefit of this method is that the py.test process is not terminated and the test run can complete normally.
The main issue to look out for with this method is that it may interfere with the code under test. If the code under test uses SIGALRM itself things will go wrong and you will have to choose the thread method.
Specifying the Timeout Method
The timeout method can be specified by using the timeout_method option in the py.test configuration file, the --timeout_method command line parameter or the timeout marker. Simply set their value to the string thread or signal to override the default method. On a marker this is done using the method keyword:
@pytest.mark.timeout(method=thread) def test_foo(): pass
The timeout Marker API
The full syntax of the timeout marker is:
You can use either positional or keyword arguments for both the timeout and the method. Neither needs to be present.
- Added the PYTEST_TIMEOUT environment variable as a way of specifying the timeout (closes issue #2).
- More flexible marker argument parsing: you can now specify the method using a positional argument.
- The plugin is now enabled by default. There is no longer a need to specify timeout=0 in the configuration file or on the command line simply so that a marker would work.
- Add a marker to modify the timeout delay using a @pytest.timeout(N) syntax, thanks to Laurant Brack for the initial code.
- Allow the timeout marker to select the timeout method using the method keyword argument.
- Rename the –nosigalrm option to –method=thread to future proof support for eventlet and gevent. Thanks to Ronny Pfannschmidt for the hint.
- Add timeout and timeout_method items to the configuration file so you can enable and configure the plugin using the ini file. Thanks to Holger Krekel and Ronny Pfannschmidt for the hints.
- Tested (and fixed) for python 2.6, 2.7 and 3.2.