pytest_runtime_xfail package
Project description
pytest-runtime-xfail
pytest allows you to mark tests as expected to fail, or xfail, in two ways.
-
@pytest.mark.xfail
. This allows you to mark tests or test parametrizations asxfail
during test collection time.- pytest runs tests marked with
xfail
just like any other test. - If the test fails, it will result in
XFAIL
. - If it passes,
XPASS
. Unless you havexfail_strict=true
or@pytest.mark.xfail(strict=True)
, in which case, passing xfail-marked tests will result inFAIL
.- This is useful to be alerted when an expected failing test starts to pass.
- pytest runs tests marked with
-
pytest.xfail()
. If you need information only known at runtime to decide ifxfail
is appropriate, you can callpytest.xfail()
during a test or fixture.- pytest runs the test as normal UNTIL
pytest.xfail()
is called. - When
pytest.xfail()
is called, the test execution stops and the test results inXFAIL
. - The rest of the test is not run.
- There is no way to get
XPASS
frompytest.xfail()
. xfail_strict
has no effect.
- pytest runs the test as normal UNTIL
There are times where we want a combination of these behaviors.
- We don't know until runtime if we should mark a test as
xfail
. - We want the test run.
- We want the possibility of both
XFAIL
andXPASS
results. - We want to be able to use
xfail_strict=true
to alert us when the test starts passing.
This plugin fills that gap.
Installation
Install with pip:
pip install pytest-runtime-xfail
Usage
Include the fixture, then call it if you want to mark a test as xfail
during runtime.
def test_something(runtime_xfail):
if (runtime_condition):
runtime_xfail()
# ... the rest of your test
Can also be used in a fixture, of course.
@pytest.fixture()
def foo(runtime_xfail):
if (runtime_condition):
runtime_xfail()
# ... the rest of your fixture
def test_something(foo):
# ... the rest of your test
Example found in example/test_xfail.py
"""
Run this with
* pytest -v
* pytest -v -o xfail_strict=true
"""
import pytest
@pytest.mark.xfail()
def test_marker_pass():
'Can be XPASS or FAIL (if xfail_strict)'
assert True
@pytest.mark.xfail()
def test_marker_fail():
'Will always be XFAIL'
assert False # this statememt will be run
def test_old_xfail_pass():
'Will always be XFAIL'
pytest.xfail()
assert True # this statememt will NOT be run
def test_old_xfail_fail():
'Will always be XFAIL'
pytest.xfail()
assert False # this statememt will NOT be run
def test_runtime_xfail_pass(runtime_xfail):
runtime_xfail()
assert True # this statement will be run
def test_runtime_xfail_fail(runtime_xfail):
runtime_xfail()
assert False # this statement will be run
def test_runtime_xfail_reason(runtime_xfail):
runtime_xfail(reason="for demo")
assert False # this statement will be run
Output:
(venv) $ pytest -v test_xfail.py
========================= test session starts ==========================
collected 7 items
test_xfail.py::test_marker_pass XPASS [ 14%]
test_xfail.py::test_marker_fail XFAIL [ 28%]
test_xfail.py::test_old_xfail_pass XFAIL [ 42%]
test_xfail.py::test_old_xfail_fail XFAIL [ 57%]
test_xfail.py::test_runtime_xfail_pass XPASS [ 71%]
test_xfail.py::test_runtime_xfail_fail XFAIL [ 85%]
test_xfail.py::test_runtime_xfail_reason XFAIL (for demo) [100%]
==================== 5 xfailed, 2 xpassed in 0.05s =====================
(venv) $ pytest -v test_xfail.py -o xfail_strict=true
========================= test session starts ==========================
collected 7 items
test_xfail.py::test_marker_pass FAILED [ 14%]
test_xfail.py::test_marker_fail XFAIL [ 28%]
test_xfail.py::test_old_xfail_pass XFAIL [ 42%]
test_xfail.py::test_old_xfail_fail XFAIL [ 57%]
test_xfail.py::test_runtime_xfail_pass FAILED [ 71%]
test_xfail.py::test_runtime_xfail_fail XFAIL [ 85%]
test_xfail.py::test_runtime_xfail_reason XFAIL (for demo) [100%]
===================== 2 failed, 5 xfailed in 0.04s =====================
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for pytest-runtime-xfail-0.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | dc8eaaf4b5112c5b1bf845418149857a77b08f01bcfcb691ccc4aa1439c563a3 |
|
MD5 | 007c3a8d1f2459809f809c1699051474 |
|
BLAKE2b-256 | 6bc1f8317f37f7c6cae578f5c635370083dcceb1a449b0b26252361652ce56a4 |
Close
Hashes for pytest_runtime_xfail-0.0.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 57beb039690755d158d5fd7fd0497667e2b6177fc3302a0430e53b90944bf090 |
|
MD5 | 4769cb5b5e7b89a9580b19a26e5c913d |
|
BLAKE2b-256 | 4033bc18435e4917e245787168cddfefbbf5d9ab703283b0a3415bdcb31f44a1 |