assert-pytest-test-can-fail

Assert that every pytest test is able to fail.

Why Must a Test Be Able to Fail?

A test earns its place by being able to fail. A test whose assertion is satisfied on every path it can reach runs, costs what it costs, and reports green for the whole life of the thing it claims to watch. Coverage counts it. The suite counts it. The only way anybody finds out is by reading it.

One shape produces this reliably. A test sets a flag to False, asks a remote service about a resource, sets the flag to True when the call succeeds, and sets the same flag to True again in the handler for the error that means the resource is absent. Every other error is re-raised. Then it asserts the flag. Every path that reaches the assertion has set it to True, so the assertion is decoration, and a genuinely missing resource is reported as present.

Mutation testing answers this question in general and answers it better, by mutating the code under test and seeing whether a test dies. It cannot answer it for a test with no code under test — one that asserts about a deployed resource through a client library — and it costs a suite run per mutant. This tool asks a narrower question that a parse can settle.

Installation

pip install assert-pytest-test-can-fail

Usage

Command Line

# Scan specific files
assert-pytest-test-can-fail test_example.py

# Scan directories recursively
assert-pytest-test-can-fail test/

# Use glob patterns
assert-pytest-test-can-fail "test/**/test_*.py"

# Exclude patterns
assert-pytest-test-can-fail test/ --exclude "**/conftest.py"

# Verbose output
assert-pytest-test-can-fail test/ --verbose

# Fail fast (exit on first finding)
assert-pytest-test-can-fail test/ --fail-fast

# Warn only (always exit 0)
assert-pytest-test-can-fail test/ --warn-only

GitHub Actions

- uses: 10U-Labs/assert-pytest-test-can-fail@latest
  with:
    files: "test/"
    verbose: "true"

As a Python Module

python -m assert_pytest_test_can_fail test/

Output Format

Default output shows one finding per line:

path/to/test_file.py:23:test_bucket_exists: every path to the assert sets the flag, so the test cannot fail

Format: file_path:line_number:function_name, followed by the reason. The line number is the assert statement.

Exit Codes

• 0: No findings (or --warn-only specified)
• 1: Findings detected
• 2: Error (missing files, syntax errors, etc.)

What Is Reported

A function whose name begins with test is reported when all of these are true:

• it contains a try statement with at least one except handler
• the statement that ends the try body — or the else clause, when there is one — assigns True to a bare name
• every handler on that try is closed: it ends in raise, or it ends assigning True to that same name, or it ends in an if/else whose two branches each do one of those
• an assert on that bare name follows the whole try statement

The last one matters. An assertion sitting inside the try, or above it, is asking a different question, and is left alone.

Each reported function is reported once, at the first assertion that qualifies.

What Is Never Reported

The shape alone is not the defect, and a rule that says otherwise is a rule nobody keeps. A permission check written this way is correct: it asks whether the credentials may inspect a resource, a 404 answers yes, and any error code the handler does not name leaves the flag False, so the assertion fails.

def test_can_describe_bucket():
    has_permission = False
    try:
        client.head_bucket(Bucket=name)
        has_permission = True
    except ClientError as error:
        code = error.response["Error"]["Code"]
        if code == "403":
            pytest.fail("No permission to inspect the bucket")
        if code == "404":
            has_permission = True
    assert has_permission

That handler ends in an if with no else, so it can complete with the flag still False. It is not reported. Neither is a handler that ends in pass, nor a pytest.fail that leaves other codes falling through — only raise and an assignment on every branch close a handler.

Also never reported:

• a try with no except handler at all, such as try/finally
• a success path that computes the flag rather than asserting it, such as exists = client.bucket_exists(name)
• a chained assignment, exists = asked = True, or an attribute target, record.exists = True
• an assertion on anything but a bare name, such as assert count == 1
• a try inside a nested function, a lambda or a class the test defines: the tool does not descend into a scope the test opens
• a function whose name does not begin with test

Only files pytest would collect are scanned: test_*.py and *_test.py.

Options

License

Apache 2.0 - See LICENSE.txt