plain.testing

Write and run tests: a client for your app, plain functions with bare asserts, and nothing a test depends on that you can't see in its file.

Overview
Making requests
- GET requests
- POST requests
- Other HTTP methods
- Following redirects
- Custom headers
- Schemes and hosts
Inspecting responses
- Streaming responses
Cookies, logins and sessions
Expected exceptions
Test metadata
- Skipping from inside a test
Overriding context
Capturing what happened
- Spans
- Metrics
- Log records
- Annotating helpers, and writing a capture of your own
WebSockets
Building a request
Running tests
- Selecting tests
- Exit codes
- Environment
Where tests live
- Shared helpers
Reading the output
- Where the time went
- Failures
- Large values
- What the test wrote
- Skipped tests
- Collection errors
- As JSON
Assertions
What packages do for every test
Project lifecycle
Test lifecycles
Testing code outside the app
How it works
Design rules
FAQs
Installation

Overview

A test is a function that makes an assertion. Client makes a request to your app without a server, and gives you back the response that would have been sent.

# tests/test_homepage.py
from plain.testing import Client


def test_homepage():
    response = Client().get("/")
    assert response.status_code == 200

Run it with plain test:

$ plain test
Collected 1 test
.

1 passed in 0.04s

Everything a test uses is an import, a call, or a with block in its own file. The one exception is protection the runner gives every test without being asked, like wrapping it in a database transaction. If you can read the test file, you know what happens.

This page has two halves. The first is how to write a test: the client, raises, the decorators, and the with helpers, all imported from plain.testing. The second, from Running tests on, is the plain test command: how tests are found, what the output means, and what a failed assert shows.

Making requests

The client speaks the same vocabulary as the rest of Plain: form_data= arrives as request.form_data, json_data= as request.json_data, files= as request.files, and query_params= as request.query_params. Everything after the path is keyword-only.

GET requests

response = client.get("/search/", query_params={"q": "hello"})

POST requests

Send a form:

response = client.post(
    "/submit/", form_data={"name": "Alice", "email": "alice@example.com"}
)

A form is urlencoded, and becomes multipart only when files= is given — the same choice a browser makes, so the view under test sees the content type it will see in production.

A file goes in files=, never in form_data=. A file object or bytes in form_data raises TypeError and names the key: it would have been sent as a text field.

Send JSON — the value is serialized for you:

response = client.post("/api/users/", json_data={"name": "Alice"})

Send file uploads alongside form fields:

response = client.post(
    "/upload/", form_data={"title": "Report"}, files={"file": file_obj}
)

Send a raw body with an explicit content type:

response = client.post("/webhooks/", body=payload_bytes, content_type="application/xml")

Other HTTP methods

The client has a method for get, head, options, post, put, patch, and delete. The body-carrying methods take the same arguments as post.

response = client.put("/api/users/1/", json_data={"name": "Bob"})
response = client.patch("/api/users/1/", json_data={"name": "Bob"})
response = client.delete("/api/users/1/")

For any other method, request() takes the method by name and the same keywords:

response = client.request("PROPFIND", "/files/")
assert response.status_code == 405

Following redirects

Redirects aren't followed unless you ask — where a request lands is usually the thing worth asserting.

response = client.post("/signup/", form_data={"email": "a@example.com"})
assert response.redirect_to == "/welcome/"

Set follow_redirects=True to follow the chain to its destination:

response = client.get("/old-url/", follow_redirects=True)
assert response.status_code == 200  # Final destination
assert response.redirect_chain == [("/new-url/", 302)]

Custom headers

response = client.get("/api/", headers={"Authorization": "Bearer token123"})

You can also set default headers when creating the client.

client = Client(headers={"Accept-Language": "en-US"})

Schemes and hosts

A path makes a request to https://testserver. When the scheme, host or port is what you're testing, pass a full URL:

response = client.get("http://testserver/")  # a request that didn't come over HTTPS
assert response.redirect_to == "https://testserver/"

response = client.get("https://shop.example.com:8443/cart/")

A URL without a port goes to its scheme's port. A followed redirect resolves its Location the way a browser does, against the request that was redirected.

Inspecting responses

Responses are data, not assertion methods — bare assert is the assertion API. A ClientResponse has these names, and no others:

Name	What it is
`status_code`	The status that went out
`headers`	The response headers
`cookies`	The cookies this response set
`body`	The bytes the response sent
`text`	The body decoded as a string
`json_data`	The body parsed as JSON (requires a JSON content type)
`redirect_to`	The redirect target on a 3xx response, `None` otherwise
`redirect_chain`	The `(url, status_code)` of each redirect that was followed; empty if none were
`request`	The request that produced this response, after middleware ran
`exception`	The exception behind a 5xx, when `raise_exceptions=False` kept it a response
`returned_response`	The `Response` object the app returned

The route that handled the request is response.request.resolver_match, which is None when a middleware answered it or nothing matched. Whether the body was streamed is response.returned_response.streaming.

response = client.get("/api/users/")
assert response.json_data["users"][0]["name"] == "Alice"

Everything but the last describes what was sent. returned_response is the object itself, for an assertion about its type or about an attribute only that type has:

from plain.http import FileResponse

response = client.get("/report.pdf")
assert isinstance(response.returned_response, FileResponse)

Its own content and status_code can differ from what went out — after a HEAD, a 204, or a streaming body that failed — which is why the client's response doesn't pass them through. Reading any other name raises an AttributeError that lists the ones above.

By default, the client re-raises unhandled view exceptions so failures point at the real error. Pass Client(raise_exceptions=False) to get the 500 response instead.

Streaming responses

The client reads a streaming body to the end before it returns, the way a server sends it, so body (and text, json_data) hold what a StreamingResponse, FileResponse, or AsyncStreamingResponse sent:

response = client.get("/export.csv")
assert response.body.startswith(b"id,name")

Because the whole body is read, a stream that never ends (an endless event feed) makes the request never return — test those views' pieces directly instead. HEAD requests and bodiless statuses (204, 304) never read the body.

If a body raises partway through, the error is re-raised from the request unless the client was created with raise_exceptions=False, in which case response.exception holds it and body has what came before. A body that fails before producing anything is answered with a 500, as a server would, and status_code says so.

Cookies, logins and sessions

A client keeps the cookies its responses set and sends them with every request after, so a flow that logs in through a form stays logged in. client.cookies is that jar, a SimpleCookie:

client.cookies["theme"] = "dark"
response = client.get("/")  # sent with Cookie: theme=dark

client.cookies.clear()

Logging in and reading the session are built on it, and ship with the packages that own them:

plain.auth: login_client(client, user) and logout_client(client)
plain.sessions: get_client_session(client)

Expected exceptions

Use raises to assert that a block raises:

from plain.testing import raises


def test_invalid_email():
    with raises(ValidationError) as caught:
        validate_email("nope")
    assert "email" in str(caught.exception)

Pass match= to also require the message to match a regex.

caught.exception is typed as the exception class you asked for, so its own attributes are reachable without a cast (caught.exception.messages on a ValidationError). It's readable only after the block exits — inside the block nothing has been caught yet, and reading it says so rather than handing back a None.

Test metadata

Decorators declare static facts about a test — they never inject runtime values:

from plain.testing import cases, skip, tag


@cases(
    ("a@example.com", True),
    ("nope", False),
)
def test_email_validation(email, valid):
    assert is_valid_email(email) is valid


@skip("Waiting on the new billing API")
def test_invoice_totals(): ...


@tag("slow")
def test_big_import(): ...

cases — each tuple becomes its own test run
skip — always skipped, reason shown in the report
tag — labels for selection (plain test --tag slow)

@cases is the only way a test takes parameters. Nothing is passed to a test by name, so a test whose parameters no case fills is rejected when the file is collected, with a message naming the test and the parameters.

A case is reported by its values, joined with -: test_email_validation[a@example.com-True] and test_email_validation[nope-False]. That's what a failure is headed with, and what its re-run command names.

Cases are named that way when every value of every case is a string, a number, a boolean, None or an enum member (by its name), and no two cases come out the same. When one can't be, because a value is a list or an object, a string is empty or has a line break in it, or the name would pass 60 characters, all of that test's cases are numbered instead: test_email_validation[0], [1]. One test's cases are all named or all numbered.

Wrap a case in case to give it a name of its own:

from plain.testing import case, cases


@cases(
    case("a@example.com", True, id="plain address"),
    case("nope", False, id="no at sign"),
)
def test_email_validation(email, valid):
    assert is_valid_email(email) is valid

That reports as test_email_validation[no at sign]. The id sits on the case it names, so adding or reordering cases can't shift the names onto the wrong values. The ids of one test's cases are all different, however each got its id: a name that is what another case is called or numbered raises.

A test takes one @cases. A second one raises instead of replacing the first. Each case is one flat tuple: the test's values, in the order of its parameters. For every combination of two lists, build the cases from both:

from plain.testing import cases

SOURCES = ["kwargs", "object"]
OPERATIONS = [("insert", 1), ("update", 0)]


@cases(
    *[
        (source, operation, rows_added)
        for source in SOURCES
        for operation, rows_added in OPERATIONS
    ]
)
def test_write_paths(source, operation, rows_added): ...

Each for names what one entry of its list holds: a single value, or the values of a tuple. itertools.product is right only when every list holds single values. Given a list of tuples it nests them, and each case comes out as ("kwargs", ("insert", 1)).

Skipping from inside a test

@skip is for a test that never runs. When only the running test can tell, call skip_test in its body:

from plain.testing import cases, skip_test


@cases("text", "encrypted")
def test_field_supports_contains(kind):
    if kind == "encrypted":
        skip_test("Encrypted fields have no substring match")
    assert "contains" in lookups_for(kind)

The test stops there and is reported as skipped, with the reason. Everything it entered still exits: with blocks unwind and the database transaction is rolled back. An except Exception: around the call doesn't swallow the skip.

Overriding context

Runtime state changes are context managers, so their scope is visible as indentation:

from plain.testing import override_settings, patch


def test_debug_error_page():
    with override_settings(DEBUG=True):
        response = Client().get("/broken/")


def test_external_call():
    with patch(billing, "charge_card", lambda **kwargs: "ch_123"):
        checkout(cart)

override_settings — set Plain settings for the block, restored on exit
patch — replace an attribute (or a mapping key, e.g. os.environ) for the block

patch takes the object and the attribute name, not a dotted string. A target written as a dotted string ("app.billing.charge_card") is still unittest.mock.patch. The two work differently and can sit in one file, so import that one as from unittest import mock and write mock.patch(...).

On exit the target holds what it held before, which isn't always what reading the attribute finds:

A class or an instance that only inherited the attribute inherits it again. Nothing is left behind on it.
A class gets back the staticmethod or classmethod it held, not the function that reading it returns.
A property, a slot, a setting on plain.runtime.settings and a key of a mapping get back the value they had.

Capturing what happened

capture_spans, capture_metrics and capture_logs each record one kind of thing while a block runs, and all three hand back the same shape: a read-only sequence of what was captured, in the order it happened.

from plain.testing import Client, capture_logs, capture_spans


def test_homepage():
    with capture_spans() as spans, capture_logs() as logs:
        Client().get("/")

    assert "GET /" in [span.name for span in spans]
    assert logs == []

It's a sequence, so len(), indexing, slicing, iteration, in and truthiness work, and there's no method to call to get at the items. It compares equal to a list or tuple holding the same items, so assert logs == [] means what it says.

Read a capture after its block. A capture is complete when the block ends: that's when a span that was still open has ended, and when metrics are collected. Reading it inside the block raises, so a test can't pass or fail on part of what happened:

RuntimeError: capture_spans() is still capturing — read what it captured after the `with capture_spans()` block ends, not inside it.

A block that raises still finishes its capture, so what was captured up to that point is there to read. Captures can be nested, and one opened inside another doesn't take anything from the outer one.

Spans

from opentelemetry.trace import SpanKind

from plain.testing import capture_spans


def test_homepage_span():
    with capture_spans() as spans:
        Client().get("/")

    [server_span] = spans.filter(kind=SpanKind.SERVER)
    assert server_span.attributes is not None
    assert server_span.attributes["http.route"] == "/"

capture_spans captures the spans that end during the block. Each is OpenTelemetry's own ReadableSpan, so name, kind, attributes, status, events, parent and context are all there.

spans.filter(name=..., kind=...) returns the spans with that name, of that kind, or both, as a list. Unpack it when there should be exactly one ([span] = spans.filter(name="claim job")), or index it when there may be several.

Metrics

from plain.testing import capture_metrics


def test_request_duration_is_recorded():
    with capture_metrics() as metrics:
        Client().get("/")

    [point] = metrics.histogram_points(
        "http.server.request.duration",
        attributes={"http.route": "/"},
    )
    assert point.count == 1

capture_metrics captures the metrics recorded during the block. Each item is OpenTelemetry's own Metric, but what a test usually wants are a metric's data points, and those come in two kinds:

metrics.number_points(name): the points of a counter, an up-down counter or a gauge. Each has a value.
metrics.histogram_points(name): the points of a histogram. Each has a count, a sum, a min and a max.

Both take attributes={...} to keep only the points that carry all of those attributes, and both return an empty list for a metric nothing was recorded for. Asking for the wrong kind raises and names the right one.

Metrics are collected when the block ends, and that's also when an observable instrument is asked for its value. If a gauge reports the size of a pool, the pool has to still be there, so open the capture inside the block that keeps it alive:

def test_pool_reports_its_connections():
    connection = pool.acquire()
    try:
        with capture_metrics() as metrics:
            pass
    finally:
        pool.release(connection)

    assert metrics.number_points("db.client.connection.count")

Log records

from plain.testing import Client, capture_logs


def test_server_error_is_logged():
    with capture_logs() as logs:
        Client(raise_exceptions=False).get("/broken/")

    assert "Server error" in logs.messages
    assert logs[0].path == "/broken/"

capture_logs captures the records logged during the block. Each is a logging.LogRecord, and logs.messages is the formatted message of every one. Structured context passed as context={...} lands on the record as ordinary attributes, so logs[0].path reads it back.

With no arguments it captures the whole plain and app trees; name loggers to narrow it (capture_logs("plain.jobs")). Plain's loggers don't propagate to the root logger, so attaching a handler there would see nothing. This attaches to the named loggers directly, lowers their level for the block, and restores everything on exit.

logs.span_context_for(message) returns the OpenTelemetry span context that was current when that record was logged. That's the check behind "this exception log landed inside its error span": a record logged with no span current exports with empty trace and span ids, and the one failure gets reported twice downstream (the span's exception event plus an orphaned error log).

from plain.testing import capture_logs, capture_spans


def test_claim_failure_log_is_correlated():
    with capture_spans() as spans, capture_logs("plain.jobs") as logs:
        run_the_failing_claim()

    [span] = spans.filter(name="claim job")
    assert span.context is not None
    assert logs.span_context_for("Failed to claim job").trace_id == (
        span.context.trace_id
    )

Annotating helpers, and writing a capture of your own

What the three yield is importable for annotating your own helpers: CapturedSpans, CapturedMetrics and CapturedLogs.

from opentelemetry.trace import SpanKind

from plain.testing import CapturedSpans


def route_of(spans: CapturedSpans) -> str:
    [server_span] = spans.filter(kind=SpanKind.SERVER)
    assert server_span.attributes is not None
    return str(server_span.attributes["http.route"])

All three are a Captured, and a package that ships its own capture helper builds on the same class so it reads the same way. capture_queries in plain.postgres.testing is one.

A helper makes a Captured and calls its finish(items) when the block ends. When every capture of a kind reads from one list that grows as things happen, a CaptureSource does that for you, and is what makes the captures nest:

from contextlib import contextmanager

from plain.testing import Captured, CaptureSource

from .registry import registry

registered = CaptureSource(read=registry.log_entries, clear=registry.clear_log)


@contextmanager
def capture_registrations():
    captured = Captured(helper="capture_registrations")
    with registered.capturing_into(captured):
        yield captured

Each capture gets what was added to the list during its own block. The list is emptied when the outermost capture ends, so nothing captured is kept for the rest of the run.

WebSockets

Client.websocket() runs the handshake through the same pipeline as any request (cookies and auth included), then drives the view's websocket() in-process:

from plain.testing import Client, WebSocketRejected, raises


def test_echo():
    with Client().websocket("/live/", subprotocols=("binary",)) as ws:
        assert ws.subprotocol == "binary"
        ws.send("hello")
        assert ws.receive() == "echo: hello"


def test_login_required():
    with raises(WebSocketRejected) as caught:
        Client().websocket("/live/")
    assert caught.exception.response.status_code == 403

It takes query_params= and headers= like get(), plus subprotocols= and timeout=.

ws.send(message) sends one message to the view; ws.receive() returns the next one it sends.
ws.close(code=1000, reason="") closes from the client side and waits for the view to finish. Leaving the with block closes it if the test didn't.
ws.subprotocol is the negotiated subprotocol. ws.request is the handshake request and ws.response is the 101 that answered it, for asserting on its headers and cookies.
Every call has a timeout (5 seconds by default, receive(timeout=...) per call) and raises TimeoutError when it elapses.
An exception raised by the view surfaces from receive() and again when the with block exits; a view that closes the socket makes receive() raise WebSocketClosed (from plain.http) with its code and reason.
A handshake that doesn't produce a socket — a 403, a redirect — raises WebSocketRejected. Its .response is the same kind of response client.get() returns. A handshake the app raised from raises that exception, as client.get() would, unless the client was created with raise_exceptions=False.

The view runs on the test's own thread, inside a copy of the test's context, so the test database transaction is visible to it. Because the connection steps its own event loop, Client.websocket() is for synchronous tests, not async def ones.

Building a request

build_request() builds a Request without sending it, for calling a view or a middleware yourself. It takes the method, the path, and the keywords the client's methods take, without follow_redirects=.

from plain.testing import build_request

request = build_request("GET", "/hello/", query_params={"name": "Alice"})
request = build_request("POST", "/hello/", json_data={"name": "Alice"})
request = build_request("PROPFIND", "/files/")

view = HelloView(request=request)
response = view.get_response()

The body is encoded the way the client encodes it, and a path goes to https://testserver unless you pass a full URL. The client's cookies and default headers belong to the client, so a built request has only the headers= you give it.

It returns an ordinary Request. When the body is already bytes and you don't need it encoded, you can construct one directly.

Running tests

plain test                                      # everything under the current directory
plain test tests/test_views.py                  # one file
plain test tests/checkout                       # one directory
plain test tests/test_views.py::test_homepage   # one test
plain test tests/test_views.py:42               # the test line 42 is in
plain test --match signup                       # tests whose id contains "signup"
plain test --tag slow                           # only tests tagged "slow"
plain test --exclude-tag slow                   # everything but
plain test --fail-fast                          # stop at the first failure
plain test --verbose                            # one line per test, with its duration
plain test --full-values                        # print every value in a failure whole
plain test --show-output                        # let what tests print through as they write it
plain test --json                               # one JSON document, when the run is over

Flag	What it does
`--match TEXT`	Keep the tests whose id contains `TEXT`
`--tag NAME`	Keep the tests with this tag. Repeat it to allow more
`--exclude-tag NAME`	Drop the tests with this tag. Repeat it to drop more
`--fail-fast`	Stop at the first failure
`--verbose`	Print one line per test
`--full-values`	Print every value and all the output in a failure
`--show-output`	Let what tests print and log through as it's written
`--json`	Print the run as one JSON document
`--list-passed`	With `--json`, list the tests that passed too

Each flag has one name. plain test --help prints the same list, and the forms a target can take.

Tests run in the same order every time. Within a file, that's the order they're written in.

Selecting tests

A target is a path, optionally followed by :: and a name, or by : and a line. You can pass several.

Target	Runs
`tests/checkout`	Every test file under that directory
`tests/test_views.py`	Every test in that file
`tests/test_views.py::test_homepage`	That test, and every case of it
`'tests/test_email.py::test_valid[empty]'`	That one case
`tests/test_views.py:42`	The test line 42 is in

A line is any line of the test, from its first decorator to the last line of its body, and with @cases every case of the test runs. A traceback gives a file and a line, which is enough to run the test again without knowing its name. A line that's in no test is an error, which names the nearest tests above and below it:

No test at tests/test_views.py:12: line 12 is in no test. It is between test_homepage (lines 5 to 9) and test_about (lines 14 to 20).

Paths are relative to the directory you run from. Quote a target that names a case, since the shell reads [ and spaces itself. The re-run command a failure prints is already quoted.

--match keeps a test when TEXT is somewhere in its id. It's the text as written, not a pattern or an expression, and it's matched against the whole id, which starts with the file's path. So --match checkout keeps every test in tests/checkout/ as well as any test with "checkout" in its name.

Targets, --match and the tag flags combine: a test runs when it's inside a target and passes every filter.

Exit codes

Code	Meaning
`0`	Every test that ran passed. Skipped tests and warnings don't change this
`1`	A test failed, or a file couldn't be collected
`2`	The command can't be used as given: a target doesn't exist, flags don't go together, or the project lifecycle is wrong
`3`	Setting up failed, so no test was run: the app couldn't be set up, or a lifecycle couldn't, such as the database's
`4`	No tests matched
`130`	The run was stopped with Ctrl-C. What had run by then is reported

1 says the tests need fixing. 3 says what they run on does.

Environment

plain test sets PLAIN_ENV=test unless you've set it yourself. It reads no .env files of its own. plain.dev loads them, as it does for every command, and under PLAIN_ENV=test that means .env.test and .env, with .env.local left out so personal credentials don't reach the suite. Without plain.dev installed, no .env file is loaded.

It also sets PLAIN_TEST_RUNNING=1. Plain's own command output reads that to leave out color codes, so a test asserting on the output of a command sees plain text.

Where tests live

Put tests in tests/, beside app/. The runner looks for:

Files named test_*.py, in any subdirectory
Functions named test_*
async def test_*, which the runner awaits for you

# tests/test_cart.py
from plain.testing import Client


def test_empty_cart():
    assert Client().get("/cart/").status_code == 200


def test_checkout_requires_login():
    assert Client().get("/checkout/").redirect_to == "/login/"


async def test_price_lookup():
    assert await fetch_price("A-1") == 1200

A test is a function, and a file is the group. To group tests, put them in a file of their own, and to share what they set up, write a function they call.

Nothing that looks like a test is left out without a word. These are collection errors, each saying what to write instead:

A test with a yield in it. Calling it would make a generator and run none of its body
A class with test_* methods in it, whatever it's named and whatever it inherits from. None of them would run
A test_* function imported from another module. A test is run by the file that defines it

An async def test uses the client the way any test does. client.get() and the rest are ordinary calls, not awaited, whether the view they reach is sync or async, and the test's event loop waits while the request runs. The exception is client.websocket(), which is for synchronous tests.

With no target, plain test searches the directory you ran it from. It doesn't look in directories whose name starts with a dot, in node_modules, or in __pycache__.

Tests aren't kept in app/. It's the application: the package your code imports as app, and what gets deployed. A test file in it would ship with it, and would be part of that package, where a helper module can only be imported through the app's name. So there's one place for tests, and it's beside the application.

A test_*.py file in app/ that has tests in it isn't run, and isn't passed over without a word either. It's a collection error that says which files they are and where they belong, whether the search came to them or you named one as a target. A module of your application that is named test_*.py and defines no tests, such as a test_connection.py that checks a connection, is left alone.

Shared helpers

Shared setup is ordinary Python. Write a module and import it:

# tests/helpers.py
from app.users.models import User


def create_user(*, email="test@example.com"):
    return User.query.create(email=email)

# tests/accounts/test_profile.py
from helpers import create_user
from plain.testing import Client


def test_profile_shows_the_email():
    user = create_user(email="ada@example.com")
    response = Client().get(f"/users/{user.id}/")
    assert "ada@example.com" in response.text

tests/ is on the import path, so a helper module is imported by its path from tests/. tests/helpers.py is helpers. That's true for a test file in any subdirectory, whichever directory you run from, and whatever target you pass.

A helper that belongs to one directory of tests can live in it. It's imported the same way, by its path from tests/, by the test file beside it and by any other:

# tests/billing/refund_helpers.py is `billing.refund_helpers`
from billing.refund_helpers import create_refund

The directory needs no __init__.py.

It's the only way to import one. These are collection errors that say what to write instead:

from tests.helpers import ..., through the name of the tests directory
from .refund_helpers import ..., a relative import
from refund_helpers import ..., by the end of its path, when the module is further down than tests/

A module imported under two names is loaded twice, and the two copies don't share their state.

tests/ comes first on the import path, so a helper module named like an installed package takes its place. Give a helper module a name nothing else has.

tests/ is not a package. Nothing imports the directory, so an __init__.py in it is never run.

Nothing is passed to a test by name, and no file is read for what tests might need. The runner reads files named test_*.py and your tests/lifecycle.py. Every other file is there to be imported by one that needs it.

Only files named test_*.py have their assertions rewritten. An assert in a helper module fails as a bare AssertionError, without the values inside it.

Reading the output

A run that passes prints how many tests it found and what came of them, and nothing for each test:

Collected 41 tests

41 passed in 0.62s

The last line counts what happened:

41 passed, 1 failed, 2 skipped, 1 collection error in 0.62s

When stdout is a terminal, one line says how far the run has got while it runs (12 of 41, 1 failed). It's written over itself and erased before the report, so nothing of it is left, and it's never written to a pipe or a file. With --verbose each test gets a line of its own as it finishes, with its outcome as --json spells it:

passed  tests/test_cart.py::test_empty_cart (0.012s)
skipped tests/test_cart.py::test_refund (Waiting on the new billing API)

What a test prints or logs is held while it runs: a test that fails has it printed with its failure, and a test that passes has it thrown away. Warnings are counted and listed once each, and what was written outside any test is printed at the end.

Where the time went

A run is mostly not tests. Before the first one the app is set up, its packages are imported, the test files are read and the test database is made, and after the last one it's dropped. When all of that came to more than a second, the report says where it went, and with --verbose it always does:

1 passed in 0.52s

where the 1.27s went
  python startup           0.04s  cpu time, before Plain was imported
  command                  0.05s
  runtime setup            0.77s
    hook dev-setup         0.30s
    other hooks (6)        0.01s
    settings               0.02s
    import app.agents      0.40s
    other imports (27)     0.03s
    ready()                0.01s
  helper modules           0.00s
  lifecycles loaded        0.01s
  collection               0.02s
    rewrote 1 file         0.01s
  lifecycle setup          0.15s
    EmailTestLifecycle     0.00s
    PostgresTestLifecycle  0.15s
      cloned template      0.04s
  tests                    0.02s
  lifecycle teardown       0.05s
    PostgresTestLifecycle  0.05s
    EmailTestLifecycle     0.00s
  report                   0.00s
  unaccounted              0.00s

It's a timeline, in the order things happened. The clock starts when Plain is first imported. What came before that, the interpreter starting and Python's own imports, a process can't time, so the first line is the CPU time it had used by then, which for startup work is close to the same thing. unaccounted is what the marks between phases don't cover. Printing the report comes after the last mark and isn't in it.

The parts an app can do something about are under runtime setup. A plain.setup hook (plain.dev's reads the .env files and finds the database) or a package's import that took 50 ms or more gets a line of its own. An import that slow is usually a module importing a client library at the top, which every command then pays for. The rest share the other imports line. lifecycle setup is each package's lifecycle by name, and under it what the lifecycle says its setup did: plain.postgres says whether it built the template the run's database is cloned from, which the first run of a schema does, or cloned it, which every run after does.

Under a second outside the tests, a passing run is still two lines. The JSON document has the phases always, as phases.

Failures

Every failure is listed after the run. It says where the test failed, what the values inside the assert were, what else the test had in hand, and how to run it again:

failed tests/test_signup.py::test_signup_redirects

  Traceback (most recent call last):
    File "/project/tests/test_signup.py", line 10, in test_signup_redirects
      assert response.status_code == 302
  AssertionError

  assert response.status_code == 302
    response.status_code = 404
      response = <ClientResponse status_code=404 of <Response status_code=404, "text/plain; charset=utf-8">>

  locals:
    client = <Client cookies=[]>
    email = 'a@example.com'

Re-run: plain test tests/test_signup.py::test_signup_redirects

The traceback starts at your test. The runner's own frames are left out.

Under the assert is each part of its expression and what it was, from the outside in: response.status_code was 404, and the response it was read from is indented under it. What's written out in the expression (302) isn't repeated. See Assertions.

A large value is printed once. When the failure has it again under another name, the second says <the same as rows>.

What your test file defines is called what the file calls it. A FakeGateway class in tests/test_billing.py prints as <FakeGateway object at 0x...>, not under the name the runner loaded the file by.

locals: is every name the test function had bound when it failed, in the order they were bound, without the ones the assert already printed. It's there for every failure, not only a failed assert, and it's always the test function's own: when the error was raised in something the test called, the traceback shows where and the locals show what the test called it with. A name bound to a module, a class or a function is left out.

The command is quoted wherever a shell would read the id for itself, so you can paste it as it is:

Re-run: plain test 'tests/test_price.py::test_total[annual plan]'

Large values

Two values that were expected to be equal, and are too large to read side by side, are printed as what differs between them:

  assert profile == expected
    profile = <dict with 7 keys>
    expected = <dict with 6 keys>

  diff:
    --- profile
    +++ expected
    @@ -4,4 +4,3 @@
      'plan': 'annual',
      'renews': '2027-01-01',
    - 'seats': 3,
    - 'trial': False}
    + 'seats': 5}

- lines are the left side's and + lines are the right side's. What is compared depends on what the values are:

Values	Compared by
Text of more than one line	Line. By each line's `repr` when they differ in a trailing space or line ending
Two long strings on one line	The first character that differs, with what is around it on each side
Dicts	Key, in sorted order
Lists, tuples and sets	Item
Dataclasses	Field
Anything a package describes	Line of its description. A model instance is one field to a line

Values that each fit on a line of 80 characters are printed whole and aren't diffed.

A single value is printed up to 2,000 characters and a diff up to 60 lines. When either is cut short, the last line says by how much:

      ... 143,000 more characters (--full-values prints them)

Run the test again with --full-values to print everything.

A value whose repr raises doesn't stop the report. It's printed as <Order: its repr raised ValueError: no total>, and everything else is printed as usual.

Secrets

A report is read by whoever reads the run: a terminal, a CI log, an agent. It leaves out what is known to be a secret, and says that it did:

  assert "PAYMENTS_KEY" not in os.environ
    os.environ = environ(3 names, values withheld: HOME, PATH, PAYMENTS_KEY)

  locals:
    for_a_subprocess = {'DEBUG': '1', <3 names from the environment>: <values withheld: HOME, PATH, PAYMENTS_KEY>}
    user = User(id=1, email='a@example.com', password=<withheld>)

Value	Printed as
`os.environ`	Its names
A dict with items taken from the environment	Its own items, and the names of the ones it took
The settings	`<Settings "app.settings">`, with no values
A model instance	Its fields, with an encrypted field or a password as `<withheld>`. See plain.postgres

That holds wherever the value is: on its own, or inside a list, a tuple, a dict or a set. It holds in the JSON document and with --full-values.

A secret that is a string like any other by the time the test has it is printed like any other: a local holding settings.SECRET_KEY, or os.environ["PAYMENTS_KEY"] written in an assert.

What the test wrote

What a failed test wrote to stdout and stderr is printed with its failure, each under its own name and in the order it was written:

failed tests/test_orders.py::test_order_total

  Traceback (most recent call last):
    File "/project/tests/test_orders.py", line 16, in test_order_total
      assert order["total"] == 42
  AssertionError

  assert order["total"] == 42
    order["total"] = 0
      order = {'total': 0}

  stdout:
    pricing 2 items

  stderr:
    No price for kettle

Re-run: plain test tests/test_orders.py::test_order_total

It's held however it was written: print(), a log handler, a subprocess the test started, a C extension. What a test reads for itself stays the test's, so contextlib.redirect_stdout, capture_logs, a CliRunner and a subprocess with its own pipes all work as they do anywhere.

A test's output starts when its lifecycles enter and ends when they've exited, so what around_test() writes on the way in and out belongs to the test. What's written before the first test, while the app is set up and the test database is made, belongs to no test. It's printed on its own.

The last 10,000 characters of each stream are kept, and the failure says how much came before them. --full-values prints all of it:

  stdout:
    ... 40,015 characters before this (--full-values prints them)
    line 4000
    line 4001

--show-output holds nothing: everything is written where it would have been, as it happens. Use it to watch a test that hangs, or to see what a passing test prints.

A run stopped with Ctrl-C reports what it had: the failures so far, and what the test that was running had written.

interrupted tests/test_sync.py::test_every_page

  stdout:
    fetching page 1
    fetching page 2

Interrupted: 12 passed, 30 not run in 4.18s

Warnings

A warning a test raises is counted, whether the test passes or fails. Each distinct warning is listed once, with how many times it was raised and where it was raised first:

warnings

  DeprecationWarning: old_price() is going away
    raised 3 times, first at tests/test_orders.py:9 in tests/test_orders.py::test_order_total

41 passed, 1 warning in 0.62s

A warning is the same warning when it's the same kind saying the same thing, wherever it's raised from. A deprecated function called from two hundred places is one line.

DeprecationWarning and PendingDeprecationWarning are shown, which Python leaves out unless the script being run raises them. Pass -W to Python, or set PYTHONWARNINGS, and the filters are as you gave them. A test that turns warnings into errors (warnings.simplefilter("error")) still does, and a test that catches its own still catches them.

Written outside any test

What's written while the app is set up, while the lifecycles are set up before the first test, and while they're taken down after the last, belongs to no test. It's printed after the run, whether the run passed or not:

written outside any test

  stderr:
    Managed Postgres unavailable: is Docker running?

41 passed in 0.62s

Most runs write nothing there, and print nothing for it. What a test file writes while it's loaded isn't here: it's thrown away, unless the file couldn't be collected.

A run that couldn't start

When the app can't be set up, or a lifecycle's setup_worker() raises or exits, no test is run. The run says what failed and what had been written by then, and exits 3:

PostgresTestLifecycle.setup_worker() exited, with 2.

  Traceback (most recent call last):
    ...
  SystemExit: 2

  stderr:
    Got an error creating the test database: connection refused

No test was run.

This goes to stderr, and stdout has only Collected 41 tests. The lifecycles that had been set up before the one that failed are taken down again.

Skipped tests

A skipped test is listed with its reason, whether it came from @skip or from skip_test, and it's counted in the summary:

skipped tests/test_uploads.py::test_upload_to_bucket (No bucket reachable from this machine)

Collection errors

A file that can't be turned into tests is a collection error. The other files still run, and the run exits 1.

collection error tests/test_signup.py

  These tests can't be run as written:

    test_signup(user, plan) takes parameters, and nothing passes them in.
    test_welcome_email(user) takes parameters, and nothing passes them in.

    user  2 tests
    plan  1 test

  Nothing is passed to a test by name. A test gets what it needs in its
  body, by calling a helper or entering a `with` block, and takes values
  only from @cases(...).

Each parameter is listed with how many of the file's tests take it. A file with more than three such tests says how many, not which.

Every problem in the file is reported at once, so a file with four things wrong is fixed in one go:

collection error tests/test_orders.py

  These imports can't be used in a test file:

    line 2: `from tests.helpers import value` should be `from helpers import value`

  These tests can't be run as written:

    line 11: test_price has 2 @cases. A test takes one.
    test_steps() has a `yield` in it.
    test_total(order) takes parameters, and nothing passes them in.

    order  1 test

  Each case is one flat tuple: the test's values, in the order of its
  parameters. For every combination of two lists, build the cases from
  both:

      @cases(*[(a, b, c) for a in FIRST for b, c in SECOND])
  ...

What is true of many files is said once. The first file whose tests take parameters carries the paragraph that says what a test takes, and the ones after it say what is wrong with their own file and name the first.

Test files in the application are one error for the run, reported first. More than three are counted by directory:

collection error app

  19 test files are in app/, and none of them was run:

    app/tests          17 files
    app/billing/tests  2 files

  Tests live in tests/, beside app/. app/ is the application: it is
  imported as `app`, and it is what gets deployed. Move them to tests/. A
  test file is the same file there, and a helper module it imports is
  imported by its path from tests/.

When more than one file has tests that take parameters, the run adds them up after the last file's error, the parameter most tests take first:

parameters nothing passes in

  Tests in 2 files take these:

  user    3 tests in 2 files
  client   1 test in 1 file

Here is what each message is asking for:

Message	What to do
`test_x(user, plan) takes parameters, and nothing passes them in.`	Remove the parameters. Build what the test needs in its body, or pass values with `@cases`
`test_x(a, b) doesn't fit its @cases: case [0] passes 1 value, for a. Nothing fills b.`	Make that case pass a value for each parameter, in their order
`line 8: TestCart is a class with 3 tests in it.`	Write each of its tests as a function of the file, and what they shared as functions they call
`test_x() has a yield in it.`	Pass the values it yielded with `@cases`, or move setup and cleanup into a `@contextmanager` helper
`test_x is defined in billing, not in this file.`	Define the test in this file, or import what isn't a test under a name that doesn't start with `test_`
`line 8: test_x has 2 @cases. A test takes one.`	Use one `@cases`, built from both lists, each case one flat tuple. The message shows how
`line 8: cases() ids must be unique`	Give the case another name. It is what another case is called, or numbered
`line 8: @skip requires a reason`	Write `@skip("why")`, not a bare `@skip`
`line 8: @tag requires at least one name`	Write `@tag("slow")`, not a bare `@tag`
`from tests.helpers import x should be from helpers import x`	Write the import the message gives. A helper module is imported by its path from `tests/`
`19 test files are in app/, and none of them was run:`	Move them to `tests/`, beside `app/`. Tests aren't kept in the application

Each of those is the runner telling you a test is written in a way it can't run, so it prints the message and nothing else. They're all one error, TestDefinitionError.

Anything else is an error of the file's own, raised while it was being imported. It's printed with its traceback, starting at the test file, and with what the file wrote while it was loading:

collection error tests/test_billing.py

  Traceback (most recent call last):
    File "/project/tests/test_billing.py", line 1, in <module>
      from billing_helpers import create_invoice
  ModuleNotFoundError: No module named 'billing_helpers'

As JSON

plain test --json prints nothing while the run goes, and one document when it's over. The document is all that's written to stdout, so it can be piped straight to what reads it. It says everything the text report says, with each thing the runner knew as a field of its own: no file, line, id or value has to be read out of a string.

plain test --json
plain test --json tests/test_orders.py --fail-fast
plain test --json --list-passed

{
    "version": 1,
    "outcome": "failed",
    "exit_code": 1,
    "duration": 0.0024,
    "command": {
        "argv": [
            "plain",
            "test",
            "--json"
        ],
        "directory": "/project",
        "targets": [],
        "match": null,
        "tags": [],
        "exclude_tags": [],
        "fail_fast": false,
        "full_values": false
    },
    "counts": {
        "selected": 3,
        "passed": 1,
        "failed": 1,
        "skipped": 1,
        "not_run": 0,
        "collection_errors": 1,
        "warnings": 1
    },
    "tests_listed": "failed_and_skipped",
    "tests": [
        {
            "id": "tests/test_orders.py::test_order_total",
            "file": "tests/test_orders.py",
            "line": 19,
            "name": "test_order_total",
            "tags": [
                "checkout"
            ],
            "outcome": "failed",
            "duration": 0.0021,
            "skip_reason": null,
            "failure": {
                "error_type": "AssertionError",
                "error_message": "",
                "file": "tests/test_orders.py",
                "line": 23,
                "traceback": "Traceback (most recent call last):\n  File \"/project/tests/test_orders.py\", line 23, in test_order_total\n    assert order == {\n    ...<5 lines>...\n    }\nAssertionError\n",
                "frames": [
                    {
                        "file": "tests/test_orders.py",
                        "line": 23,
                        "function": "test_order_total"
                    }
                ],
                "assert": {
                    "expression": "order == {\n    'items': ['tea', 'kettle'],\n    'currency': 'USD',\n    'subtotal': 40,\n    'shipping': 2,\n    'total': 42,\n}",
                    "message": null,
                    "parts": [
                        {
                            "source": "order",
                            "depth": 0,
                            "evaluated": true,
                            "value": {
                                "text": "<dict with 5 keys>",
                                "cut_characters": 0,
                                "same_as": null
                            }
                        }
                    ],
                    "diff": {
                        "lines": [
                            "--- order",
                            "+++ {'items': ['tea', 'kettle'], 'currency': 'USD', 'subtotal': 40, 'shipping': 2, 'total': 42}",
                            "@@ -1,5 +1,5 @@",
                            " {'currency': 'USD',",
                            "  'items': ['tea', 'kettle'],",
                            "- 'shipping': 0,",
                            "+ 'shipping': 2,",
                            "  'subtotal': 40,",
                            "- 'total': 40}",
                            "+ 'total': 42}"
                        ],
                        "cut_lines": 0
                    }
                },
                "locals": [
                    {
                        "name": "items",
                        "value": {
                            "text": "['tea', 'kettle']",
                            "cut_characters": 0,
                            "same_as": null
                        }
                    }
                ],
                "stdout": {
                    "text": "pricing 2 items\n",
                    "cut_characters": 0
                },
                "stderr": {
                    "text": "",
                    "cut_characters": 0
                },
                "rerun_command": "plain test tests/test_orders.py::test_order_total"
            }
        },
        {
            "id": "tests/test_orders.py::test_refund",
            "file": "tests/test_orders.py",
            "line": 32,
            "name": "test_refund",
            "tags": [],
            "outcome": "skipped",
            "duration": 0.0,
            "skip_reason": "Waiting on the new billing API",
            "failure": null
        }
    ],
    "collection_errors": [
        {
            "file": "tests/test_invoices.py",
            "line": 1,
            "is_definition_error": false,
            "error_type": "ModuleNotFoundError",
            "message": "No module named 'billing_helpers'",
            "traceback": "Traceback (most recent call last):\n  File \"/project/tests/test_invoices.py\", line 1, in <module>\n    from billing_helpers import create_invoice\nModuleNotFoundError: No module named 'billing_helpers'",
            "stdout": {
                "text": "",
                "cut_characters": 0
            },
            "stderr": {
                "text": "",
                "cut_characters": 0
            }
        }
    ],
    "parameters_nothing_passes": [],
    "warnings": [
        {
            "category": "DeprecationWarning",
            "message": "An order with no items is going away",
            "count": 1,
            "first_test": "tests/test_orders.py::test_empty_order",
            "file": "tests/test_orders.py",
            "line": 9
        }
    ],
    "stopped": null,
    "interrupted": null,
    "teardown_errors": [],
    "stdout": {
        "text": "",
        "cut_characters": 0
    },
    "stderr": {
        "text": "",
        "cut_characters": 0
    },
    "phases": [
        {
            "name": "python_startup",
            "seconds": 0.0381,
            "measured": "cpu",
            "parts": []
        },
        {
            "name": "command",
            "seconds": 0.0512,
            "measured": "wall",
            "parts": []
        },
        {
            "name": "runtime_setup",
            "seconds": 0.7706,
            "measured": "wall",
            "parts": [
                {
                    "name": "hook dev-setup",
                    "seconds": 0.3021,
                    "parts": []
                },
                {
                    "name": "other hooks (6)",
                    "seconds": 0.0104,
                    "parts": []
                },
                {
                    "name": "settings",
                    "seconds": 0.0188,
                    "parts": []
                },
                {
                    "name": "import app.agents",
                    "seconds": 0.4013,
                    "parts": []
                },
                {
                    "name": "other imports (27)",
                    "seconds": 0.0296,
                    "parts": []
                },
                {
                    "name": "ready()",
                    "seconds": 0.0084,
                    "parts": []
                }
            ]
        },
        {
            "name": "helper_modules",
            "seconds": 0.0012,
            "measured": "wall",
            "parts": []
        },
        {
            "name": "lifecycles_loaded",
            "seconds": 0.0058,
            "measured": "wall",
            "parts": []
        },
        {
            "name": "collection",
            "seconds": 0.0193,
            "measured": "wall",
            "parts": [
                {
                    "name": "rewrote 1 file",
                    "seconds": 0.0117,
                    "parts": []
                }
            ]
        },
        {
            "name": "lifecycle_setup",
            "seconds": 0.1503,
            "measured": "wall",
            "parts": [
                {
                    "name": "EmailTestLifecycle",
                    "seconds": 0.0,
                    "parts": []
                },
                {
                    "name": "PostgresTestLifecycle",
                    "seconds": 0.1503,
                    "parts": [
                        {
                            "name": "cloned template",
                            "seconds": 0.0412,
                            "parts": []
                        }
                    ]
                }
            ]
        },
        {
            "name": "tests",
            "seconds": 0.0024,
            "measured": "wall",
            "parts": []
        },
        {
            "name": "lifecycle_teardown",
            "seconds": 0.0471,
            "measured": "wall",
            "parts": [
                {
                    "name": "PostgresTestLifecycle",
                    "seconds": 0.0471,
                    "parts": []
                },
                {
                    "name": "EmailTestLifecycle",
                    "seconds": 0.0,
                    "parts": []
                }
            ]
        },
        {
            "name": "report",
            "seconds": 0.0003,
            "measured": "wall",
            "parts": []
        },
        {
            "name": "unaccounted",
            "seconds": 0.0019,
            "measured": "wall",
            "parts": []
        }
    ]
}

outcome is "passed", "failed", "interrupted" or "stopped", and exit_code is what the command exits with.
tests lists the tests that failed and the tests that were skipped. The ones that passed are counted in counts. --list-passed lists them too, and tests_listed says which it was: "failed_and_skipped" or "all". A suite of 1,700 passing tests is a document of 5 KB without them and 640 KB with them.
Every test has the same fields, whatever came of it. skip_reason and failure are null when there's nothing to say.
A test's file and line are where it's defined. A failure's file and line are the statement in the test that failed, or that called what failed. frames is the traceback as data, outermost first, down to where the error was raised. A failure that came from a lifecycle, not the test, has null for both.
assert is null for a failure that wasn't a failed assert. Its parts are what the text report prints under the assert: each part of the expression as written, how far inside it is, and what it was. A part Python never evaluated has evaluated: false and a null value.
A value is an object: text is what the text report prints, and cut_characters is how much the cap left off the end of it. same_as is null, or the name the failure has already printed this value under, and then text says so in place of the value. stdout and stderr are objects too, with cut_characters counting what was left off the start. --full-values leaves nothing off either.
What's known to be a secret is left out of every value here as it is from the text report. See Secrets.
warnings has each distinct warning once. count is how many times it was raised, and first_test, file and line are where it was raised first. counts.warnings is how many distinct ones there were.
The document's own stdout and stderr are what was written outside any test. A failure's are what its test wrote.
collection_errors have is_definition_error: true and no traceback when the file is written in a way the runner can't run. message says what to write instead. line is the line it is about when it is about one, such as a @cases() with no cases in it, and null when it is about several.
Paths are relative to command.directory, where the command was run from. A path outside it is absolute.
phases is where the time went, in the order it went there, and is there for every run, however short. A stopped run has the phases it got through. measured is "wall" for every phase but python_startup, which is "cpu": the CPU time the process had used before Plain was imported, the nearest it has to how long that took. parts are the lines under a phase in the text report, and seconds is the phase's whole, whatever its parts add up to.
version goes up when a field is renamed, removed, or changes what it means. A field being added doesn't change it.
parameters_nothing_passes is what the collection errors come to, when tests take parameters and nothing passes them in: each parameter, how many tests across the run take it, and in how many files. The one taken by the most tests is first.

{
    "parameters_nothing_passes": [
        {
            "name": "user",
            "tests": 3,
            "files": 2
        },
        {
            "name": "client",
            "tests": 1,
            "files": 1
        }
    ]
}

--json takes the same targets and filters as any run. It can't be combined with --verbose or --show-output: there's one document, and nothing else goes to stdout.

A run stopped with Ctrl-C still prints its document. interrupted says which test was running and what it had written:

{
    "interrupted": {
        "id": "tests/test_sync.py::test_every_page",
        "file": "tests/test_sync.py",
        "line": 1,
        "stdout": {
            "text": "fetching page 1\n",
            "cut_characters": 0
        },
        "stderr": {
            "text": "",
            "cut_characters": 0
        }
    }
}

A run that couldn't start prints one too, with no tests in it. reason says why, and exit_code follows from it:

`reason`	What happened	`exit_code`
`"lifecycle_error"`	`tests/lifecycle.py` can't be used	`2`
`"target_not_found"`	A target doesn't exist, or its line is in no test	`2`
`"setup_error"`	The app couldn't be set up, or a lifecycle's `setup_worker()` raised or exited	`3`
`"no_tests_found"`	No tests matched	`4`
`"interrupted"`	Ctrl-C, before the first test	`130`

stdout and stderr are everything that had been written by then, which for a "setup_error" is usually where the reason is.

{
    "stopped": {
        "reason": "lifecycle_error",
        "message": "/project/tests/lifecycle.py doesn't define a TestLifecycle subclass, so it would protect nothing. Define one:\n\n    from contextlib import contextmanager\n\n    from plain.testing import TestLifecycle\n\n\n    class AppTestLifecycle(TestLifecycle):\n        @contextmanager\n        def around_test(self, test):\n            ...\n            yield",
        "traceback": null,
        "stdout": {
            "text": "loading the lifecycle\n",
            "cut_characters": 0
        },
        "stderr": {
            "text": "",
            "cut_characters": 0
        }
    }
}

Two things aren't a document. An error in the runner itself is a traceback on stderr, and the command exits 1. A test file that calls sys.exit() while it's loaded ends the run with the code it gave. Either way what had been written is printed to stderr first, and stdout is left empty.

teardown_errors has an entry for each lifecycle that raised while being taken down, after the last test:

{
    "teardown_errors": [
        {
            "traceback": "Traceback (most recent call last):\n  File \"/project/tests/lifecycle.py\", line 7, in teardown_worker\n    raise RuntimeError(\"still in use\")\nRuntimeError: still in use\n",
            "stdout": {
                "text": "dropping the database\n",
                "cut_characters": 0
            },
            "stderr": {
                "text": "",
                "cut_characters": 0
            }
        }
    ]
}

Assertions

Use bare assert. When one fails, the failure shows the expression as you wrote it, and under it every value inside it:

def test_order_total():
    order = create_order(lines=2)
    assert len(order.lines) == expected_lines(order)

  assert len(order.lines) == expected_lines(order)
    len(order.lines) = 2
      order.lines = [<Line 1>, <Line 2>]
        order = <Order 7>
    expected_lines(order) = 3

That's every kind of expression: a comparison, a chain of them, a call, a membership test, and, or, not, arithmetic, await. There's nothing to add to an assert to see what went into it.

A message is for saying why, not for printing values. It's printed where Python prints it, at the end of the traceback:

assert items, "the cart should keep what was added before login"

  Traceback (most recent call last):
    File "/project/tests/test_cart.py", line 12, in test_cart_survives_login
      assert items, "the cart should keep what was added before login"
  AssertionError: the cart should keep what was added before login

  assert items
    items = []

What an assert does to your test

Nothing you can observe. Each part of the expression is evaluated once, in the order Python evaluates it, so an assert that calls something with a side effect behaves the way it would anywhere else. The error raised is an ordinary AssertionError, with your message or none.

A part Python never evaluated isn't evaluated by the runner either. The right side of an and whose left side was false is reported that way:

  assert user.is_admin and user.username == "grace"
    user.is_admin = False
      user = User(id=1, is_admin=False, username='ada')
    user.username == "grace"  (not evaluated)

An assert holds on to nothing once it has finished, so a test that checks an object has been freed isn't affected by the assert before it.

Three things are kept whole, without the values inside them: a comprehension, an f-string, and a generator passed to a call (all(n > 0 for n in rows) shows what all() returned). The values they were built from are in locals: when they're the test's own.

Only files named test_*.py are rewritten this way. An assert in a helper module fails as a bare AssertionError, with the traceback and the test's locals.

For an exception you expect, use raises from plain.testing.

What packages do for every test

An installed Plain package can wrap every test in protection of its own, with no setup on your part. Two do:

plain.postgres creates a test database for the run, and rolls back everything each test wrote.
plain.email collects sent mail in memory, so no test sends a real one, and empties the outbox before each test.

A package's protection applies only when the package is in your app's INSTALLED_PACKAGES.

Packages ship their test helpers too, in plain.<package>.test, and document them in their own READMEs. The Testing section of a package's README is where to look.

Project lifecycle

Some things have to be true for every test, and a test that forgets one fails in a way that's hard to see: a request reaches a real payment provider, or a rate limiter still holds the last test's count. That's protection, and it doesn't belong in each test's body. Declare it once, in tests/lifecycle.py:

# tests/lifecycle.py
from contextlib import contextmanager

from plain.testing import TestLifecycle, override_settings

from app.accounts import throttles


class AppTestLifecycle(TestLifecycle):
    @contextmanager
    def around_test(self, test):
        throttles.reset_all()
        # No test reaches the payment provider, whatever the environment holds.
        with override_settings(PAYMENTS_API_KEY=""):
            yield

The runner finds the file by its path. There is nothing to register and no other place it looks.

One file, one class. tests/lifecycle.py defines exactly one TestLifecycle subclass, under any name. around_test(test) wraps each test, and setup_worker() / teardown_worker() run once, before the first test and after the last. describe_value(value) says how a failure prints a value of your app's.
It fails loudly. If the file is there and doesn't import, defines no TestLifecycle subclass, defines more than one, or defines one that can't be created without arguments, the run stops before any test with a message saying which. When the file raised an error of its own, its traceback follows.
It runs closest to the test. Package lifecycles enter first, in the order of their entry point names, and the project's enters last. So the database transaction is already open when yours starts, and yours exits before the transaction is rolled back.
It's for protection, not setup. What tells them apart is whether a test's assertions depend on it. Protection is the same for every test in the suite, and no test asserts on it: it keeps the tests from the outside world (no request reaches the payment provider, whatever the environment holds) and from each other (a rate limiter starts from zero). Setup is anything a test's assertions depend on: a user, an organization, a logged-in client, and a setting that has to have one value for some tests to pass. A test gets those in its body, by calling a helper or entering a with block, even when every test in a directory needs the same one.
There is one, for the whole suite. There is no lifecycle for a directory or a file. A lifecycle that looks at which test it is wrapping to decide what to do is setup in the wrong place.
It's imported after your app is set up, and before any test file is. So it can import from app at the top of the file, as a test file does.

The file has to be tests/lifecycle.py. A lifecycle written somewhere the runner doesn't read would protect nothing, so the places one ends up by mistake are checked: tests/lifecycles.py, tests/life_cycle.py, tests/lifecycle/__init__.py, and lifecycle.py or lifecycles.py beside tests/. If one of those is there and mentions TestLifecycle, the run stops and says where the file belongs. A lifecycle.py deeper inside tests/ isn't checked, and isn't loaded.

tests/lifecycle.py imports helper modules the way a test file does, by their paths from tests/.

tests/ is the directory beside app/. If you run plain test from inside a directory named tests, the file is that directory's lifecycle.py.

Test lifecycles

A TestLifecycle is what happens around every test without the test asking. plain.postgres wraps each test in a transaction and rolls it back, and plain.email empties the outbox. It's for protection, which keeps tests from reaching each other or the outside world. It isn't for setup a test reads, which the test gets in its own body.

If you're writing a package, subclass it:

# mypackage/test.py
from contextlib import contextmanager

from plain.testing import TestLifecycle

from .registry import Entry, registry


class MyPackageTestLifecycle(TestLifecycle):
    required_package = "mypackage"

    def setup_worker(self):
        registry.use_in_memory_store()

    def teardown_worker(self):
        registry.use_default_store()

    @contextmanager
    def around_test(self, test):
        registry.clear()
        yield

    def describe_value(self, value):
        if isinstance(value, Entry):
            return f"Entry(key={value.key!r}, expires={value.expires!r})"
        return None

Then register it under the plain.testing entry point group:

# pyproject.toml
[project.entry-points."plain.testing"]
mypackage = "mypackage.test:MyPackageTestLifecycle"

setup_worker() runs once before the first test, and teardown_worker() once after the last.
around_test(test) is a context manager entered around each test. test is a CollectedTest, which you can import from plain.testing to annotate it. test.id is the id the runner prints (tests/test_cart.py::test_add[empty]), test.name is the part after the file (test_add[empty]), and test.tags holds its @tag names, so a lifecycle can treat a tagged test differently. That's how @isolated_db works.
describe_value(value) is what a failure prints for a value your package owns, in place of its repr. Return None for anything that isn't yours. Use it when the repr says too little to fix a test by. The text is printed as it is, and a description of several lines is diffed by line. It's called for the values in a failed assert and in the test's locals, and for what's inside them when they're a list, a tuple, a dict or a set. It must not change anything the test did, or do anything the test didn't: no queries, no requests. And it must not print what your package knows to be a secret: a report is read by more than the person who ran the test.
required_package keeps the lifecycle from loading unless that package is in the app's INSTALLED_PACKAGES. An entry point is visible whenever the package is installed in the environment, which is wider than "the app uses it".
Lifecycles are entered in the order of their entry point names. The runner creates each one with no arguments.

Your package never imports the runner. The entry point is a string, and the runner imports your class when it runs.

A project declares its own lifecycle in tests/lifecycle.py, with nothing to register. See Project lifecycle.

Testing code outside the app

plain test works in a project with no Plain app at all. The runner notices there's no app and leaves the app out:

Collection, assertion rewriting, targets and the flags all work.
So does everything in plain.testing that doesn't need an app: raises, @cases, @skip, @tag, skip_test, patch.
tests/lifecycle.py is still loaded.
Packages' protection is not. There's no test database and no outbox.
Client and override_settings need an app, and fail without one.
.env.test is loaded if plain.dev is installed, and not otherwise.

There's nothing to configure. Whether there's an app is decided the way every plain command decides it.

One run is one app. In a repository with several apps, run plain test once in each.

How it works

Everything about testing is in this one package, and it's a dev dependency. Nothing in Plain itself imports it.

plain.testing is what a test file imports. Client, build_request, raises, the decorators, the with helpers, the TestLifecycle class, and the CollectedTest a lifecycle is handed.

plain.testing.runner is the plain test command. It finds tests, rewrites assertions, runs them and reports. Nothing imports it, your tests included. It imports you. None of its modules is an API, and it knows nothing about the web: a runner that only collects and runs functions is why plain test works in a project with no app.

The client is built on Plain's public API. It constructs a Request and hands it to plain.server.inprocess, which runs the same middleware, routing and response handling the server does, without a socket. So a test sees what a browser would.

Each package owns its own testing. Its helpers are in plain.<package>.test, and what it does around every test is a TestLifecycle it registers under the plain.testing entry point group:

# plain-postgres/pyproject.toml
[project.entry-points."plain.testing"]
postgres = "plain.postgres.testing.lifecycle:PostgresTestLifecycle"

An entry point is a string in pyproject.toml, so a package declares its lifecycle without depending on this one.

There are two ways to extend the runner, and no others: a package's entry point, and your project's tests/lifecycle.py. There are no plugins and no hooks.

your tests            ──import──▶  plain.testing  +  plain.<package>.test
plain.testing.runner  ──drives──▶  lifecycle entry points  (one per package)
plain.testing.runner  ──drives──▶  tests/lifecycle.py      (your project's)

Design rules

plain.testing is built by eight rules. When a question about the API comes up, these settle it.

No backwards compatibility. There is one spelling of each thing, and no alias kept for an old one. /plain-upgrade rewrites what changes.
Explicit over implicit. Everything a test depends on is visible in its own file, as an import, a call, or a with block. Decorators declare, bodies acquire: a decorator attaches a static fact (its cases, its tags), and runtime state always enters in the body, where its scope is indentation.
One name per thing. If two spellings do the same job, one goes. The bytes a response sent are body, and nothing else.
Fail early, and say what to do. A mistake is rejected when the file is collected, with a message naming the fix. Nothing is silently overwritten or ignored: a test that asks for parameters nothing passes in, a second @cases, a bare @skip.
The report tells the truth. A test that didn't run is reported as skipped, with its reason. Nothing disappears from the count.
Repetition earns a helper, never magic. When the same lines appear in many tests, the answer is a named function or context manager the tests call. It is never injection.
Standard library first, and helpers live with their owner. The runner doesn't wrap what Python already does well (tempfile.TemporaryDirectory, math.isclose, contextlib.redirect_stdout). A helper specific to a package ships in that package's plain.<package>.test.
Setup is explicit, protection is automatic. State a test reads is acquired in its body. What guards tests from each other and from the outside world belongs in a lifecycle, which wraps every test without being asked. Packages ship theirs; your project declares its own in one place.

FAQs

What is the difference between Client and build_request?

Client sends the request through the full middleware and view pipeline and returns the response. build_request() only constructs the Request object — you call the view or middleware yourself.

How do I test file uploads?

Pass file-like objects via files={...} — they're encoded into a multipart body together with form_data.

Where are the database and email helpers?

With their packages. plain.testing holds only what isn't specific to one package, and each package documents its own helpers:

plain.postgres: isolated_db, capture_queries, max_queries
plain.email: outbox
plain.auth: login_client, logout_client
plain.sessions: get_client_session

How do I get a temporary directory, or read what was printed?

From the standard library. tempfile.TemporaryDirectory() gives a directory that's removed when the block exits, and contextlib.redirect_stdout(io.StringIO()) collects what was printed.

How do I debug a failing test?

Put breakpoint() where you want to stop and run the test. The debugger gets the terminal: from the breakpoint until that test is over, output is written as it happens. Calling pdb.set_trace() yourself doesn't do that, so use breakpoint(), or run with --show-output. Every failure prints the command that runs it again, so you can copy that to run the one test.

To see what a test prints without stopping it, print() and make it fail, or run it with --show-output.

Does coverage work?

Yes. python -m plain.testing is the same runner as plain test, so coverage run -m plain.testing works with no plugin.

Are there plugins?

No. There's nothing to load a plugin into: a package's entry point and your project's tests/lifecycle.py are the two ways to extend the runner. A library that needs no runner of its own keeps working as a library: freezegun and time-machine for freezing time, hypothesis for generated inputs, unittest.mock for mocks.

What about my editor's test explorer?

Editors find and run tests through a protocol this runner doesn't speak. Run tests from the terminal. Every failure prints the command that runs it again. Something that wants a run as data reads plain test --json.

Why is nothing passed to a test by name?

A test that is handed something by the name of its parameter depends on a file it doesn't mention. Here the same needs are met two other ways. Protection every test needs comes from a lifecycle: the database and the outbox from packages, your own from tests/lifecycle.py. Everything else is a function or a context manager the test imports and calls.

What if every test in a file needs the same setup?

Write it as a function, or as a context manager if it cleans up after itself, and call it in each test. That repeats a line in each test, and it's the line that says what the test depends on. What every test in the project needs for protection (no network, counters reset) belongs in the project lifecycle.

How do I compare two floats?

With the standard library: math.isclose(a, b, abs_tol=...) inside a bare assert.

Installation

Install the plain.testing package from PyPI as a dev dependency:

uv add plain.testing --dev

Then run your tests:

plain test

There's no configuration file. plain.dev loads .env.test, installed packages add their protection, and tests/lifecycle.py adds yours.

Metadata

Release files for plain.testing 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for plain.testing 0.1.0
File	Size	Uploaded
plain_testing-0.1.0.tar.gz	196.0 kB	Oct 6, 2026	Details

Built distribution (wheel)

Table of built distributions (wheels) for plain.testing 0.1.0
File	Interpreter	ABI	Platform	Reset
plain_testing-0.1.0-py3-none-any.whl 157.0 kB Oct 6, 2026	Python 3	none	any	Details

Total release size: 353.0 kB

Release files / plain_testing-0.1.0.tar.gz

Download URL	plain_testing-0.1.0.tar.gz
Size	196.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`7df47edab78f8559429c3d36a033f9cbd83689cf1616725ea25b55460085e658`
BLAKE2b-256 checksum How to use checksums	`90362d9ca3cfbe270b96a6c8a22174f3ad46207e6a856eddf03382f138d64473`
Upload date	Oct 6, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release files / plain_testing-0.1.0-py3-none-any.whl

Download URL	plain_testing-0.1.0-py3-none-any.whl
Size	157.0 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`343f3700a09f822ff69ec95c5f4331b003867a6ab03cbd3b0b1cb00aae305fab`
BLAKE2b-256 checksum How to use checksums	`eb699d9714e956290ab94e6285a7e0b263b41438765d40b5863226b49c58def4`
Upload date	Oct 6, 2026
Uploaded using Trusted Publishing? What is trusted publishing?	Yes
Uploaded via	`uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}`

Release history Release notifications | RSS feed

This release

0.1.0 This release

Oct 6, 2026 2 release files

plain.testing 0.1.0