due-work-harness

Crash-test your background jobs. Pytest proofs that background work survives lost messages, dead workers and uncertain external calls, and that the test saying so isn't lying.

What it is, in 45 seconds (with sound):

https://github.com/user-attachments/assets/498ac071-080c-45d2-abdb-13aee405568b

Your happy-path test is an optimist

Most applications record something now and finish it later: send the confirmation email, generate the thumbnails, publish the post, retry the failed upload. That is due work, and it fails quietly:

• The commit landed. The message didn't. The order exists; its email never will.
• The email went out. Then the worker died. Nothing recorded the send, so recovery sends it again.
• The stale worker finished last. It overwrote the fresh result its replacement had just written.

Nothing throws, nothing pages anyone, and the test that runs the happy path once stays green. due-work-harness kills your worker on purpose, after every commit and after every external call, and fails unless every one of those histories ends where normal operation does.

Not theoretical: it found these in code you've heard of

_{Real output, recorded with vhs from docs/demo.tape. CI replays every command it types (demos/test_readme_gif.py), so the GIF can't claim output the harness no longer prints.}

The demos run the harness against demo applications and a real e-commerce platform, exactly as their projects ship them, pinned to a commit:

Demos are teaching code, and good at what they teach. The point is what the harness finds when code like this is copied into production, and the one change that makes the same proof pass. Every finding is a strict xfail with its explanation, so an upstream fix turns it red. The full write-up: demos/README.md.

The findings, in 30 seconds: Saleor's checkout run, its fix passing, then DBOS, procrastinate and the coverage scan, all real output:

https://github.com/user-attachments/assets/ed065709-3372-40dd-8dc9-7f64dc60ef90

Install

pip install due-work-harness            # the core: pytest and pydantic, nothing else
pip install "due-work-harness[django]"  # plus the Django/PostgreSQL integration

Or uv add --dev due-work-harness. The extras are [django], [celery], [procrastinate], [dbos], [redis] and [rq]; combine as needed. Python 3.12+.

Kill it on purpose: crash histories

Give the harness one transition that hands work off. It runs normal operation to learn the outcome, then replays the same transition:

• with every message it published lost;
• with the worker killed right after each commit it makes;
• with the worker killed right after each external call returns;
• with each after-commit callback failing, each signal receiver or job callback failing, the broker refusing each publication, and the reply to each commit lost (the write lands, the worker sees a connection error and carries on), where your host supports them (the Django host breaks callbacks; the Redis host loses replies; add the Celery integration's breaker to refuse publishes).

Every history must reach the outcome normal operation reached. The harness finds the commit boundaries itself; you don't name them.

from due_work_harness import CallableDelivery, ExternalCall, HandoffHistory, assert_crash_at_every_commit_converges


def test_placing_an_order_survives_any_death():
    assert_crash_at_every_commit_converges(
        CallableDelivery(name="orders", recover=run_workers_until_idle),  # what production runs after a crash
        HandoffHistory(
            name="place order",
            arrange=new_cart,
            transition=place_order,  # your real view or service method
            observe=lambda cart: (order_status(cart), mailbox.count(cart)),  # include what external systems saw
            external_calls=(ExternalCall(mailer, "send"),),
        ),
    )

A failure names the history that went wrong and what it left behind:

AssertionError: orders: handoff 'place order': normal operation reaches ('SENT', 1), but these
histories reach something else: {'worker died after external call 1': ('SENT', 2)}. Work was lost
or repeated. ...

That's the duplicate email, caught in a test instead of in a support ticket. When a test can't reach the worker at all (a workflow engine's own executor), process_histories runs the real program in a child process, kills it at named points with os._exit, restarts it the way production would, and applies the same verdict. That is how the DBOS demo above works.

Six ways due work goes missing: lifecycle profiles

A crash history proves one handoff. A declarative DueWorkContract binds your production selection, tick and transitions, and asks for a disposition (claim, decline with a reason, not applicable, or known gap) for each of six profiles:

Every contract also disposes of two execution-safety profiles: replay safety and bounded retry. Read what a green result means before you treat a pass as a guarantee.

Green you can trust

A conformance suite can lie in ways ordinary tests can't. An adapter that copies production's query, or implements a tiny state machine in the test file, makes every proof measure the copy, and it stays green forever. So the harness:

• refuses test-authored selections and transitions, and requires every binding to reach code in your production_packages;
• forbids waiving its own integrity checks;
• pairs every "must not happen" proof with a positive control showing the same binding would act;
• fails a run where every crash history agrees but recovery changed nothing: agreeing with an inert recovery is not a pass.

The full catalogue of lies it refuses: false greens.

No handoff left behind: the coverage scan

A crash history proves one handoff; the coverage check makes sure none are forgotten. It scans production code for every call that hands work off: on_commit, a Celery .delay, a procrastinate .defer, a DBOS workflow start, a Dramatiq .send, an RQ enqueue, a Django task's .enqueue. It attributes each to the exact function that makes it, and requires exactly one disposition per function: a contract suite that insures it, or an exemption that proves losing it costs nothing. In Saleor it finds 137 handoff sites in 124 functions.

uv run due-work-harness check      # static: imports neither your app nor your tests

@due_work_contract_suite(ORDER_NOTIFICATIONS, covers=(DueWorkSource(OrderService.place),))
class TestOrderNotificationsDueWork:
    pass

Aliases, re-exports and handoffs passed along uncalled are still found. A declaration counts only if pytest would actually run it, and pytest --due-work-verify then checks that it did. An existing project adopts with a baseline that only shrinks.

In GitHub Actions, gigaverse-app/due-work-harness/check@v0 runs the scan and gigaverse-app/due-work-harness/test@v0 runs the generated suites (every generated case carries the due_work mark). The path from install to CI is in ADOPTING.md.

Bring your own framework

The core depends only on pytest and pydantic: no Django, SQLAlchemy, Celery, procrastinate or DBOS, and CI imports every core module with none of them installed. Proofs take plain callables. The few facts only a framework knows (whether a transaction is open, how to interrupt a commit, what plan a query runs) come from a Host you configure once:

# conftest.py
from due_work_harness import configure
from due_work_harness.integrations.django import django_host

configure(django_host(production_packages={"myapp"}))

Status

Alpha. The proofs were extracted from a production codebase, where they guard its background workflows in CI; the public API may still change before 1.0. Python 3.12+. How it's built: ARCHITECTURE.md.

Development

Managed with uv (uv sync --all-extras), linted with Ruff and type-checked with Pyrefly. See CONTRIBUTING.md.

License

Apache-2.0. See LICENSE.