durably

Durable execution for AI agents, with just Postgres.

Your agent crashes after a paid LLM call. durably resumes it on another worker without repeating the call — and without running a separate workflow server.

Early release (0.x): the core guarantees are tested; the API may change before 1.0. See the changelog.

What it is

durably is a small workflow engine for long-running agent loops. A workflow is an ordinary async Python function that calls activities (an LLM call, a tool call). The engine records every step in an append-only history in PostgreSQL. If a worker dies, another worker reruns the function against that history: recorded steps return their saved result instead of running again, and execution continues from the first unrecorded step.

Example. A research agent plans, runs three searches, summarizes with a paid LLM call, then sends an email. The worker is killed after the summary. A naive restart repeats the paid call and may send the email twice. With durably, the restart skips everything recorded and sends the email once.

Install

Requires Python 3.12+ and PostgreSQL. CI tests against PostgreSQL 16.

pip install durably

Then create durably's tables in your database (safe to run again):

durably init-db --database-url postgres://user:pass@host:5432/db

Every command and durably.Client.connect() read DATABASE_URL when no URL is given.

To see it work first, run the research agent example, which plans, searches, summarizes, waits for approval and sends an email:

git clone https://github.com/tarang802/Durable-agent-runtime && cd Durable-agent-runtime
docker compose run --rm tests python -m examples.research_agent.run

Usage

Activities do the real work. Each one receives a stable idempotency_key (workflow_id:activity_seq) to pass to the provider:

import durably

@durably.activity("llm")
async def llm(prompt, *, idempotency_key):
    return await my_llm_client.complete(prompt, idempotency_key=idempotency_key)

@durably.activity("send_email")
async def send_email(to, body, *, idempotency_key):
    await my_email_client.send(to, body, idempotency_key=idempotency_key)

A workflow is plain async code that calls activities through ctx:

@durably.workflow("research_agent")
async def research_agent(ctx, input):
    plan = await ctx.activity("llm", [f"Plan research on {input['topic']}"])
    summary = await ctx.activity("llm", [f"Summarize: {plan}"], retry={"max_attempts": 5}, timeout=60)
    await ctx.activity("send_email", [input["email"], summary])
    return {"summary": summary}

Workflows can sleep for minutes or days and wait for outside events. Both survive crashes and deploys:

@durably.workflow("publish_with_approval")
async def publish_with_approval(ctx, draft_id):
    draft = await ctx.activity("llm", [f"Write post {draft_id}"])
    decision = await ctx.wait_signal("review", timeout=3 * 24 * 3600)  # None if nobody answers in 3 days
    if decision is None or not decision["approved"]:
        return "not published"
    await ctx.sleep(3600)  # publish an hour later
    await ctx.activity("publish", [draft])
    return "published"

await client.signal(wf_id, "review", {"approved": True})

Long-running activities heartbeat so a hung attempt is detected and retried, resuming from the last saved progress:

@durably.activity("embed_documents")
async def embed_documents(doc_ids, *, idempotency_key):
    start = durably.activity_info().heartbeat_details or 0  # progress from a previous attempt
    for i in range(start, len(doc_ids)):
        await embed(doc_ids[i])
        await durably.heartbeat(i + 1)

await ctx.activity("embed_documents", [doc_ids], heartbeat_timeout=30)

Use ctx.now(), ctx.random() and ctx.uuid4() instead of datetime.now(), random and uuid: the first value is recorded, so replays see the same one.

Agents that run indefinitely (a chat assistant, a monitor) roll over to a fresh run before their history grows large, so replay stays fast:

@durably.workflow("assistant")
async def assistant(ctx, state):
    while True:
        message = await ctx.wait_signal("message")
        reply = await ctx.activity("llm", [state["memory"], message])
        state["memory"] = await ctx.activity("summarize", [state["memory"], message, reply])
        if ctx.should_continue_as_new():  # history reached --max-history events
            ctx.continue_as_new(state)  # unread messages carry over to the new run

Run workers, then start workflows from your app:

durably init-db
durably worker --kind workflow --module my_app.workflows
durably worker --kind activity --module my_app.activities

client = await durably.Client.connect()
wf_id = await client.start(research_agent, {"topic": "durable execution", "email": "me@example.com"},
                           idempotency_key="research:42")
print(await client.result(wf_id, timeout=300))

Guarantees

Each guarantee is checked by a test, most of them with real worker processes that are killed or frozen mid-run.

Crash matrix

A worker is killed at each failpoint, a replacement finishes the workflow, and the fake provider counts calls:

The one duplicate is unavoidable: the provider did the work but the worker died before recording it, so the retry calls again. It carries the same idempotency key, so an idempotent provider does not bill it twice.

Side effects run at least once, with a stable idempotency key (workflow_id:activity_seq). Exactly-once is impossible in general, so activities should be idempotent.

Benchmarks

Generated by the scripts in experiments/ on a laptop (Docker Desktop on Windows, 16 CPUs, Python 3.12, PostgreSQL 16). Full output: RESULTS.md. Regenerate with:

docker compose run --rm tests python -m experiments.run_all

Paid LLM calls when a 5-step agent's worker is killed after step k. A naive restart pays again for every completed step; durably pays for each step once.

Recovery time after a workflow worker dies, from its claim to the workflow finishing on a healthy worker. It tracks the lease length, so the lease is the knob.

Replay time vs history length (load from Postgres plus replay). Continue-As-New keeps a run at or below its limit, 1,000 events by default.

Activity throughput vs worker processes (2,000 no-op activities, each a claim, run and record). In this run two processes measured below one; throughput then grows with more processes.

When to use durably vs. Temporal

Choose durably when you want crash-safe agents inside an existing Python + Postgres backend without another service to run. Choose Temporal when you need multi-language workflows or scale beyond a single database.

Known limitations

• Activities in one workflow run one at a time; asyncio.gather over activities is not supported.
• An activity that runs longer than the worker lease (30 s by default) without a heartbeat_timeout is treated as crashed and run again. Heartbeat long activities.
• Workflow code must make the same calls on every replay: use ctx.now(), ctx.random() and ctx.sleep() rather than the clock, random or asyncio.sleep, and do network I/O in activities.
• Replay reads a run's whole history, so workflows that loop indefinitely should use Continue-As-New.
• No schema migrations: upgrading between 0.x releases may require re-creating tables.
• Side effects are at-least-once; make activities idempotent with the key provided.

Stack

Python 3.12+, asyncio and asyncpg (the only runtime dependency), PostgreSQL 16. Tests use pytest, Hypothesis, Toxiproxy and Docker Compose.

Development

Tests run in a Linux container next to Postgres 16 and Toxiproxy, so they behave the same on every machine (crash tests rely on POSIX signals). You only need Docker.

docker compose run --rm --build tests

Crash tests use failpoints: FAILPOINT=<name> makes a worker exit abruptly at that point, and FAILPOINT=<name>:pause freezes it with SIGSTOP. You can use them to crash-test your own workflows the same way.

What the suite covers:

• Crash recovery with real worker processes killed or frozen at each failpoint.
• Dropped connections: Toxiproxy cuts the workers' Postgres connections mid-activity; they back off, reconnect and finish without billing a step twice.
• Model-based testing: a Hypothesis state machine runs random interleavings of starts, signals, worker steps and crashes, and checks history against a reference model after every step.

When the model test fails it prints the minimal failing steps and a @reproduce_failure(...) line to paste above the test. To rerun a whole session deterministically, pass a seed (more examples with DURABLY_MODEL_EXAMPLES):

docker compose run --rm tests pytest tests/test_model.py --hypothesis-seed=12345

To create the schema in your own Postgres:

pip install -e .
durably init-db --database-url postgres://user:pass@host:5432/db

License

MIT