doeff

Write your Python logic once. Choose later how it talks to the world.

doeff brings algebraic effects to Python. Your functions yield requests such as "fetch this URL" or "summarize this text" instead of doing the I/O themselves. Handlers decide what each request means: real I/O in production, fakes in tests, recorded responses when you debug. Swap a handler, not your code.

pip install doeff          # Python 3.11+: the core (doeff, doeff-vm, doeff-core-effects)
pip install "doeff[hy]"    # + Hy and the handlers written in Hy (files, SQL, processes, cache, …)
pip install "doeff[http]"  # + the HTTP client handlers (Hy and httpx)
pip install "doeff[all]"   # both: what `pip install doeff` installed in 0.5.0

A 30-second example

from dataclasses import dataclass
from doeff import EffectBase, Pass, Resume, do, handler, run

@dataclass(frozen=True)
class Fetch(EffectBase):       # a request your code can make
    url: str

@dataclass(frozen=True)
class Summarize(EffectBase):
    text: str

@do
def digest(url):               # your logic: no I/O inside
    page = yield Fetch(url)
    return (yield Summarize(page))

@do
def fake_world(effect, k):     # a handler: answers the requests it knows
    if isinstance(effect, Fetch):
        return (yield Resume(k, "<h1>Hello</h1>"))
    if isinstance(effect, Summarize):
        return (yield Resume(k, f"{len(effect.text)} chars"))
    yield Pass(effect, k)      # anything else goes to the next handler

print(run(handler(fake_world)(digest("https://example.com"))))  # 14 chars

To go live, replace fake_world with handlers that call your HTTP client and your model. digest does not change, and the fake stays as your test.

Why doeff?

You want to…	Plain Python	doeff
Test code that does I/O	mock.patch("app.requests.get") for every import path	Install a fake handler. Nothing is patched
Add logging, retries or a cost cap	One wrapper per client	One handler that sees every effect
Run work in parallel	Rewrite the path with async or threads	Spawn, Gather and Race in the same program

• Batteries included. Production and test handlers ship for files, HTTP (record / replay), SQL (in-memory SQLite for tests), subprocesses, randomness, caching and metrics.
• Plain Python. Programs are generator functions; no new syntax. Optional Hy macros live in doeff-hy.
• Rust VM. Effects are dispatched by a VM written in Rust, with one-shot continuations.

One handler, every effect

A handler can watch every request without touching digest. Add this below the example above:

seen = []

@do
def trace(effect, k):          # sees every request first, then passes it on
    seen.append(effect)
    return (yield Pass(effect, k))

print(run(handler(fake_world)(handler(trace)(digest("https://example.com")))))  # 14 chars
print(seen)  # [Fetch(url='https://example.com'), Summarize(text='<h1>Hello</h1>')]

Before and after: fakes in a test

welcome loads a user from the database and sends a welcome mail. Its test must reach neither.

Before: plain Python with mock.patch. The test patches each dependency by the name app looks it up under. Forget one patch and the test still passes, after sending a real mail:

# app.py
from myapp.db import load_user
from myapp.mail import send_mail

def welcome(user_id):
    user = load_user(user_id)
    body = f"Welcome, {user.name}!"
    send_mail(user.email, body)
    return body

# test_app.py
from unittest import mock
import app
from myapp.db import User

def test_welcome():
    with mock.patch("app.load_user", return_value=User("Alice", "alice@example.com")):
        assert app.welcome(1) == "Welcome, Alice!"
    # Passes. app.send_mail was never patched, so the test sent a real mail.

After: doeff. welcome yields its requests. The test answers them with one fake handler per dependency, and nothing is patched:

from dataclasses import dataclass
from doeff import EffectBase, Resume, do, handler, run

@dataclass(frozen=True)
class User:
    name: str
    email: str

@dataclass(frozen=True)
class LoadUser(EffectBase):
    user_id: int

@dataclass(frozen=True)
class SendMail(EffectBase):
    to: str
    body: str

@do
def welcome(user_id):
    user = yield LoadUser(user_id)
    body = f"Welcome, {user.name}!"
    yield SendMail(user.email, body)
    return body

sent = []

@do
def fake_db(effect: LoadUser, k):      # the annotation picks the effects it answers
    return (yield Resume(k, User("Alice", "alice@example.com")))

@do
def fake_mail(effect: SendMail, k):    # keeps the mail instead of sending it
    sent.append(effect.to)
    return (yield Resume(k, None))

print(run(handler(fake_db)(handler(fake_mail)(welcome(1)))))  # Welcome, Alice!
print(sent)  # ['alice@example.com']

Forget fake_mail and nothing is sent: no handler answers SendMail, so the run stops and names it. Add this below the example above:

try:
    run(handler(fake_db)(welcome(1)))  # fake_mail left out
except Exception as error:
    print(type(error).__name__)  # UnhandledEffect

In production, welcome stays as it is and handlers that call your database and mail server take the place of the fakes. The doeff version is longer: each request is a class, and welcome is a generator. What the extra lines buy is a test that cannot reach production by accident.

The same test, written in other styles, with one fake left out:

Style	The logic is	The test swaps in fakes by	Forget one fake and
Plain Python, mock.patch	plain functions	patching each name where the code looks it up	the real function runs
Plain Python, dependencies as arguments	functions that take their dependencies	passing fake functions	a TypeError if the test builds its own set; the real function runs if the test starts from production's set
effect 1.1.0	generators that yield Effect(...)	listing the expected effects and their answers in order	an AssertionError (Performer not found)
effectful 0.4.0	plain functions that call @defop operations	a dict of handlers	no error: the call returns a symbolic term instead of a value
doeff	generators that yield effects	installing one handler per effect	an UnhandledEffect error

A doeff handler without an effect annotation sees every effect: without : LoadUser, fake_db would answer SendMail too, and the test without fake_mail would pass.

Next steps

• Tutorial: Getting started
• Why effects?: Effects vs. dependency injection
• Built-in effects and handlers: doeff-core-effects
• Writing programs in Hy: doeff-hy
• Command line: doeff run

Runtime API

Entrypoint	Signature	Use case
run	run(doexpr)	Execute a DoExpr program to completion

run() takes a single DoExpr (program node) and returns the result value directly. Handlers are composed explicitly by calling a Program -> Program handler installer. Use scheduled(prog) to wrap with the scheduler for concurrency effects.

Handler Composition

Handlers are installed by direct call. Stack multiple handlers by wrapping the program:

from doeff import do, run
from doeff_core_effects import Ask
from doeff_core_effects.handlers import reader

@do
def prog():
    return (yield Ask("greeting"))

result = run(reader(env={"greeting": "hello"})(prog()))
print(result)  # hello

Reusable custom handlers should expose the same Program -> Program shape. In Hy, defhandler creates that shape directly:

(require doeff-hy.handle [defhandler])
(require doeff-hy.macros [do! <-])
(import doeff [Ask run])

(defhandler ask-env
  (Ask [key]
    (resume (get {"greeting" "hello"} key))))

(setv greet
  (do!
    (<- greeting (Ask "greeting"))
    greeting))

(print (run (ask-env greet)))  ; hello

Hy comes with the extra hy: pip install "doeff[hy]". Save the example as greet.hy and run it with hy greet.hy. Macros such as do!, <- and defhandler are brought in with require, never with import.

Using a Hy module from Python

Python reads a .hy file only after import hy has run, so put import hy before the import of a Hy module:

(require doeff-hy.macros [do! <-])
(import doeff [Ask])

(setv greet
  (do!
    (<- greeting (Ask "greeting"))
    greeting))

import hy  # first: registers Hy's import hook, so Python can find greet.hy
from doeff import run
from doeff_core_effects.handlers import reader
from greet import greet

print(run(reader(env={"greeting": "hello"})(greet)))  # hello

Common Hy mistakes and the line that fixes each

You see	Why	Fix
NameError: name 'hyx_doXexclamation_markX' is not defined (or name 'defk' is not defined)	A macro was used without a require, so Hy compiled it as a call to an undefined function. doeff prints a HyMacroUseWarning with the line to add just before the traceback.	Add (require doeff-hy.macros [do! <-])
ImportError: cannot import name 'defhandler' from 'doeff_hy.handle'	A macro was brought in with import.	Write (require doeff-hy.handle [defhandler]) instead of (import doeff-hy.handle [defhandler])
ModuleNotFoundError: No module named 'greet' with a greet.hy next to the script	Python does not read .hy files until import hy has run.	Add import hy above from greet import greet

Effect Surface

Core effects (from doeff_core_effects):

• Reader: Ask(key), Local(env, program)
• State: Get(key), Put(key, value)
• Writer: Tell(message), slog(msg, **kwargs)
• Error: Try(program) — returns Ok(value) or Err(error)
• Observe: Listen(program, types=...) — collect effects during execution

Scheduler effects (from doeff_core_effects.scheduler):

• Concurrency: Spawn(program), Wait(task), Gather(*tasks), Race(*tasks)
• Promises: CreatePromise(), CompletePromise(p, v), FailPromise(p, e)
• External: CreateExternalPromise() — bridge external threads
• Semaphores: CreateSemaphore(n), AcquireSemaphore(s), ReleaseSemaphore(s)

Handlers

Built-in handlers (from doeff_core_effects.handlers):

Handler	Installer	Effects handled
Reader	reader(env={...})	Ask
Lazy Ask	lazy_ask(env={...})	Ask, Local (with caching)
State	state(initial={...})	Get, Put
Writer	writer	Tell / WriterTellEffect
Try	try_handler	Try
Slog	slog_handler	Slog
Local	local_handler	Local
Listen	listen_handler	Listen
Await	await_handler()	Await
Scheduler	scheduled(prog) (from doeff_core_effects.scheduler)	Spawn, Wait, Gather, Race, etc.

writer keeps its log in state: install state() outside it, and read the log with yield writer_log() (from doeff_core_effects.handlers).

Scheduler and Concurrency

from doeff import do, run
from doeff_core_effects import Tell
from doeff_core_effects.handlers import state, writer
from doeff_core_effects.scheduler import scheduled, Spawn, Gather

@do
def worker(label):
    yield Tell(f"working: {label}")
    return label

@do
def main():
    t1 = yield Spawn(worker("a"))
    t2 = yield Spawn(worker("b"))
    results = yield Gather(t1, t2)
    return results

# writer keeps its log in state, so install state() outside writer.
result = run(scheduled(state()(writer(main()))))
print(result)  # ['a', 'b']

CLI

# Run a program with auto-discovered interpreter
doeff run --program myapp.module.program

# Inline Python code
doeff run -c 'return 42'

# Inline Hy code (preferred — compose handlers inline)
doeff run --hy '(import myapp [p]) (import doeff-core-effects [lazy-ask]) ((lazy-ask :env myapp.env_dict) p)'

# JSON output
doeff run --program myapp.program --format json

Note: Legacy flags --interpreter, --env, --set, --apply, and --transform still work but emit deprecation warnings. Use --hy for inline handler composition instead.

The CLI supports automatic interpreter/environment discovery via doeff-indexer. See docs/14-cli-auto-discovery.md for marker syntax and hierarchy rules.

Related packages

The add-on packages (LLM clients, agents, workflows and more) live in packages/.

doeff-agents 0.1.1 on PyPI was built for doeff 0.4 and does not work with doeff 0.5 or later — importing its handlers fails with ImportError: cannot import name 'WithHandler' from 'doeff'. To keep using that release, pin the doeff 0.4 line: pip install doeff-agents "doeff<0.5". The doeff-agents in this repository (packages/doeff-agents, with its CLI) is not published to PyPI.

Repository map

The paths for users come first. The rest are for people who work on doeff itself.

Path	For	What is inside
doeff/, packages/	users	the library and its add-on packages
docs/index.md	users	start here: guides and API notes
examples/	users	runnable examples
ide-plugins/	users	editor plugins for PyCharm, VS Code and Neovim
docs/adr/	contributors	design decisions written as executable tests
tests/, scripts/, tools/, benchmarks/	contributors	the test suite, development tools and benchmarks
docs/design/, docs/design-checks/	maintainers	development records; not user documentation
docs/history/	maintainers	older planning notes
publications/, images/	maintainers	drafts of articles about doeff and their images

docs/README.md tells the user guides in docs/ apart from the development records.

Contributing

Setting up the development environment, running the tests and running the benchmarks are described in CONTRIBUTING.md.

License

MIT License. See LICENSE.