doeff - Algebraic Effects for Python

doeff is an algebraic effects runtime for Python with one-shot continuations and a Rust VM. Programs are written with generator-based @do notation and executed through explicit handler stacks.

Installation

pip install doeff

Note on doeff-agents from PyPI: doeff-agents 0.1.1 on PyPI was built for doeff 0.4 and does not work with doeff 0.5 — importing its handlers fails with ImportError: cannot import name 'WithHandler' from 'doeff'. To keep using that release, pin the doeff 0.4 line: pip install doeff-agents "doeff<0.5". The doeff-agents in this repository (packages/doeff-agents) is not published to PyPI.

Quick Start

from doeff import do, run
from doeff_core_effects import Get, Put, Tell
from doeff_core_effects.handlers import reader, state, writer
from doeff_core_effects.scheduler import scheduled

@do
def counter_program():
    yield Put("counter", 0)
    yield Tell("Starting computation")
    count = yield Get("counter")
    yield Put("counter", count + 1)
    return count + 1

# Compose handlers explicitly by calling each handler installer.
prog = counter_program()
prog = writer(prog)
prog = state()(prog)
prog = reader(env={"greeting": "hello"})(prog)
result = run(scheduled(prog))
print(result)  # 1

Runtime API

run() takes a single DoExpr (program node) and returns the result value directly. Handlers are composed explicitly by calling a Program -> Program handler installer. Use scheduled(prog) to wrap with the scheduler for concurrency effects.

Handler Composition

Handlers are installed by direct call. Stack multiple handlers by wrapping the program:

from doeff import do, run
from doeff_core_effects import Ask
from doeff_core_effects.handlers import reader

@do
def prog():
    return (yield Ask("greeting"))

result = run(reader(env={"greeting": "hello"})(prog()))
print(result)  # hello

Reusable custom handlers should expose the same Program -> Program shape. In Hy, defhandler creates that shape directly:

(require doeff-hy.handle [defhandler])
(require doeff-hy.macros [do! <-])
(import doeff [Ask run])

(defhandler ask-env
  (Ask [key]
    (resume (get {"greeting" "hello"} key))))

(setv greet
  (do!
    (<- greeting (Ask "greeting"))
    greeting))

(print (run (ask-env greet)))  ; hello

Effect Surface

Core effects (from doeff_core_effects):

• Reader: Ask(key), Local(env, program)
• State: Get(key), Put(key, value)
• Writer: Tell(message), slog(msg, **kwargs)
• Error: Try(program) — returns Ok(value) or Err(error)
• Observe: Listen(program, types=...) — collect effects during execution

Scheduler effects (from doeff_core_effects.scheduler):

• Concurrency: Spawn(program), Wait(task), Gather(*tasks), Race(*tasks)
• Promises: CreatePromise(), CompletePromise(p, v), FailPromise(p, e)
• External: CreateExternalPromise() — bridge external threads
• Semaphores: CreateSemaphore(n), AcquireSemaphore(s), ReleaseSemaphore(s)

Handlers

Built-in handlers (from doeff_core_effects.handlers):

writer keeps its log in state: install state() outside it, and read the log with yield writer_log() (from doeff_core_effects.handlers).

Scheduler and Concurrency

from doeff import do, run
from doeff_core_effects import Tell
from doeff_core_effects.handlers import state, writer
from doeff_core_effects.scheduler import scheduled, Spawn, Gather

@do
def worker(label):
    yield Tell(f"working: {label}")
    return label

@do
def main():
    t1 = yield Spawn(worker("a"))
    t2 = yield Spawn(worker("b"))
    results = yield Gather(t1, t2)
    return results

# writer keeps its log in state, so install state() outside writer.
result = run(scheduled(state()(writer(main()))))
print(result)  # ['a', 'b']

CLI

# Run a program with auto-discovered interpreter
doeff run --program myapp.module.program

# Inline Python code
doeff run -c 'return 42'

# Inline Hy code (preferred — compose handlers inline)
doeff run --hy '(import myapp [p]) (import doeff-core-effects [lazy-ask]) ((lazy-ask :env myapp.env_dict) p)'

# JSON output
doeff run --program myapp.program --format json

Note: Legacy flags --interpreter, --env, --set, --apply, and --transform still work but emit deprecation warnings. Use --hy for inline handler composition instead.

The CLI supports automatic interpreter/environment discovery via doeff-indexer. See docs/14-cli-auto-discovery.md for marker syntax and hierarchy rules.

The agent session host that connects to an agent control plane (doeff-sessionhost) and the doeff-agents CLI are documented in packages/doeff-agents/README.md.

Performance Benchmarks

Rust VM dispatch micro-benchmarks live in packages/doeff-vm-core/benches/ and use Criterion. Run them from the VM core crate with the VM feature enabled:

cd packages/doeff-vm-core
cargo bench --features python_bridge

If PyO3 selects the wrong Python installation on macOS, pin it to the uv environment:

cd packages/doeff-vm-core
PYO3_PYTHON="$(cd ../.. && uv run python -c 'import sys; print(sys.executable)')" \
  cargo bench --features python_bridge

Python end-to-end benchmarks live in benchmarks/benchmark_runner.py. A normal run writes JSON to benchmarks/results/<yyyymmdd>-<hostname>.json:

uv run python benchmarks/benchmark_runner.py --runs 20

Compare a fresh run against a committed baseline with:

uv run python benchmarks/benchmark_runner.py \
  --compare benchmarks/results/<baseline>.json

CI uses smoke mode only. It runs one iteration with the smallest N values and checks that the benchmark entrypoints still execute; it does not gate on performance:

make bench-smoke

Development

make sync            # install deps (Rust VM is rebuilt only when its sources change)
uv run pytest        # run full test suite

Warning: uv sync alone does NOT rebuild the Rust VM extension. Always use make sync after editing .rs files under packages/doeff-vm/.

Maintainers: the landing queue and the verification rigs live in docs/maintainers.md.

License

MIT License. See LICENSE.