AWS Durable Execution SDK for Python: compute worker

This package is experimental. Its API can change in any release.

The compute worker runs jobs from durable functions on compute that is defined outside the durable function. Each compute type has its own subpackage. Today the package has one subpackage: aws_durable_execution_sdk_python_compute_worker.microvm, the worker for AWS Lambda MicroVMs. The rest of this README describes the MicroVM worker.

Run jobs from a durable function inside an AWS Lambda MicroVM, and report each job's result to the durable function.

The worker runs in the MicroVM image, not in the durable function. It does not depend on the durable execution SDK. The job document is the same in every SDK language. So a durable function in any language can send jobs to this worker.

Install

pip install aws-durable-execution-sdk-python-compute-worker

Quick start

import logging

from aws_durable_execution_sdk_python_compute_worker.microvm import (
    MicrovmJobContext,
    start_microvm_worker,
)


def run_build(job_input: dict, context: MicrovmJobContext) -> dict:
    return {"repo": job_input["repo"], "passed": True}


if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    worker = start_microvm_worker(handler=run_build)
    worker.wait()

Start the worker when the image's process starts. start_microvm_worker serves in a daemon thread and returns. So the entry point ends with worker.wait().

The handler's return value becomes the durable function's result. An exception fails the job with the exception's class name and message.

Common uses

Serve named routes

from aws_durable_execution_sdk_python_compute_worker.microvm import (
    MicrovmJobContext,
    start_microvm_worker,
)


def run_build(job_input: dict, context: MicrovmJobContext) -> dict:
    return {"passed": True}


def clone_build(job_input: dict, context: MicrovmJobContext) -> dict:
    return {"cloned": job_input["repo"]}


if __name__ == "__main__":
    worker = start_microvm_worker(
        handler=run_build,  # Jobs that name no route.
        routes={"/clone-build": clone_build},  # Jobs sent to this path.
    )
    worker.wait()

The durable function sends a job to a route with MicrovmRequestConfig(path=...) or MicrovmInvokeOptions(path=...). Use the same string in both places.

Stop a cancelled job

context.cancelled is a threading.Event. The worker sets it when the durable function stops waiting, for example after a heartbeat timeout, and when Lambda calls the terminate hook. Python cannot stop a running thread. So check the event in long work.

import subprocess

from aws_durable_execution_sdk_python_compute_worker.microvm import MicrovmJobContext


def run_build(job_input: dict, context: MicrovmJobContext) -> dict:
    for command in job_input["commands"]:
        if context.cancelled.is_set():
            return {"cancelled": True}
        subprocess.run(command, shell=False, check=True)
    return {"passed": True}

Run async code

A handler must be a plain function. An async def handler raises TypeError. Call asyncio.run inside the handler instead:

import asyncio

from aws_durable_execution_sdk_python_compute_worker.microvm import MicrovmJobContext


async def build(job_input: dict) -> dict:
    await asyncio.sleep(1)
    return {"passed": True}


def run_build(job_input: dict, context: MicrovmJobContext) -> dict:
    return asyncio.run(build(job_input))

Advanced

Image configuration

Set these hooks in CreateMicrovmImage. port must match the worker's port, 8080 by default.

{
  "hooks": {
    "port": 8080,
    "microvmHooks": {
      "run": "ENABLED",
      "runTimeoutInSeconds": 10,
      "resume": "ENABLED",
      "resumeTimeoutInSeconds": 10,
      "suspend": "ENABLED",
      "suspendTimeoutInSeconds": 10,
      "terminate": "ENABLED",
      "terminateTimeoutInSeconds": 10
    },
    "microvmImageHooks": { "ready": "ENABLED" }
  }
}

• The run hook is required. It delivers small jobs, or tells the worker that jobs arrive over HTTP.
• The suspend and resume hooks are needed for sessions that suspend. Without resume, the first job after a resume can wait about 30 seconds.
• The terminate hook lets the worker fail running jobs at once when the MicroVM ends. Give it at least 10 seconds.
• The service requires the ready hook when any lifecycle hook is enabled.

Suspending

The worker suspends its own MicroVM when the session sets auto_suspend_on_idle=True. It never suspends during a job. The SDK never calls ResumeMicrovm. So the session's MicroVM must set idlePolicy.autoResumeEnabled, and the service resumes it on the next request. Setting that is the job of the durable function's developer. Set maxIdleDurationSeconds above the MicroVM's maximumDurationInSeconds.

The worker counts only jobs. It does not see background threads, processes, or other inbound traffic. Leave auto_suspend_on_idle off for a session with such work.

With suspend_after, the durable function suspends the MicroVM itself. The worker does nothing for it.

Results and heartbeats

• Return a JSON-serializable value of at most 256 KB. The worker reports any other value as a failure of the job. Store a larger result elsewhere, for example in S3, and return its key.
• The worker sends heartbeats while a job runs, when the job sets a heartbeat timeout.
• The worker ignores a second delivery of a job that it already ran. It remembers the last 1,000 finished jobs. So make side effects safe to repeat across process restarts.

Logging

The worker logs to the standard library logger aws_durable_execution_sdk_python_compute_worker.microvm. Pass logger= to replace it. Configure logging in the image to see its INFO lines.

Permissions

The MicroVM's execution role needs lambda:SendDurableExecutionCallbackSuccess, lambda:SendDurableExecutionCallbackFailure, and lambda:SendDurableExecutionCallbackHeartbeat on the durable function's ARN (arn:aws:lambda:<region>:<account>:function:<name>:*).

With auto_suspend_on_idle, the role also needs lambda:SuspendMicrovm on the MicroVM image. That permission lets a MicroVM suspend any MicroVM of the same image. So use one image per trust boundary.

The worker needs boto3 1.43.35 or later.

Reference

• docs/worker.md: the request flow, the lifecycle hooks, idle suspend outcomes, the job documents, reporting, heartbeats, logging fields, and best practices.
• extras: the microvm and microvm_session operations that send jobs.
• Source: lower-level building blocks such as MicrovmWorkerListener and CallbackReporter.

License

This project is licensed under the Apache-2.0 License.