Skip to main content

sfnkit

Write AWS Step Functions workflows as plain Python. Steps are decorated functions that run as Lambda container images. The workflow body is ordinary Python, including match, if and with blocks, and is compiled into a state machine.

pip install sfnkit

Requires Python 3.12+. Deploying also needs Docker (to build the Lambda images) and AWS credentials that can create IAM roles.

from sfnkit import step, workflow, attempt, Ok, Err, each, subworkflow, wait, fail, is_null
from sfnkit import Lambda, Retry, Allow, Policy

base = Policy(Allow("s3:GetObject", "arn:aws:s3:::data/*"))

@step(compute=Lambda(memory=1024, timeout=120), deps=["httpx>=0.27"])
def fetch(url: str) -> list[str]: ...

@step(deps=["pyyaml"], retry=Retry(on=[ValueError], max_attempts=3))
def parse(doc: str) -> dict: ...

@workflow(iam=base)
def pipeline(url: str, label: str = "run"):
    docs = fetch(url)
    rows = parse.map(docs, max_concurrency=10)          # Map state

    with subworkflow("scoring"):                         # child state machine
        score = score_rows(rows)

    match score["tier"]:                                 # Choice state
        case "gold" | "silver":
            order = publish(rows)
        case "review" if score["value"] > 0.5:
            wait(seconds=300)
            order = queue_for_review(rows)
        case _:
            fail("LowScore", "score below 0.5")

    match attempt(charge, order):                        # Catch
        case Ok(receipt):
            pass
        case Err("PaymentDeclined", cause):
            receipt = refund(order, cause)

    return {"receipt": receipt}
sfnkit local  myapp.flows:pipeline --input '{"url": "https://..."}'
sfnkit synth  myapp.flows:pipeline -o out/
sfnkit deploy myapp.flows:pipeline --region us-west-2
sfnkit run    myapp.flows:pipeline --input '{"url": "https://..."}'
sfnkit destroy myapp.flows:pipeline

How a workflow body becomes a state machine

A step called inside a @workflow doesn't run. It returns a ref to its future output, and the call is recorded. Called anywhere else, a step is a normal Python function.

A condition on a ref (==, <, in, and, if x:) is a decision. The tracer runs the workflow body once per combination of decisions and merges the runs into one graph:

  • Calls that don't depend on each other run in one Parallel state.
  • Each decision becomes a Choice state.
  • A variable assigned in several branches is merged automatically, so it can be used after the if/match like normal Python.

Because the body runs once per path, it must be deterministic and free of side effects. Paths are capped at max_paths (default 1024). Each @workflow is traced on its own, so splitting a large body into child workflows keeps the count down.

Control flow

Python Step Functions
independent step calls Parallel
if/elif/else, match with literal, value and OR patterns and guards Choice
step.map(items, **kw), with each(items) as it: Map (inline), or Distributed with distributed=True
match attempt(fn, ...) with case Ok(v) / case Err(name, cause) Catch
with subworkflow("name"):, calling another @workflow child state machine (startExecution.sync:2)
wait(seconds=...), wait(until=...) Wait. A barrier: everything before it runs first
fail(error, cause) Fail

attempt(fn, *args) runs a step, a workflow, or a plain function containing several steps.

  • It catches only the error names its match handles. A Python exception's name is its class name, so Err(errors.PaymentDeclined, c) and Err("PaymentDeclined", c) match the same error.
  • Errors no case handles fail the workflow, just as they would without attempt.
  • case Err(error, cause): catches everything. cause is Lambda's error object: errorMessage, errorType and stackTrace.

each(items) takes items from a workflow input, a literal list, a step's output (or a field of it), or another inline map's results. A Distributed Map can also read straight from S3 with S3Objects(bucket, prefix) or S3File(bucket, key, format="json"|"jsonl"|"csv"). Inside the block, it.item is the current item and it.result(x) sets that item's result. After the block, it.value is the list of results.

Checked when the decorator runs, because tracing would silently take the wrong branch otherwise:

  • case None/True/False and is compare by identity. Use is_null(x).
  • List and dict patterns aren't supported. Neither are class patterns other than Ok/Err.
  • A bare exception name inside Err(...) captures every error instead of matching that exception. Write the dotted name or a string.
  • try/except isn't supported; use attempt. Neither is while.
  • return must be the last statement of the workflow.

Not supported in a workflow body: arithmetic on refs, len() (use length(x)), and iterating a ref (use .map/each). Do those in a step.

IAM

Every step's role gets three layers:

  1. The runtime baseline. This is added automatically: its own logs, plus read and write on the artifacts bucket under runs/.
  2. The workflow's base set, from @workflow(iam=base) or subworkflow(..., iam=base). Steps with no iam= inherit the nearest one.
  3. The step's own iam=, which replaces the base set.
    • Write iam=base + Allow(...) to keep the base set and add to it.
    • Write iam=Policy(Allow(...)) for granular permissions with none of the base set.

Policy objects compose with +, which takes their union and removes duplicates. A step that ends up under two different policies is deployed as two functions. A reusable @workflow uses its own iam=. Resources may be CloudFormation intrinsics, for example {"Fn::Sub": "arn:${AWS::Partition}:s3:::${ArtifactsBucket}/fixtures/*"}.

Data between steps

Step arguments and results must be JSON-serializable. Step Functions caps each state's input and output at 256 KiB, so any value over offload_threshold (default 32 KiB) goes to the artifacts bucket. Step code only ever sees plain values.

A condition is evaluated inside Step Functions, which can't read S3. So the step that produces a value also returns, inline, the specific fields conditions read (facts). Conditions can therefore branch on a value even when the rest of it went to S3.

The wire format is in src/sfnkit/runtime/protocol.py.

Dependencies and images

A step's image is built from the workflow's deps + the step's deps (+ system_packages, installed with microdnf), on public.ecr.aws/lambda/python:<python>. Steps with the same set share an image. Tags are content hashes, so an unchanged image isn't rebuilt or pushed again. Build contexts go in <project>/.sfnkit/build/.

Deploying

deploy creates two stacks:

  • <stack>-assets: the ECR repository and the artifacts bucket.
  • <stack>: the functions, their roles, log groups and state machines.

destroy deletes both, emptying the bucket first. The deploying credentials must be allowed to create IAM roles. Credentials from sts get-session-token can't call IAM without MFA, so use long-term or assumed-role credentials.

Adding a compute backend

Every step runs through one interface, sfnkit.compute.base.ComputeBackend. To add one:

  1. Subclass ComputeSpec for the user-facing settings, e.g. Fargate(cpu=..., memory=...).
  2. Implement four methods on the backend:
    • resources(fn, ctx): the CloudFormation resources that run it.
    • baseline_statements(fn, ctx): IAM its own role needs.
    • task_state(fn): the Task state. Put PAYLOAD where the envelope goes.
    • state_machine_statements(fn): IAM the state machine needs to start it.
  3. Register it with register_backend or the sfnkit.compute entry point.

The compiler, CloudFormation assembly and deployer only call this interface. Containers can run the same image with the entrypoint python -m sfnkit.runtime.container, which reads the envelope and task token from the environment and reports the result with SendTaskSuccess/SendTaskFailure. tests/core/test_compile.py registers a fake ECS backend to prove nothing else needs to change.

Development

Work happens in the isb sandbox described by isb.yaml:

isb up -d
scripts/sandbox.sh -- 'uv sync && uv run pytest'                    # unit tests, no AWS
scripts/sandbox.sh -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION \
  -- 'uv run pytest -m aws'                                            # validate ASL with AWS
scripts/sandbox.sh -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY -e AWS_DEFAULT_REGION \
  -- 'PYTHONPATH=. uv run pytest -m e2e -s'                            # deploy, run, destroy
  • Golden ASL lives in tests/core/golden/. Refresh it with SFNKIT_UPDATE_GOLDEN=1 and review the diff.
  • The e2e test deploys examples/pipeline.py under a random sfnkit-e2e-* stack name, runs every path, and always tears down.

Metadata

Release files for sfnkit 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for sfnkit 0.1.0
File Size Uploaded Provenance Actions
sfnkit-0.1.0.tar.gz 153.5 kB Without provenance Details for sfnkit-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for sfnkit 0.1.0
File Interpreter ABI Platform Actions
sfnkit-0.1.0-py3-none-any.whl Python 3 none any Details for sfnkit-0.1.0-py3-none-any.whl

Total release size: 235.5 kB

Release files / sfnkit-0.1.0.tar.gz

Download URL sfnkit-0.1.0.tar.gz
Size 153.5 kB
Tags Source
SHA-256 checksum
How to use checksums
adafdfb9267a07b870bb4765e9b39c2d79ccbf59319f725a821f979522cd3044
BLAKE2b-256 checksum
How to use checksums
f68ccf62c95d65c78a07312c05a8a21c0ff24f68ffa37fe267033db1bc3fb07c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / sfnkit-0.1.0-py3-none-any.whl

Download URL sfnkit-0.1.0-py3-none-any.whl
Size 82.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
50d2c74bfb6b2493285bfe682723d57895920fc9c9fc853ca6a9311566896f50
BLAKE2b-256 checksum
How to use checksums
788ebdd0394bdcedf740effaf37f2d60312cd09d6d97794b6a1f997431c7eb48
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.4 {"installer":{"name":"uv","version":"0.13.4","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"26.04","id":"resolute","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for sfnkit 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page