py-yonder 0.1.10

Run Python functions inside containers via a decorator.

yonder

Run Python functions on container-backed runners via a decorator. Functions and their arguments are serialized with cloudpickle, shipped to the runner, executed there, and the result is shipped back.

Install

# core package
pip install py-yonder

# with the bundled Docker runner
pip install py-yonder[docker]

# with the bundled Kubernetes runner
pip install py-yonder[k8s]

# with all extras
pip install py-yonder[all]

Quick start

import yonder
from yonder import DockerRunner, missing

# Existing image. If the image doesn't already have cloudpickle, set
# inject_cloudpickle=True to derive an image that does (cached across
# runs, keyed by the base image). `when=` is an optional zero-arg
# predicate that makes the runner a no-op (calls run locally) when it
# returns False. The bundled `missing("pkg")` predicate fits the
# common case "use the container only when the host can't import the
# package locally".
runnerA = DockerRunner(
    image="url/to/image",
    env={"ENV_VARA": "value"},
    # Declare host <-> container file sync. In image=/dockerfile=
    # mode an unfiltered entry is promoted to a zero-copy bind
    # mount at container creation; adding include=/exclude= globs
    # or sync_deletes=True keeps the entry on the snapshotting
    # sync path instead.
    workspace={"/host/path": "/in/container"},
    inject_cloudpickle=True,
    when=missing("liba"),  # dispatch only when host lacks `liba`
)

# Build from a Dockerfile (context defaults to the Dockerfile's directory).
runnerB = DockerRunner(
    dockerfile="./docker/myimg.Dockerfile",
    build_args={"VERSION": "1.2.3"},
)

# On-demand: fresh container per call instead of one long-lived container.
ephemeral = DockerRunner(image="url/to/image", on_demand=True)

# Attach to a container you manage outside yonder. yonder will start it
# if it's stopped, but never removes it on close.
external = DockerRunner(container="my-dev-container")

# Register them by name.
yonder.register({
    "runnerA": runnerA,
    "runnerB": runnerB,
    "ephemeral": ephemeral,
    "external": external,
})

# Decorate. Dispatch policy lives on the runner (via the runner's
# `when=`); the decorator just names the target.
@yonder.to("runnerA")
def functionA(a, b):
    return a + b

@yonder.to("runnerB")
def functionB(a, b):
    return a * b

Lazy start by default

A registered runner is not built/started until it’s actually needed. The first decorated call that targets a runner triggers its start() (image build/pull, persistent container creation). Runners that are never called incur no cost.

You can pre-warm explicitly when you’d rather pay startup cost up front:

runnerA.start()       # pre-warm a specific runner
yonder.wait()         # pre-warm every registered runner (parallel)

Both are opt-in. On-demand runners always behave the same way — they spin up a fresh container per call regardless of pre-warming.

Tear down

Tear down when done:

yonder.closeAll()
# or, per-runner:
runnerA.close()

DockerRunner is also a context manager:

with DockerRunner(image="python:3.11-slim") as r:
    yonder.register({"tmp": r})
    yonder.wait()
    ...
# container removed on exit

DockerRunner options

DockerRunner(
    # Exactly one of these three is required:
    image=None,          # use/pull an existing image; yonder creates the container
    dockerfile=None,     # build an image locally first; yonder creates the container
    container=None,      # attach to an existing container (managed outside yonder)

    build_context=None,  # override the build context directory (dockerfile mode)
    build_args=None,     # dict of Docker ARGs (dockerfile mode)

    env=None,            # environment variables (image/dockerfile mode only)
    workspace=None,      # {host_path: runner_path | {to, dir, include, exclude}}
                         # — declarative file sync. In image=/dockerfile=
                         # mode, simple entries auto-promote to bind mounts.
    sync_deletes=False,  # propagate deletions across sync (default: False)
    on_demand=False,     # one-shot container per call (image/dockerfile mode only)
    name=None,           # container/tag name (auto-generated if omitted)
    inject_cloudpickle=False,  # derive `FROM image + RUN pip install cloudpickle`
                               # (image mode only; tagged by base-image hash so the
                               # docker layer cache makes re-runs cheap)
    mode="cloudpickle",  # transport: "cloudpickle" or "reference"
    source_path=None,    # host path(s) made importable in the container
                         # (mounted for image=/dockerfile=, tarred and shipped
                         # for container=). In mode="reference", None
                         # defaults to the caller's __file__ dir; pass []
                         # to opt out
    source_include=(".py", ".yaml", ".yml", ".json"),
                         # extension allowlist for shipped tar (applies to
                         # container= mode); None = ship every file
    client=None,         # pre-built docker.DockerClient (else docker.from_env())
)

K8sRunner

Run inside a Kubernetes pod. Three modes — yonder either creates the pod from an image, or attaches to one you already manage.

from yonder import K8sRunner

# Create a pod from an image. Yonder owns it: close() deletes it.
owned = K8sRunner(
    image="myorg/worker:1.2.3",
    namespace="default",
    env={"LOG_LEVEL": "INFO"},
    image_pull_secrets=["my-registry-secret"],  # private registries
)

# Attach by pod name.
by_name = K8sRunner(
    pod="my-pod-abc123",
    namespace="default",
    container="app",  # optional; defaults to the pod's first container
)

# Attach via label selector — first Running pod that matches is used.
by_selector = K8sRunner(
    selector="app=my-app,env=dev",
    namespace="default",
)

For anything beyond what image= exposes (resources, service account, tolerations, custom volumes, sidecars, …) deploy the pod with your own tooling and attach via pod= / selector=.

K8sRunner(
    # Exactly one of these three is required:
    pod=None,            # attach by pod name (yonder doesn't manage lifecycle)
    selector=None,       # attach by label selector (yonder doesn't manage lifecycle)
    image=None,          # create a pod from this image (yonder owns its lifecycle)

    namespace="default", # pod namespace
    container=None,      # container within the pod (default: first)
    kubeconfig=None,     # kubeconfig path (else KUBECONFIG/~/.kube/config,
                         # falling back to in-cluster config)
    context=None,        # kubeconfig context
    name=None,           # runner name (auto-derived if omitted)
    mode="cloudpickle",  # transport: "cloudpickle" or "reference"
    source_path=None,    # host path(s) tarred and shipped into the pod
                         # under /tmp/yonder-src/<sha>/ (cached by digest).
                         # In mode="reference", None defaults to the
                         # caller's __file__ dir; pass [] to opt out
    source_include=(".py", ".yaml", ".yml", ".json"),
                         # extension allowlist for shipped tar;
                         # None = ship every file

    # image= only:
    env=None,                  # dict of env vars to set on the container
    image_pull_secrets=None,   # list of existing Secret names
    startup_timeout=60,        # seconds to wait for pod to reach Running

    api_client=None,     # pre-built kubernetes.client.ApiClient
)

The target container must have python on PATH, plus whatever the runner’s transport mode needs — see Container requirements below.

Architecture

• Runner (yonder.Runner) — abstract base. Implementations hold their own config and expose run(payload) -> bytes, plus optional start() / close() lifecycle hooks.

• DockerRunner — default implementation; uses the Docker SDK for Python. Supports existing images, Dockerfile builds, and attaching to externally-managed containers; persistent and on-demand modes.

• K8sRunner — Kubernetes implementation; uses the Kubernetes Python client to exec into a pod selected by name or label selector.

• YonderSession — process-wide name→runner registry. Populate with yonder.register({...}), bring everything online with yonder.wait(), tear everything down with yonder.closeAll().

• Decorator — serializes (func, args, kwargs) with cloudpickle, calls the named runner’s run(payload), deserializes the envelope, and either returns the value or re-raises the exception.

Writing your own Runner

from yonder import Runner, register

class MyRunner(Runner):
    def __init__(self, **config):
        ...

    def start(self):
        # eager init (build/pull image, warm container, etc.)
        ...

    def run(self, payload: bytes) -> bytes:
        # send payload to your execution environment,
        # return the cloudpickled envelope
        ...

    def close(self):
        ...

register({"mine": MyRunner(...)})

Container requirements

Every runner shells out to python inside the target environment, so the image, container, or pod must have python on PATH. Beyond that, the requirements depend on which transport mode the runner uses to ship your function across the wire — not on which runner you’re using. Both DockerRunner and K8sRunner accept mode="cloudpickle" (default) or mode="reference" and have identical environment requirements for each.

Cloudpickle mode

mode="cloudpickle" (the default) serializes the function as bytecode with cloudpickle and unpickles it on the other side. Works for closures, lambdas, locally-defined functions, and code typed at a REPL — nothing needs to be importable in the container.

The target environment must have:

• cloudpickle installed (pip install cloudpickle).

• The same Python minor version as the host (e.g. host 3.11 ↔ container 3.11). CPython bytecode is not portable across minor versions; yonder raises a clear error on mismatch.

If a base image doesn’t ship with cloudpickle, DockerRunner can inject it for you — pass inject_cloudpickle=True and yonder will build a tiny derived image (FROM <your-image> + RUN pip install cloudpickle) and cache it locally. Injection isn’t available when attaching to an existing container (container=) or when building from a Dockerfile — add RUN pip install cloudpickle to the Dockerfile yourself in that case. K8sRunner never modifies the pod, so cloudpickle must already be present in the target container.

Reference mode

mode="reference" only ships (module, qualname, args, kwargs). The container imports its own copy of the function and calls it — bytecode never crosses the wire, so host and container Python versions may differ, and cloudpickle is not required in the container.

The target environment must have:

• The function’s module importable inside the container. The simplest way is to pass source_path= to the runner — yonder will get the source tree there for you. On DockerRunner with image= / dockerfile=, that’s a bind mount at /workspace/<basename>. On DockerRunner with container= or K8sRunner, the tree is tarred and shipped to /tmp/yonder-src/<sha>/ on the first call (cached by content digest, so an unchanged tree only ships once per runner). Either mechanism also helps cloudpickle mode if the function does import helpers against a sibling.

Reference mode requires the function to be defined at module scope in a non-__main__ module; closures, lambdas, and REPL-defined functions won’t work.

Shipping requires sh and tar on the target’s PATH; distroless / minimal images that strip them are incompatible. Pre-bake the source or use a different base image.