run-yonder 0.1.0

Run Python functions inside containers via a decorator.

yonder

Run Python functions on container-backed runners via a decorator. Functions and their arguments are serialized with cloudpickle, shipped to the runner, executed there, and the result is shipped back.

Install

# core package
pip install -e .

# with the bundled Docker runner
pip install -e ".[docker]"

# with the bundled Kubernetes runner
pip install -e ".[k8s]"

Quick start

import yonder
from yonder import DockerRunner

# Existing image. If the image doesn't already have cloudpickle, set
# inject_cloudpickle=True to derive an image that does (cached across
# runs, keyed by the base image). `when=` is an optional zero-arg
# predicate that makes the runner a no-op (calls run locally) when it
# returns False — useful for "use the container only if the library
# isn't already installed on the host".
runnerA = DockerRunner(
    image="url/to/image",
    env={"ENV_VARA": "value"},
    mounts={"/host/path": "/in/container"},
    inject_cloudpickle=True,
    when=lambda: liba_not_present(),
)

# Build from a Dockerfile (context defaults to the Dockerfile's directory).
runnerB = DockerRunner(
    dockerfile="./docker/myimg.Dockerfile",
    build_args={"VERSION": "1.2.3"},
)

# On-demand: fresh container per call instead of one long-lived container.
ephemeral = DockerRunner(image="url/to/image", on_demand=True)

# Attach to a container you manage outside yonder. yonder will start it
# if it's stopped, but never removes it on close.
external = DockerRunner(container="my-dev-container")

# Register them by name.
yonder.register({
    "runnerA": runnerA,
    "runnerB": runnerB,
    "ephemeral": ephemeral,
    "external": external,
})

# Decorate. Dispatch policy lives on the runner (via the runner's
# `when=`); the decorator just names the target.
@yonder.yonder(runner="runnerA")
def functionA(a, b):
    return a + b

@yonder.yonder(runner="runnerB")
def functionB(a, b):
    return a * b

Lazy start by default

A registered runner is not built/started until it’s actually needed. The first decorated call that targets a runner triggers its start() (image build/pull, persistent container creation). Runners that are never called incur no cost.

You can pre-warm explicitly when you’d rather pay startup cost up front:

runnerA.start()       # pre-warm a specific runner
yonder.wait()         # pre-warm every registered runner (parallel)

Both are opt-in. On-demand runners always behave the same way — they spin up a fresh container per call regardless of pre-warming.

Tear down

Tear down when done:

yonder.closeAll()
# or, per-runner:
runnerA.close()

DockerRunner is also a context manager:

with DockerRunner(image="python:3.11-slim") as r:
    yonder.register({"tmp": r})
    yonder.wait()
    ...
# container removed on exit

DockerRunner options

DockerRunner(
    # Exactly one of these three is required:
    image=None,          # use/pull an existing image; yonder creates the container
    dockerfile=None,     # build an image locally first; yonder creates the container
    container=None,      # attach to an existing container (managed outside yonder)

    build_context=None,  # override the build context directory (dockerfile mode)
    build_args=None,     # dict of Docker ARGs (dockerfile mode)

    env=None,            # environment variables (image/dockerfile mode only)
    mounts=None,         # {host_path: container_path} (image/dockerfile mode only)
    on_demand=False,     # one-shot container per call (image/dockerfile mode only)
    name=None,           # container/tag name (auto-generated if omitted)
    inject_cloudpickle=False,  # derive `FROM image + RUN pip install cloudpickle`
                               # (image mode only; tagged by base-image hash so the
                               # docker layer cache makes re-runs cheap)
    client=None,         # pre-built docker.DockerClient (else docker.from_env())
)

K8sRunner

Attach to an existing Kubernetes pod and exec into one of its containers. Pods are treated like externally-managed containers — yonder never creates or removes them.

from yonder import K8sRunner

# By pod name.
by_name = K8sRunner(
    pod="my-pod-abc123",
    namespace="default",
    container="app",  # optional; defaults to the pod's first container
)

# By label selector — first Running pod that matches is used.
by_selector = K8sRunner(
    selector="app=my-app,env=dev",
    namespace="default",
)

K8sRunner(
    # Exactly one of these two is required:
    pod=None,            # attach by pod name
    selector=None,       # attach by label selector

    namespace="default", # pod namespace
    container=None,      # container within the pod (default: first)
    kubeconfig=None,     # kubeconfig path (else KUBECONFIG/~/.kube/config,
                         # falling back to in-cluster config)
    context=None,        # kubeconfig context
    name=None,           # runner name (auto-derived if omitted)
    api_client=None,     # pre-built kubernetes.client.ApiClient
)

The target container must have python on PATH and cloudpickle installed (same Python minor version as the host is strongly recommended).

Architecture

• ``Runner`` (yonder.Runner) — abstract base. Implementations hold their own config and expose run(payload) -> bytes, plus optional start() / close() lifecycle hooks.

• ``DockerRunner`` — default implementation; uses the Docker SDK for Python. Supports existing images, Dockerfile builds, and attaching to externally-managed containers; persistent and on-demand modes.

• ``K8sRunner`` — Kubernetes implementation; uses the Kubernetes Python client to exec into a pod selected by name or label selector.

• ``YonderSession`` — process-wide name→runner registry. Populate with yonder.register({...}), bring everything online with yonder.wait(), tear everything down with yonder.closeAll().

• Decorator — serializes (func, args, kwargs) with cloudpickle, calls the named runner’s run(payload), deserializes the envelope, and either returns the value or re-raises the exception.

Writing your own Runner

from yonder import Runner, register

class MyRunner(Runner):
    def __init__(self, **config):
        ...

    def start(self):
        # eager init (build/pull image, warm container, etc.)
        ...

    def run(self, payload: bytes) -> bytes:
        # send payload to your execution environment,
        # return the cloudpickled envelope
        ...

    def close(self):
        ...

register({"mine": MyRunner(...)})

Container requirements

Images used with DockerRunner must have:

• python on PATH

• cloudpickle installed (pip install cloudpickle). If your base image doesn’t have it, pass inject_cloudpickle=True and yonder will build a derived image that does (one-time, cached).

• Ideally the same Python minor version as the host (cloudpickle pickles function bytecode, which is not portable across CPython minor versions).