py-yonder 0.2.6

Install

# core package
pip install py-yonder

# with all extras (DockerRunner, K8sRunner, ApiRunner, etc.)
pip install py-yonder[all]

Quick start

import yonder
from yonder import DockerRunner, missing

# Existing image#
runnerA = DockerRunner(
    image="url/to/image",
    env={"ENV_VARA": "value"},
    workspaces={"/host/path": "/in/container"},
    when=missing("liba"),  # dispatch only when host lacks `liba`, otherswise run locally
)

# Build from a Dockerfile (context defaults to the Dockerfile's directory).
runnerB = DockerRunner(
    dockerfile="./docker/myimg.Dockerfile",
    build_args={"VERSION": "1.2.3"},
)

# On-demand: fresh container per call instead of one long-lived container.
ephemeral = DockerRunner(image="url/to/image", on_demand=True)

# Attach to a container you manage outside yonder. yonder will start it
# if it's stopped, but never removes it on close.
external = DockerRunner(container="my-dev-container")

# Register them by name.
yonder.register({
    "runnerA": runnerA,
    "runnerB": runnerB,
    "ephemeral": ephemeral,
    "external": external,
})

# Decorate. Dispatch policy lives on the runner (via the runner's
# `when=`); the decorator just names the target.
@yonder.to("runnerA")
def functionA(a, b):
    return a + b

@yonder.to("runnerB")
def functionB(a, b):
    return a * b

runnerA.start()       # pre-warm a specific runner
yonder.wait()         # pre-warm every registered runner (parallel)

runnerA.close()       # tear down a specific runner
yonder.closeAll()     # tear down every registered runner (parallel)

DockerRunner options

DockerRunner(
    # Exactly one of these three is required:
    image=None,          # use/pull an existing image; yonder creates the container
    dockerfile=None,     # build an image locally first; yonder creates the container
    container=None,      # attach to an existing container (managed outside yonder)

    build_context=None,  # override the build context directory (dockerfile mode)
    build_args=None,     # dict of Docker ARGs (dockerfile mode)

    env=None,            # environment variables (image/dockerfile mode only)
    ports=None,          # publish container ports to the host, like
                         # `docker run -p`. Orientation is container:host.
                         #   8080                       # host 8080 -> container 8080
                         #   "8080:3333"                # host 3333 -> container 8080
                         #   [8080, "9000:9001/udp"]    # list; per-port protocol
                         #   {8080: 3333}               # dict, {container: host}
                         #   {8080: None}               # ephemeral host port
                         #   {8080: ("127.0.0.1", 3333)}# bind a host IP
                         # image/dockerfile mode only (rejected with container=).
                         # Read assigned ports back via .published_ports.
    workspaces=None,      # declarative file sync. Accepts a list, a dict,
                         # or a mix:
                         #   ["host/path/a", "host/path/b"]       # list-form, auto-mapped
                         #   {host: runner_path}                  # dict, explicit runner path
                         #   {host: {to?, dir?, include?,         # dict, full opts ("to" optional;
                         #           exclude?, name?}}            #   omit it to auto-map)
                         #   [host, {host: {dir: "up"}}, ...]     # list of strings + dicts
                         # In image=/dockerfile= mode, simple entries
                         # auto-promote to bind mounts.
    sync_deletes=False,  # propagate deletions across sync (default: False)
    on_demand=False,     # one-shot container per call (image/dockerfile mode only)
    inject_cloudpickle=False,  # derive `FROM image + RUN pip install cloudpickle`
                               # (image mode only; tagged by base-image hash so the
                               # docker layer cache makes re-runs cheap)
    mode="reference",    # transport: "cloudpickle" or "reference"
    source_path=None,    # host path(s) made importable in the container
                         # (mounted for image=/dockerfile=, tarred and shipped
                         # for container=). In mode="reference", None
                         # defaults to the caller's __file__ dir; pass []
                         # to opt out
    source_include=(".py", ".yaml", ".yml", ".json"),
                         # extension allowlist for shipped tar (applies to
                         # container= mode); None = ship every file
    client=None,         # pre-built docker.DockerClient (else docker.from_env())
)

K8sRunner

Run inside a Kubernetes pod. Three modes — yonder either creates the pod from an image, or attaches to one you already manage.

from yonder import K8sRunner

# Create a pod from an image. Yonder owns it: close() deletes it.
owned = K8sRunner(
    image="myorg/worker:1.2.3",
    namespace="default",
    env={"LOG_LEVEL": "INFO"},
    image_pull_secrets=["my-registry-secret"],  # private registries
)

# Attach by pod name.
by_name = K8sRunner(
    pod="my-pod-abc123",
    namespace="default",
    container="app",  # optional; defaults to the pod's first container
)

# Attach via label selector — first Running pod that matches is used.
by_selector = K8sRunner(
    selector="app=my-app,env=dev",
    namespace="default",
)

For anything beyond what image= exposes (resources, service account, tolerations, custom volumes, sidecars, …) deploy the pod with your own tooling and attach via pod= / selector=.

K8sRunner(
    # Exactly one of these three is required:
    pod=None,            # attach by pod name (yonder doesn't manage lifecycle)
    selector=None,       # attach by label selector (yonder doesn't manage lifecycle)
    image=None,          # create a pod from this image (yonder owns its lifecycle)

    namespace="default", # pod namespace
    container=None,      # container within the pod (default: first)
    kubeconfig=None,     # kubeconfig path (else KUBECONFIG/~/.kube/config,
                         # falling back to in-cluster config)
    context=None,        # kubeconfig context
    name=None,           # runner name (auto-derived if omitted)
    mode="cloudpickle",  # transport: "cloudpickle" or "reference"
    source_path=None,    # host path(s) tarred and shipped into the pod
                         # under /tmp/yonder-src/<sha>/ (cached by digest).
                         # In mode="reference", None defaults to the
                         # caller's __file__ dir; pass [] to opt out
    source_include=(".py", ".yaml", ".yml", ".json"),
                         # extension allowlist for shipped tar;
                         # None = ship every file
    ports=None,          # reach pod ports on localhost via port-forward
                         # (the kubectl port-forward equivalent). Same forms
                         # as DockerRunner; orientation is pod_port:local_port.
                         # tcp only. Works in every mode (image/pod/selector).
                         #   {8080: None}  # OS-chosen local port
                         # Read assigned ports back via .forwarded_ports.

    # image= only:
    env=None,                  # dict of env vars to set on the container
    image_pull_secrets=None,   # list of existing Secret names
    startup_timeout=60,        # seconds to wait for pod to reach Running

    api_client=None,     # pre-built kubernetes.client.ApiClient
)

The target container must have python on PATH, plus whatever the runner’s transport mode needs — see Container requirements below.

Architecture

• Runner (yonder.Runner) — abstract base. Implementations hold their own config and expose run(payload) -> bytes, plus optional start() / close() lifecycle hooks.

• DockerRunner — default implementation; uses the Docker SDK for Python. Supports existing images, Dockerfile builds, and attaching to externally-managed containers; persistent and on-demand modes.

• K8sRunner — Kubernetes implementation; uses the Kubernetes Python client to exec into a pod selected by name or label selector.

• YonderSession — process-wide name→runner registry. Populate with yonder.register({...}), bring everything online with yonder.wait(), tear everything down with yonder.closeAll().

• Decorator — serializes (func, args, kwargs) with cloudpickle, calls the named runner’s run(payload), deserializes the envelope, and either returns the value or re-raises the exception.

Writing your own Runner

from yonder import Runner, register

class MyRunner(Runner):
    def __init__(self, **config):
        ...

    def start(self):
        # eager init (build/pull image, warm container, etc.)
        ...

    def run(self, payload: bytes) -> bytes:
        # send payload to your execution environment,
        # return the cloudpickled envelope
        ...

    def close(self):
        ...

register({"mine": MyRunner(...)})

Container requirements

Every runner shells out to python inside the target environment, so the image, container, or pod must have python on PATH. Beyond that, the requirements depend on which transport mode the runner uses to ship your function across the wire — not on which runner you’re using. Both DockerRunner and K8sRunner accept mode="cloudpickle" (default) or mode="reference" and have identical environment requirements for each.