icefold-runner

A self-hosted execution runner for IceFold nodes — like a GitHub self-hosted CI runner. You start it on your own machine; it reverse-connects to an IceFold server (so it works behind NAT with no inbound ports, no public IP, no tunnel), receives node-execution jobs, runs them locally, and streams results back.

It is the place where your uploaded node code runs — on your hardware, with full subprocess / ffmpeg / GPU / any-dependency access. The IceFold server never executes third-party code; it only renders it into bundles for a runner.

How it works

   your machine (private, behind NAT)              IceFold server (public)
 ┌──────────────────────────────────┐  reverse WSS  ┌───────────────────────────┐
 │ icefold-runner                    │ ───────────► │ /v1/ws/worker?worker_id=…   │
 │  • dials out, token auth          │  node_exec ◄─│ routes node runs (per user) │
 │  • reconnect + keepalive          │  node_done ─►│                             │
 │  • bundle runner:                 │              │                             │
 │    GET /v1/bundles/<hash>         │   HTTP pull  │ /files  /scratch            │
 │    import bundle + preflight deps │ ◄──────────► │ /v1/workers/output          │
 │    await __icefold_run__          │              │                             │
 └──────────────────────────────────┘   HTTP push  └───────────────────────────┘

• Control plane rides the reverse WebSocket (node_exec / cancel → node_status / node_done / missing_dep) as plain JSON text frames — TLS (wss) is the confidentiality layer. The runner token travels in the X-Worker-Token request header; only the non-secret worker_id is in the query string. Each node_exec frame carries a bundle_hash, the selected variant's inputs, and the confirmed variant input storage needed by the bundle's typed runtime — never node source.
• Bulk media + bundles ride plain HTTP: the runner GETs inputs from the server's /files and /scratch mounts and node bundles from /v1/bundles/<hash> (sha256-addressed, cached locally as runner_work_dir/bundles/<hash>.py, re-hashed on every download), runs the bundle, POSTs products back to /v1/workers/output (which returns server-canonical paths and accepts at most 2 GiB per product).
• Media inputs are cached across calls. Signed query strings rotate, so the cache key uses the server origin, authenticated owner, and immutable canonical /files or /scratch artifact path. A cold call downloads distinct inputs concurrently and atomically publishes each blob; concurrent calls requesting the same blob share one transfer. Warm calls use the local file without a network request. Active inputs carry leases and cannot be evicted while node code or a subprocess is using them. After a product upload succeeds, the returned canonical path is atomically linked to that local product, so a downstream node on the same runner does not download it back from the server. Age plus LRU byte ceilings bound disk use.
• The runner ships no node implementations and never compiles user source. The IceFold server renders every node (your custom ones and the platform's built-in ones) into a self-contained .py bundle, with python_deps / binary_deps declared in the bundle header. The runner imports the bundle, pre-flights the deps (sending back a structured missing_dep reply with platform-aware install hints if anything is absent), and awaits __icefold_run__(inputs, ctx_dict). So when the server adds or upgrades nodes, the runner does not need an upgrade. Runner protocol or agent changes are released as a new runner version.
• Variant planning / dimension & provider resolution all stay on the server; each job is a single already-sliced leaf call.
• Transient WebSocket loss does not restart a node. Recovery-capable server and runner versions negotiate this in worker_ready: the runner keeps the task alive, the server reattaches its pending call_id, and terminal frames plus host callbacks replay by id. If the runner process itself is lost, the server automatically resubmits the same call_id to a replacement runner. A stable worker_id owns scheduling/cache affinity, while a random per-process instance_id ensures a restarted process can never impersonate a socket reconnect and inherit coroutines that died with its predecessor. Older peers do not advertise recovery, so rolling upgrades retain the legacy cancel-on-disconnect behaviour instead of guessing across protocol versions.
• Pre-execution transfer failures can move once to a different runner. The runner retries transient input and bundle downloads locally first. If those attempts are exhausted before node code starts, runner 0.2.7+ marks the terminal result as safe to retry; a compatible server may then exclude that runner and dispatch one fresh call to a peer. Structured dependency-preflight failures are safe for the same reason. Node, provider, timeout, upload, and other post-start failures never carry this hint.

Install

Requires only Python ≥ 3.11 (it pulls in icefold-sdk).

The runner itself ships no node tooling. A node declares what it needs in its bundle header — binary_deps (for example, IceFold video nodes use google-chrome, ffmpeg, and ffprobe on PATH) and python_deps (whatever your custom nodes import) — and the runner pre-flights those before each run, replying with a platform-aware install hint (missing_dep) for anything absent. So you install a node's dependencies only when you actually run a job that needs them, and the runner tells you exactly what to install.

pip install icefold-runner          # pulls in icefold-sdk

From source:

git clone <this-repo> icefold-runner
cd icefold-runner
python -m venv .venv && . .venv/bin/activate
pip install -e .

Run

Generate a token in the IceFold app (Settings → Runners), then:

install -m 600 /dev/null ~/.icefold-runner-token
read -rsp 'Runner token: ' ICEFOLD_TOKEN_INPUT
printf '%s' "$ICEFOLD_TOKEN_INPUT" > ~/.icefold-runner-token
unset ICEFOLD_TOKEN_INPUT
icefold-runner --token-file ~/.icefold-runner-token

That's it — the token (GitHub-CI style) encodes + signs your IceFold user id, so there's no server URL or user id to pass. The server is built in.

Every flag also reads an env var:

The runner honors standard proxy env vars (HTTPS_PROXY, ALL_PROXY, …) for reaching the server, including HTTP and SOCKS proxies. It reconnects automatically with backoff; an auth rejection is fatal.

The runner advertises both effective lane widths in its WebSocket hello frame. Servers that understand these fields can weight stable machine ownership and bound each runner's local CPU/GPU queues. Session work stays on its cache-warm owner when those queues fill; a designated backup is used only when no primary is healthy. Older servers safely ignore the extra fields. A newer server treats an older runner that does not advertise capacity as one slot.

Self-hosting / dev: point the runner at a different server with the ICEFOLD_RUNNER_SERVER env var (e.g. ws://127.0.0.1:7000).

Layout

icefold_runner/      the runner agent (connection, input cache, bundle exec)
  client.py            reverse-WS client: dial / auth / reconnect / keepalive
  runner.py            fetch /v1/bundles/<hash>, preflight deps, await __icefold_run__
  __main__.py          CLI entrypoint (icefold-runner)

The runner imports the bundle on demand; the bundle is self-contained and already inlines whatever it needs (the author's function body, the Inputs / Output dataclasses, and a minimal NodeContext shim). The only runtime dependency on icefold-sdk is the wire protocol + a small helper kit (get_file_id / run_blocking / write_text), used by the runner agent itself, not by node code.

Security model

• Node code runs unsandboxed here — it's your machine, your risk. That's the point: code the server refuses to execute (subprocess/ffmpeg/native deps and anything third-party) runs on the runner instead. The runner downloads each bundle from the server and executes it; it verifies the bundle's sha256 matches the requested hash, but the bundle itself is whatever the server you authenticated to sends. Only point a runner at a server you trust.
• The runner only talks to the one server you point it at, authenticated by its runner token; it pulls input files and pushes products over HTTP to that host.