icefold-runner

A self-hosted execution runner for IceFold nodes — like a GitHub self-hosted CI runner. You start it on your own machine; it reverse-connects to an IceFold server (so it works behind NAT with no inbound ports, no public IP, no tunnel), receives node-execution jobs, runs them locally, and streams results back.

It is the place where your uploaded node code runs — on your hardware, with full subprocess / ffmpeg / GPU / any-dependency access. The IceFold server never executes third-party code; it only renders it into bundles for a runner.

How it works

   your machine (private, behind NAT)              IceFold server (public)
 ┌──────────────────────────────────┐  reverse WSS  ┌───────────────────────────┐
 │ icefold-runner                    │ ───────────► │ /v1/ws/worker?worker_id=…   │
 │  • dials out, token auth          │  node_exec ◄─│ routes node runs (per user) │
 │  • reconnect + keepalive          │  node_done ─►│                             │
 │  • bundle runner:                 │              │                             │
 │    GET /v1/bundles/<hash>         │   HTTP pull  │ /files  /scratch            │
 │    import bundle + preflight deps │ ◄──────────► │ /v1/workers/output          │
 │    await __icefold_run__          │              │                             │
 └──────────────────────────────────┘   HTTP push  └───────────────────────────┘

• Control plane rides the reverse WebSocket (node_exec / cancel → node_status / node_done / missing_dep) as plain JSON text frames — TLS (wss) is the confidentiality layer. The runner token travels in the X-Worker-Token request header; only the non-secret worker_id is in the query string. Each node_exec frame carries a bundle_hash, the selected variant's inputs, and the confirmed variant input storage needed by the bundle's typed runtime — never node source.
• Bulk media + bundles ride plain HTTP: the runner GETs inputs from the server's /files and /scratch mounts and node bundles from /v1/bundles/<hash> (sha256-addressed, cached locally as runner_work_dir/bundles/<hash>.py, re-hashed on every download), runs the bundle, POSTs products back to /v1/workers/output (which returns server-canonical paths and accepts at most 2 GiB per product).
• The runner ships no node implementations and never compiles user source. The IceFold server renders every node (your custom ones and the platform's built-in ones) into a self-contained .py bundle, with python_deps / binary_deps declared in the bundle header. The runner imports the bundle, pre-flights the deps (sending back a structured missing_dep reply with platform-aware install hints if anything is absent), and awaits __icefold_run__(inputs, ctx_dict). So when the server adds or upgrades nodes, the runner does not need an upgrade. Runner protocol or agent changes are released as a new runner version.
• Variant planning / dimension & provider resolution all stay on the server; each job is a single already-sliced leaf call.
• Transient WebSocket loss does not restart a node. Recovery-capable server and runner versions negotiate this in worker_ready: the runner keeps the task alive, the server reattaches its pending call_id, and terminal frames plus host callbacks replay by id. If the runner process itself is lost, the server automatically resubmits the same call_id to a replacement runner. Older peers do not advertise recovery, so rolling upgrades retain the legacy cancel-on-disconnect behaviour instead of guessing across protocol versions.
• Pre-execution transfer failures can move once to a different runner. The runner retries transient input and bundle downloads locally first. If those attempts are exhausted before node code starts, runner 0.2.7+ marks the terminal result as safe to retry; a compatible server may then exclude that runner and dispatch one fresh call to a peer. Structured dependency-preflight failures are safe for the same reason. Node, provider, timeout, upload, and other post-start failures never carry this hint.

Install

Requires only Python ≥ 3.11 (it pulls in icefold-sdk).

The runner itself ships no node tooling. A node declares what it needs in its bundle header — binary_deps (for example, IceFold video nodes use google-chrome, ffmpeg, and ffprobe on PATH) and python_deps (whatever your custom nodes import) — and the runner pre-flights those before each run, replying with a platform-aware install hint (missing_dep) for anything absent. So you install a node's dependencies only when you actually run a job that needs them, and the runner tells you exactly what to install.

pip install icefold-runner          # pulls in icefold-sdk

From source:

git clone <this-repo> icefold-runner
cd icefold-runner
python -m venv .venv && . .venv/bin/activate
pip install -e .

Run

Generate a token in the IceFold app (Settings → Runners), then:

install -m 600 /dev/null ~/.icefold-runner-token
read -rsp 'Runner token: ' ICEFOLD_TOKEN_INPUT
printf '%s' "$ICEFOLD_TOKEN_INPUT" > ~/.icefold-runner-token
unset ICEFOLD_TOKEN_INPUT
icefold-runner --token-file ~/.icefold-runner-token

That's it — the token (GitHub-CI style) encodes + signs your IceFold user id, so there's no server URL or user id to pass. The server is built in.

Every flag also reads an env var:

The runner honors standard proxy env vars (HTTPS_PROXY, ALL_PROXY, …) for reaching the server, including HTTP and SOCKS proxies. It reconnects automatically with backoff; an auth rejection is fatal.

The runner advertises both effective lane widths in its WebSocket hello frame. Servers that understand these fields can fill a high-capacity runner before using a smaller backup; older servers safely ignore the extra fields. A newer server treats an older runner that does not advertise capacity as one slot.

Self-hosting / dev: point the runner at a different server with the ICEFOLD_RUNNER_SERVER env var (e.g. ws://127.0.0.1:7000).

Layout

icefold_runner/      the runner agent (connection, file staging, bundle exec)
  client.py            reverse-WS client: dial / auth / reconnect / keepalive
  runner.py            fetch /v1/bundles/<hash>, preflight deps, await __icefold_run__
  __main__.py          CLI entrypoint (icefold-runner)

The runner imports the bundle on demand; the bundle is self-contained and already inlines whatever it needs (the author's function body, the Inputs / Output dataclasses, and a minimal NodeContext shim). The only runtime dependency on icefold-sdk is the wire protocol + a small helper kit (get_file_id / run_blocking / write_text), used by the runner agent itself, not by node code.

Security model

• Node code runs unsandboxed here — it's your machine, your risk. That's the point: code the server refuses to execute (subprocess/ffmpeg/native deps and anything third-party) runs on the runner instead. The runner downloads each bundle from the server and executes it; it verifies the bundle's sha256 matches the requested hash, but the bundle itself is whatever the server you authenticated to sends. Only point a runner at a server you trust.
• The runner only talks to the one server you point it at, authenticated by its runner token; it pulls input files and pushes products over HTTP to that host.