strata-pool

Worker pool for dispatching Strata jobs to ephemeral machines. This is the bring-your-own-hardware path: it manages machines you own. If you want a provider to autoscale for you, register a serverless executor as a Strata worker instead.

pip install strata-pool            # library
pip install "strata-pool[server]"  # plus the HTTP service

Jobs arrive with a machine type. The pool hands each one to a warm worker of that type, starts a machine when there is none, forwards the payload over HTTP, records the result, and meters the execution. A backend provides start / stop / health and nothing else, so the pool does not know which it is talking to. Three ship: local Docker, RunPod, and Fly Machines. Anything satisfying the Backend protocol works.

from strata_pool import DockerBackend, MachineType, Pool, PoolStore

pool = Pool(
    store=PoolStore("pool.sqlite"),
    backend=DockerBackend(),
    machine_types=[MachineType(name="cpu-4x", image="strata-worker:latest")],
)
await pool.recover()          # reconcile after a restart
pool.start_scaler()           # stop paying for machines that finished

job = await pool.submit(tenant_id="acme", machine_type="cpu-4x", payload=bundle)
done = await pool.wait(job.id)

Isolation

A machine belongs to one tenant for its life and is destroyed rather than handed to another. Even scrubbed of files, a process that ran one tenant's code is not a boundary the next tenant should have to trust, and GPU memory is not reliably zeroed between processes at all. max_workers is therefore a per-tenant cap.

MachineType.cpus and memory_mb bound what a container may consume; unset, it may take the whole host. Both are enforced by the daemon, asserted against a real one in test_docker_live.py.

What that buys is process-level isolation, which is not a boundary for untrusted code: a shared kernel is one CVE away from a cross-tenant escape. A deployment running untrusted work wants a VM-backed runtime (Kata, gVisor) or a backend whose machines are already microVMs. The Backend protocol is where that choice lives.

Pool(max_workers_total=...) caps the whole fleet across every tenant. MachineType.max_workers caps one tenant, so without it the fleet is that number times however many tenants show up. Unset means no ceiling, which is right for a single-tenant pool and wrong for a hosted one. Hitting the ceiling logs a warning: a capped fleet looks exactly like a slow queue from the outside.

Still missing: any restriction on the container's own network access.

Several processes over one store

PoolStore is a SQLite file, and one pool process on it is the simple deployment: a restart pauses dispatch for seconds and fails the jobs that were in flight. To keep dispatching through a restart, run more than one process over PostgresPoolStore, each with its own instance_id:

pip install "strata-pool[postgres]"

pool = Pool(
    store=PostgresPoolStore("postgresql://pool@db/pool"),
    backend=RunPodBackend(os.environ["RUNPOD_API_KEY"]),
    machine_types=catalogue,
    instance_id=os.environ["HOSTNAME"],   # required on a shared store
)

The processes never run a job twice or start two machines for the same demand. Handing a job to a warm machine is a conditional update that only one of them wins, and deciding to start a machine counts the queue and inserts the new machine's row in one transaction, serialized across processes. Each process holds a lease on the jobs and machines it is acting on (a machine starting, running a job or stopping; a job dispatched or running) and renews it while it works. When a process dies, the scaler in another one fails its jobs and stops its machines once the lease runs out (lease_seconds, default 30), through the surviving process's own backend. A process restarted under its old instance_id takes its own rows back at once, as a single process does today. A pool given no instance_id generates one per process, so two processes over one SQLite file are never one name. Under a shared name each would read the other's leases as its own and stop machines running the other's cells. The trade is that a generated name is new on every start, so a process restarted within lease_seconds waits its old leases out instead of reclaiming them at once; set instance_id explicitly to keep that.

Lease expiry is compared by wall clock across processes, so their clocks have to agree to well within a lease. Each process holds its catalogue in memory: PUT /v1/machine-types updates the process that served it and the stored catalogue, and the others pick it up when they restart.

What it is not

It is not a cache. The pool has no idea Strata deduplicates work. Submitting a job whose result already exists boots a machine and recomputes it, so the caller checks find_by_provenance before submitting. Getting that order wrong bills customers for cache hits.

It is not the metering layer. The pool records one UsageEvent per terminal job, including failures, with a monotonic duration. What is billable, and at what price, is decided above it.

It does not keep machines warm on purpose. start_scaler() stops machines idle past their type's cool_down_seconds, and nothing else in the pool ever ends a machine that finished its work: a deployment that forgets that call bills for every machine it ever started. A machine whose stop fails keeps its row, still counted against the fleet cap, and the next pass tries the stop again. A warm floor is deliberately absent: per tenant it means paying for everyone who ever showed up, per machine type it means choosing whose latency to subsidise, and pre-warming belongs with the layer that knows a user just opened a notebook.

The same scaler pass health-checks warm machines and retires ones that stopped answering, after health_check_failures consecutive misses (default 3; 0 turns probing off for a type). One miss is a slow machine or a dropped packet, and retiring on that trades a cold start for every hiccup; three is a machine that is gone. Probes run concurrently, so a provider black-holing packets cannot stall the reaper behind one timeout per machine. And a pass in which every machine fails is treated as this process's own network rather than as the fleet dying at once, since retiring everything for a local DNS blip would hand every user a cold start for a fault that was never on the machines. Without this a dead warm machine is discovered by the next job being sent to it, and that job fails for reasons that have nothing to do with the code in the cell. A machine running a job is neither probed nor reaped.

Layout

Running it as a service

pip install 'strata-pool[server]'

from strata_pool.api import create_app

app = create_app(pool, api_token=os.environ["STRATA_POOL_TOKEN"])
# uvicorn strata_pool_service:app

The app's lifespan calls recover() and start_scaler() itself, so a deployment cannot forget the call that stops it paying for idle machines.

Replacing the catalogue takes effect at once. A new type accepts jobs straight away. A removed type accepts none: its queued jobs fail with the reason, and its machines finish what they are running, then retire once idle past the type's cool-down. A type whose image changed starts new machines on the new image. Machines already running the old image get no new jobs and retire the same way. The catalogue is stored with the pool's state, and on start it replaces the one the process was constructed with.

A job submitted with W3C traceparent / tracestate headers keeps them, and the pool forwards them to the machine it runs on. With OpenTelemetry installed (it is not a dependency) the pool also opens a pool.execute span for the job's time on the machine. The machine's work then sits under that span, which sits under the caller's.

A job that fails on the worker comes back as 502, and one that times out as 504. The caller has to be able to tell "your code raised" from "we could not run it".

Every route but /health requires Authorization: Bearer <api_token>, and job routes require X-Strata-Tenant. The caller is trusted for tenant identity: it authenticates as itself and asserts whose work this is. The pool does not authenticate end users and must never be reachable from anywhere but the proxy.

The RunPod backend

Rents real hardware. MachineType.gpu_type is the provider's own string ("NVIDIA H100 80GB PCIe"), deliberately not a normalised label, because one that maps cleanly across providers does not exist and inventing it would put a lossy translation between a user and the hardware they asked for.

RunPodBackend(os.environ["RUNPOD_API_KEY"])
MachineType(
    name="h100-80gb",
    image="strata-worker:latest",
    gpu_type="NVIDIA H100 80GB PCIe",
    boot_timeout_seconds=600,   # pulling an image onto a fresh pod is minutes
    cool_down_seconds=60,       # an idle H100 is the expensive mistake
)

Verified against a live account on 2026-09-02: the base URL, POST /pods with imageName / ports / env / containerDiskInGb / name, the id in the create response, the proxy endpoint, health through that proxy, and DELETE /pods/{id} including a second delete of the same pod. A CPU pod booted, answered, and terminated with nothing left running.

Two things that run stayed silent on, because a CPU pod does not exercise them: gpuTypeIds + gpuCount, and reading a region from machine. Every pod in the account listing carried machine: {} with no dataCenterId, so region is probably always None today. That is harmless, being metadata, but do not trust it.

RunPod has moved its API surface before. Every field lives in _create_body and is asserted by a test, so a wrong one is a one-line fix; provider_options overrides anything in the body, so a deployment can correct a field without waiting for a release; and base_url can be repointed. To re-verify, or to close the GPU gap with STRATA_POOL_RUNPOD_GPU:

export RUNPOD_API_KEY=...
STRATA_POOL_RUNPOD_LIVE=1 pytest packages/strata-pool/tests/test_runpod_live.py -v -s

That starts a billed pod, which is why it is opt-in and never runs in CI. It terminates what it starts, but a crashed interpreter can still leave a pod running, so check the console. Pods are named strata-{machine_type}-{id} so an orphan is findable.

A pod's port is published on the public internet through RunPod's proxy. The per-machine credential is what stands between that URL and anyone who finds it.

The Fly Machines backend

Boots workers as machines in one Fly app and region. A machine is reachable only on the organization's private network, at http://{machine_id}.vm.{app}.internal:{port}, so a pool running beside its notebook servers on Fly hands them workers that never face the internet.

FlyBackend(os.environ["FLY_API_TOKEN"], app="strata-workers", region="sjc")
MachineType(
    name="a100",
    image="registry.fly.io/strata-worker:latest",
    cpus=8,
    memory_mb=65536,
    gpu_type="a100-80gb",       # Fly's own gpu_kind string
    cool_down_seconds=120,
)

start creates the machine with the worker token and the machine type's env in its environment, with restart policy no and auto_destroy: false so its lifetime stays the pool's. stop destroys it by force (a second destroy is a no-op). health needs Fly to report the machine started before it probes the worker's /health, because a stopped machine's private address can be reused.

Not yet verified against a live account. The request shapes follow the Machines API documentation; every field lives in _create_body and is asserted by a test, provider_options overrides anything in the machine config, and base_url can be repointed. To verify:

export FLY_API_TOKEN=... STRATA_POOL_FLY_APP=... STRATA_POOL_FLY_REGION=sjc
STRATA_POOL_FLY_LIVE=1 pytest packages/strata-pool/tests/test_fly_live.py -v -s

That starts a billed machine. Off the private network it proves create, start and destroy; add STRATA_POOL_FLY_ON_NETWORK=1 where the .internal names resolve (a Fly machine, or a fly wireguard peer) to probe the worker too.

The worker credential still matters here: every machine in the organization can reach every other.

The worker contract

strata-worker, shipped in strata-notebook, satisfies this contract as of 0.7.0. Build it with the Dockerfile in the Strata repo and the pool can drive it unmodified:

docker build -f worker.Dockerfile -t strata-worker:latest .

The image installs a pinned strata-notebook from PyPI (the STRATA_VERSION build arg) and needs 0.7.0 or newer: POST /execute, the path the pool dispatches to, ships in that release. Installing from PyPI also means building inside a checkout does not pick up local worker changes; build a wheel for that.

Layer your cells' dependencies on top (FROM strata-worker:latest). The image binds 8080 because that is DockerBackend's default worker_port; the worker's own default is 9000, so the two are made to agree explicitly rather than by luck. The pool does not pull, so build on the host that will run it. It runs as a non-root user and refuses to start without STRATA_WORKER_TOKEN. The pool mints one per machine, so this only bites when running it by hand.

A server on a private address has to be named. The worker rejects manifest URLs that resolve to loopback or private addresses. That is a real defense, since a buggy or compromised orchestrator could otherwise point it at internal services. With DockerBackend on a local daemon, or workers on Fly's private network, the Strata server is at a private address, so the first job fails with resolves to non-routable address. In a deployment, list the server's host in the machine type's env as STRATA_WORKER_ALLOWED_HOSTS (comma-separated; a leading dot is a suffix, e.g. .internal). STRATA_WORKER_ALLOW_LOCAL_HOSTS=1 relaxes the check for every host and is a local-development setting only.

The payload the pool forwards is a build manifest: the same JSON document /v1/execute-manifest takes, carrying signed URLs for the inputs, the output, and finalization. The worker fetches its own inputs and uploads its own result, so the bytes never flow through the pool. That is also why the payload has to be self-describing: the pool forwards it verbatim and sets no content type.

The pool remains image-agnostic. Anything holding up these four points works:

The token is minted per machine before it boots and passed in its environment. Without that check, /execute is an unauthenticated remote-code-execution endpoint, survivable only while the machine is bound to loopback, which stops being true the moment a backend hands out a routable address.

The Docker backend

Talks to the Docker Engine API over its UNIX socket with httpx, so the pool needs no Docker SDK and every request shape is testable without a daemon. It starts the image, publishes the worker port on loopback only, and reports the host port Docker picked. It does not care what runs inside the container.

It does not pull. An image that is not on the host fails with the daemon's own "No such image" message.

Tests

pytest packages/strata-pool/tests -v

The tests in test_docker_live.py run the whole path against real containers and skip when no daemon is reachable. Set STRATA_POOL_DOCKER_SOCKET if yours is not at /var/run/docker.sock (Docker Desktop on macOS puts it in ~/.docker/run/docker.sock), and STRATA_POOL_REQUIRE_DOCKER=1 to make a missing daemon an error instead of a skip. CI sets that, so a runner whose socket moved fails loudly rather than reporting coverage it never ran.

CI additionally installs the package into a venv with only its own dependencies, to keep it from quietly growing a dependency on the server.