gen-worker
Python SDK for writing endpoints that run on Cozy's worker pool. You write one decorated function or class; the SDK handles discovery, scheduling, model download + placement, cancellation, file I/O, streaming, and reporting back to the control plane.
Install
pip install gen-worker[torch] # for PyTorch inference/training
pip install gen-worker # plain Python (e.g. API-proxy endpoints)
Optional extras: [images] for image I/O, [audio] for audio I/O,
[trainer] for trainer-class endpoints.
Hello world
pyproject.toml — the one config value:
[tool.gen_worker]
main = "myendpoint.main"
main.py:
import msgspec
from gen_worker import RequestContext, endpoint
class Input(msgspec.Struct):
prompt: str
class Output(msgspec.Struct):
text: str
@endpoint
def echo(ctx: RequestContext, payload: Input) -> Output:
return Output(text=f"got: {payload.prompt}")
Run it locally, no orchestrator:
gen-worker run --payload '{"prompt": "hello"}'
cozyctl build / cozyctl deploy take it from here — the full path to a
deployed, billed endpoint is tensorhub docs/writing-endpoints.md.
Adding a model
Hold state in a class: setup() runs once, every public method is one
routable function. The worker downloads the binding, constructs the pipeline
from the setup() annotation, and owns device placement + low-VRAM offload —
endpoint code never touches .to("cuda") or offload config.
from diffusers import StableDiffusionXLPipeline
from gen_worker import HF, RequestContext, Resources, endpoint
@endpoint(
model=HF("stabilityai/stable-diffusion-xl-base-1.0", dtype="bf16"),
resources=Resources(vram_gb=12),
)
class Generate:
def setup(self, pipe: StableDiffusionXLPipeline) -> None:
self.pipe = pipe
def generate(self, ctx: RequestContext, payload: Input) -> Output:
image = self.pipe(payload.prompt, generator=ctx.generator(42)).images[0]
return Output(text=ctx.save_image(image).ref)
Resources(vram_gb=N) recommends a card size — the total VRAM of the smallest
card the function targets, not free bytes. It's an optional placement hint: the
platform reserves ~1 GB for driver/framebuffer/CUDA-context overhead, so
vram_gb=24 serves on any 24 GB card.
Bindings: HF(id, revision=, dtype=, subfolder=, files=, storage_dtype=),
Hub(ref, tag=, flavor=, storage_dtype=), Civitai(id, version=), ModelScope(id, ...).
The slot name comes from the models={} key or the setup() parameter —
never a constructor argument. storage_dtype="fp8" keeps denoiser weights in
fp8-E4M3 storage with per-layer upcast to the compute dtype (half the VRAM
on any card); fp8-stored #fp8 flavors get the same treatment automatically.
Multi-variant endpoints (bf16/fp8/... checkpoints with different VRAM
envelopes) declare variants={name: (binding, Resources)} — one handler body,
one routable function per variant. Streaming = an async-generator handler.
Engine-hosted endpoints declare runtime="vllm" and get a booted,
health-checked server subprocess injected into setup().
Full reference: docs/endpoint-authoring.md.
Public surface
- The decorator + bindings:
endpoint,Resources,HF,Hub,Civitai,ModelScope - Contexts:
RequestContext(≤15 members),ConversionContext,DatasetContext,TrainingContext - Errors:
ValidationError,RetryableError,CanceledError,FatalError - Streaming:
BatchItemDelta,IncrementalTokenDelta,Done,Error - Value types:
Asset,ImageAsset,AudioAsset,VideoAsset - I/O codecs:
gen_worker.io
The conversion ETL (hub ingest, dtype cast / quant, clone, Tensorhub
publish) is gen_worker.convert (see docs/convert.md).
Local development
gen-worker run --payload '{"prompt": "hello"}' # one-shot in-process
gen-worker run --list # describe functions (JSON)
gen-worker serve # warm local server
gen-worker invoke <fn> prompt=hello # client for serve
gen-worker prefetch # weights only, no GPU
stdout for results, stderr for events; exit 0 / 1 / 2 / 3 / 130 for success / user-exception / usage / model-resolution / SIGINT. Details: docs/local-dev.md; host contract: docs/host-integration.md.
Running tests
uv run --extra dev pytest
Plain uv run pytest would fall through to a global launcher — always pass
--extra dev. Never pip install gen-worker globally: a stale
~/.local install silently shadows the working tree (tests/conftest.py
hard-fails if gen_worker resolves outside src/).
Documentation
- docs/endpoint-authoring.md — the
@endpointreference: bindings, variants, Resources, contexts, streaming, runtimes. - docs/local-dev.md — the CLI:
run/serve/invoke/prefetch,field=valuegrammar,--offline, exit codes. - docs/dockerfile.md — bring-your-own-Dockerfile contract.
- docs/endpoint-envs.md — tenant envs/secrets.
Examples
examples/marco-polo/— minimal inference endpoint (sync, async, streaming)
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file gen_worker-0.14.10.tar.gz.
File metadata
- Download URL: gen_worker-0.14.10.tar.gz
- Upload date:
- Size: 398.5 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.18 {"installer":{"name":"uv","version":"0.9.18","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
4dca2faad6a8ed6f10423028873f27e6f9a5deafab667c1448e68ee345461f53
|
|
| MD5 |
a28d47cd16c166d3bf820720e4b18010
|
|
| BLAKE2b-256 |
76c9ef5b4e3c6c58a25f5e16270c856c84885dafb18a7ef3da4999b23ea5366c
|
File details
Details for the file gen_worker-0.14.10-py3-none-any.whl.
File metadata
- Download URL: gen_worker-0.14.10-py3-none-any.whl
- Upload date:
- Size: 451.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
uv/0.9.18 {"installer":{"name":"uv","version":"0.9.18","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
72397bd82f87550f7cda1f065ed4b77a5db3f1fc795bb9f7a44efe9c0439ba16
|
|
| MD5 |
3437c749f13fa86491c65f6ca85a936c
|
|
| BLAKE2b-256 |
933f45d6308a5af90c180660e868e19320bc892779f5373306130a352987e404
|
