gen-worker
Python SDK for writing endpoints that run on Cozy's worker pool. You write one decorated function or class; the SDK handles discovery, scheduling, model download + placement, cancellation, file I/O, streaming, and reporting back to the control plane.
Install
pip install gen-worker[torch] # for PyTorch inference/training
pip install gen-worker # plain Python (e.g. API-proxy endpoints)
Optional extras: [images] / [audio] / [video] for media I/O,
[vision] for torchvision.
Hello world
pyproject.toml — the one config value:
[tool.gen_worker]
main = "myendpoint.main"
main.py:
import msgspec
from gen_worker import RequestContext, endpoint
class Input(msgspec.Struct):
prompt: str
class Output(msgspec.Struct):
text: str
@endpoint
def echo(ctx: RequestContext, payload: Input) -> Output:
return Output(text=f"got: {payload.prompt}")
Run it locally, no orchestrator:
gen-worker run --payload '{"prompt": "hello"}'
cozyctl build / cozyctl deploy take it from here — the full path to a
deployed, billed endpoint is tensorhub docs/writing-endpoints.md.
Adding a model
Hold state in a class: setup() runs once, every public method is one
routable function. The worker downloads the binding, constructs the pipeline
from the setup() annotation, and owns device placement + low-VRAM offload —
endpoint code never touches .to("cuda") or offload config.
from diffusers import StableDiffusionXLPipeline
from gen_worker import HF, RequestContext, Resources, endpoint
@endpoint(
model=HF("stabilityai/stable-diffusion-xl-base-1.0", dtype="bf16"),
resources=Resources(gpu=True),
)
class Generate:
def setup(self, pipeline: StableDiffusionXLPipeline) -> None:
self.pipeline = pipeline
def generate(self, ctx: RequestContext, payload: Input) -> Output:
view = ctx.for_request(self.pipeline, seed=42)
image = view(payload.prompt, generator=view.generator).images[0]
return Output(text=ctx.save_image(image).ref)
Resources declares only what the endpoint CANNOT run without (gpu,
gpu_count, libraries, strict_vram, vcpus); VRAM requirements are
MEASURED by the platform's profiling gate, not declared (vram_gb_hint is
an optional first-build placement hint only). Handlers are exactly
(self, ctx, payload); per-request state (sampler, seed, scheduler) lives
in a ctx.for_request view over shared weights — never assigned onto the
instance.
Bindings: HF(id, revision=, dtype=, subfolder=, files=, storage_dtype=),
Hub(ref, tag=, flavor=, storage_dtype=), Civitai(id, version=), ModelScope(id, ...).
The slot name comes from the models={} key or the setup() parameter —
never a constructor argument. storage_dtype="fp8" keeps denoiser weights in
fp8-E4M3 storage with per-layer upcast to the compute dtype (half the VRAM
on any card); fp8-stored #fp8 artifacts get the same treatment automatically.
Quantization itself is ahead-of-time only — a conversion endpoint produces the
artifact, never setup() (th#1803) — and flavor= is a dying axis: selection
within a tag group becomes tensor-layout-contract compatibility
(DESIGN-RULINGS §1.33, pgw#1143). See
docs/endpoint-authoring.md.
Curated checkpoint selection is a runtime payload argument: a handler declares
model: SomeModelChoice (a ModelChoice enum of Model rows, each carrying a
ModelRef binding + typed per-model defaults) and reads payload.model.defaults
typed — one generate(model=) replaces N near-identical functions. model: SomeModelChoice | ModelRef opens BYOM. Streaming = an async-generator handler.
Engine-hosted endpoints declare runtime="vllm" and get a booted,
health-checked server subprocess injected into setup().
Slot(pipeline_cls, selected_by=, family=, default_checkpoint=) is the hub-resolved
alternative to ModelChoice: the model SET lives in platform config, not
code, and the COMPONENT TREE (pipeline.unet, pipeline.vae, ...) is
derived from the pipeline class and published to the hub — parts are never
declared as sibling slots. The per-model config SCHEMA derives from the
handler's context annotation (ctx: RequestContext[SdxlDefaults], a
gen_worker.families.GenerationDefaults vocabulary); the catalog owns the
VALUES and ctx.defaults hands the resolved recipe to the handler typed.
family="sdxl" explicitly keeps a non-root/defaultless model lane inside
that architecture's binding gate; family="" explicitly marks a shared
auxiliary as family-agnostic. Omitting it preserves the compatibility
inference.
Full reference: docs/endpoint-authoring.md.
Public surface
- The decorator + bindings:
endpoint,Resources,Compile,HF,Hub,Civitai,ModelScope,ModelRef - Model selection:
Model,ModelChoice,ModelDefaults,Slot,ResolvedSlot,gen_worker.families.GenerationDefaults - Compile envelope (the declared serving region — resolutions, text lengths,
guidance, batch):
Compile,CompileAxis,AxisClass,DynamicDim,pad_text_sequence; per-request views:ctx.for_request/gen_worker.view - Contexts:
RequestContext(≤15 members),ConversionContext,DatasetContext,TrainingContext - Errors:
ValidationError,RetryableError,CanceledError,FatalError - Streaming:
BatchItemDelta,IncrementalTokenDelta,Done,Error - Value types:
Asset,ImageAsset,AudioAsset,VideoAsset - I/O codecs:
gen_worker.io
The conversion ETL (hub ingest, dtype cast / quant, clone, Tensorhub
publish) is gen_worker.convert (see docs/convert.md).
Local development
gen-worker run --payload '{"prompt": "hello"}' # one-shot in-process
gen-worker run --list # describe functions (JSON)
gen-worker serve # warm local server
gen-worker invoke <fn> prompt=hello # client for serve
gen-worker prefetch # weights only, no GPU
stdout for results, stderr for events; exit 0 / 1 / 2 / 3 / 130 for success / user-exception / usage / model-resolution / SIGINT. Details: docs/local-dev.md; host contract: docs/host-integration.md.
Running tests
uv run --extra dev pytest
Plain uv run pytest would fall through to a global launcher — always pass
--extra dev. Never pip install gen-worker globally: a stale
~/.local install silently shadows the working tree (tests/conftest.py
hard-fails if gen_worker resolves outside src/).
Documentation
- docs/endpoint-authoring.md — the
@endpointreference: bindings, variants, Resources, contexts, streaming, runtimes. - docs/local-dev.md — the CLI:
run/serve/invoke/prefetch,field=valuegrammar,--offline, exit codes. - docs/dockerfile.md — bring-your-own-Dockerfile contract.
- docs/endpoint-envs.md — tenant envs/secrets.
- docs/compile-cache.md — compiled cells: the graph digest vs the declared envelope, kernel lanes, JIT intake.
Examples
examples/marco-polo/— minimal inference endpoint (sync, async, streaming)
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file gen_worker-0.111.0.tar.gz.
File metadata
- Download URL: gen_worker-0.111.0.tar.gz
- Upload date:
- Size: 2.0 MB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1b19a64422208c1f60bc1172201bf568d23bbdcc7952425e6704be0161ddab45
|
|
| MD5 |
18e9191262debfbb7aa173a13f245ace
|
|
| BLAKE2b-256 |
a2d30514a30623a8fc1641d99557c9bb95270987834d28cfee0a501d91502782
|
Provenance
The following attestation bundles were made for gen_worker-0.111.0.tar.gz:
Publisher:
publish.yml on cozy-creator/python-gen-worker
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
gen_worker-0.111.0.tar.gz -
Subject digest:
1b19a64422208c1f60bc1172201bf568d23bbdcc7952425e6704be0161ddab45 - Sigstore transparency entry: 2426964915
- Sigstore integration time:
-
Permalink:
cozy-creator/python-gen-worker@559a7f2b765e1608b69968d68ac111b01cca03db -
Branch / Tag:
refs/tags/v0.111.0 - Owner: https://github.com/cozy-creator
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@559a7f2b765e1608b69968d68ac111b01cca03db -
Trigger Event:
push
-
Statement type:
File details
Details for the file gen_worker-0.111.0-py3-none-any.whl.
File metadata
- Download URL: gen_worker-0.111.0-py3-none-any.whl
- Upload date:
- Size: 2.2 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d9ce8a9a72bcb249a7bd354a1c806701694bf00207fde5d5633071360948f3c2
|
|
| MD5 |
9478f7cb493ab6fdf0b78eb895c692a2
|
|
| BLAKE2b-256 |
4b005ad052fec7feb93005553627119ac4497737eba45855a471a7e948f3bb2d
|
Provenance
The following attestation bundles were made for gen_worker-0.111.0-py3-none-any.whl:
Publisher:
publish.yml on cozy-creator/python-gen-worker
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
gen_worker-0.111.0-py3-none-any.whl -
Subject digest:
d9ce8a9a72bcb249a7bd354a1c806701694bf00207fde5d5633071360948f3c2 - Sigstore transparency entry: 2426965391
- Sigstore integration time:
-
Permalink:
cozy-creator/python-gen-worker@559a7f2b765e1608b69968d68ac111b01cca03db -
Branch / Tag:
refs/tags/v0.111.0 - Owner: https://github.com/cozy-creator
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@559a7f2b765e1608b69968d68ac111b01cca03db -
Trigger Event:
push
-
Statement type:
