gen-worker
Python SDK for writing endpoints that run on Cozy's worker pool. You write one decorated function or class; the SDK handles discovery, scheduling, model download + placement, cancellation, file I/O, streaming, and reporting back to the control plane.
Install
pip install gen-worker[torch] # for PyTorch inference/training
pip install gen-worker # plain Python (e.g. API-proxy endpoints)
Optional extras: [images] / [audio] / [video] for media I/O,
[vision] for torchvision.
Hello world
pyproject.toml — the one config value:
[tool.gen_worker]
main = "myendpoint.main"
main.py:
import msgspec
from gen_worker import RequestContext, endpoint
class Input(msgspec.Struct):
prompt: str
class Output(msgspec.Struct):
text: str
@endpoint
def echo(ctx: RequestContext, payload: Input) -> Output:
return Output(text=f"got: {payload.prompt}")
Run it locally, no orchestrator:
gen-worker run --payload '{"prompt": "hello"}'
cozyctl build / cozyctl deploy take it from here — the full path to a
deployed, billed endpoint is tensorhub docs/writing-endpoints.md.
Adding a model
Hold state in a class: setup() runs once, every public method is one
routable function. The worker downloads the binding, constructs the pipeline
from the setup() annotation, and owns device placement + low-VRAM offload —
endpoint code never touches .to("cuda") or offload config.
from diffusers import StableDiffusionXLPipeline
from gen_worker import HF, RequestContext, Resources, endpoint
@endpoint(
model=HF("stabilityai/stable-diffusion-xl-base-1.0", dtype="bf16"),
resources=Resources(gpu=True),
)
class Generate:
def setup(self, pipeline: StableDiffusionXLPipeline) -> None:
self.pipeline = pipeline
def generate(self, ctx: RequestContext, payload: Input) -> Output:
view = ctx.for_request(self.pipeline, seed=42)
image = view(payload.prompt, generator=view.generator).images[0]
return Output(text=ctx.save_image(image).ref)
Resources declares only what the endpoint CANNOT run without (gpu,
gpu_count, libraries, vcpus) plus requires= — the function scope of
the one requirement vocabulary, "sm89+, vram80g" style, minimum and
recommended. A minimum gates a config WRITE, never execution: the worker runs
on whatever machine it is handed, degraded and complaining. Handlers are exactly
(self, ctx, payload); per-request state (sampler, seed, scheduler) lives
in a ctx.for_request view over shared weights — never assigned onto the
instance.
Bindings: HF(id, revision=, dtype=, subfolder=, files=, storage_dtype=),
Hub(ref, release=, storage_dtype=), Civitai(id, version=), ModelScope(id, ...).
The slot name comes from the models={} key or the setup() parameter —
never a constructor argument. storage_dtype="fp8" keeps denoiser weights in
fp8-E4M3 storage with per-layer upcast to the compute dtype (half the VRAM
on any card); fp8-stored artifacts get the same treatment automatically.
Quantization itself is ahead-of-time only — a conversion endpoint produces the
artifact, never setup() (th#1803) — and the flavor axis is DELETED
(§1.32(d), pgw#1148, finished by pgw#1319: the producer DECLARES a
precision_class and nothing infers one from a label): selection within a tag
group is tensor-layout-contract
compatibility, declared per slot as Slot(layouts=…) (§1.33, pgw#1143). See
docs/endpoint-authoring.md.
Curated checkpoint selection is a runtime payload argument: a handler declares
model: SomeModelChoice (a ModelChoice enum of Model rows, each carrying a
ModelRef binding + typed per-model defaults) and reads payload.model.defaults
typed — one generate(model=) replaces N near-identical functions. model: SomeModelChoice | ModelRef opens BYOM. Streaming = an async-generator handler.
Engine-hosted endpoints declare runtime="vllm" and get a booted,
health-checked server subprocess injected into setup().
Slot(pipeline_cls, selected_by=, family=, default_checkpoint=) is the hub-resolved
alternative to ModelChoice: the model SET lives in platform config, not
code, and the COMPONENT TREE (pipeline.unet, pipeline.vae, ...) is
derived from the pipeline class and published to the hub — parts are never
declared as sibling slots. The per-model config SCHEMA derives from the
handler's context annotation (ctx: RequestContext[SdxlDefaults], a
gen_worker.families.GenerationDefaults vocabulary); the catalog owns the
VALUES and ctx.defaults hands the resolved recipe to the handler typed.
family="sdxl" explicitly keeps a non-root/defaultless model lane inside
that architecture's binding gate; family="" explicitly marks a shared
auxiliary as family-agnostic. Omitting it preserves the compatibility
inference.
Full reference: docs/endpoint-authoring.md.
Public surface
- The decorator + bindings:
endpoint,Resources,Compile,HF,Hub,Civitai,ModelScope,ModelRef - Model selection:
Model,ModelChoice,ModelDefaults,Slot,ResolvedSlot,gen_worker.families.GenerationDefaults - Compile envelope (the declared serving region — resolutions, text lengths,
guidance, batch):
Compile,CompileAxis,AxisClass,DynamicDim,pad_text_sequence; per-request views:ctx.for_request/gen_worker.view - Contexts:
RequestContext(≤15 members) andJobContext, the one producer context — every@jobbody and every producer-kind@endpointhandler gets it, and what it may write comes from the declaration, not the kind - Errors:
ValidationError,RetryableError,CanceledError,FatalError - Incremental output (pgw#1576):
@entrypoint(streams=TokenDelta)+ctx.emit(...); chunk typesTokenDelta,ItemDelta,Delta, anditer_transformers_text_deltas. A streaming entrypoint still RETURNS its terminal struct — deltas are the droppable live lane, the return is the authoritative result - Value types:
Asset,ImageAsset,AudioAsset,VideoAsset - I/O codecs:
gen_worker.io
The conversion ETL (hub ingest, dtype cast / quant, clone, Tensorhub
publish) is gen_worker.convert (see docs/convert.md).
Local development
gen-worker run --payload '{"prompt": "hello"}' # one-shot in-process
gen-worker run --list # describe functions (JSON)
gen-worker serve # warm local server
gen-worker invoke <fn> prompt=hello # client for serve
gen-worker prefetch # weights only, no GPU
stdout for results, stderr for events; exit 0 / 1 / 2 / 3 / 130 for success / user-exception / usage / model-resolution / SIGINT. Details: docs/local-dev.md; host contract: docs/host-integration.md.
Running tests
uv run --extra dev pytest
Plain uv run pytest would fall through to a global launcher — always pass
--extra dev. Never pip install gen-worker globally: a stale
~/.local install silently shadows the working tree (tests/conftest.py
hard-fails if gen_worker resolves outside src/).
Documentation
- docs/endpoint-authoring.md — the
@endpointreference: bindings, variants, Resources, contexts, streaming, runtimes. - docs/models.md — the typed Model SDK: declare a model, bind a resolved instance to a handler parameter, generate typed bindings, run a handler hubless.
- docs/local-dev.md — the CLI:
run/serve/invoke/prefetch,field=valuegrammar,--offline, exit codes. - docs/dockerfile.md — bring-your-own-Dockerfile contract.
- docs/endpoint-envs.md — tenant envs/secrets.
- docs/compile-cache.md — compiled graphs: the graph digest vs the declared envelope, kernel lanes, JIT intake.
Examples
examples/marco-polo/— minimal inference endpoint (sync, async, streaming)
1 maintainer
Metadata
1 maintainer
Release files for gen-worker 0.128.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| gen_worker-0.128.0.tar.gz | 1.4 MB | With provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| gen_worker-0.128.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 3.0 MB
Release files / gen_worker-0.128.0.tar.gz
| Download URL | gen_worker-0.128.0.tar.gz |
|---|---|
| Size | 1.4 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
831514e7b51dc1ffd2eec2f2a18e89314088cda112edc95efa9aab7c6c7e78b2
|
|
BLAKE2b-256 checksum How to use checksums |
edc706b310433602afd01757577ef831a4af48e61d1bc23558cc3304af44b73a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 21, 2026.
Transparency logRelease files / gen_worker-0.128.0-py3-none-any.whl
| Download URL | gen_worker-0.128.0-py3-none-any.whl |
|---|---|
| Size | 1.6 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
7c524c45e49adcd0cba4a52cbc1cb79502c5c3ad4b67d285f496a3a659939aec
|
|
BLAKE2b-256 checksum How to use checksums |
3db56e6f1fd57878f33636cee79b7b8a171c5be2c0f7c478f7fddb9ba6a16740
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 21, 2026.
Transparency logSecurity for gen-worker 0.128.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
All 2 files in this release have been uploaded with provenance metadata.
Attestation for gen_worker-0.128.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 21, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
