gen-worker

Python SDK for writing endpoints that run on Cozy's worker pool. You write one decorated function or class; the SDK handles discovery, scheduling, model download + placement, cancellation, file I/O, streaming, and reporting back to the control plane.

Install

pip install gen-worker[torch]   # for PyTorch inference/training
pip install gen-worker          # plain Python (e.g. API-proxy endpoints)

Optional extras: [images] / [audio] / [video] for media I/O, [vision] for torchvision.

Hello world

pyproject.toml — the one config value:

[tool.gen_worker]
main = "myendpoint.main"

main.py:

import msgspec
from gen_worker import RequestContext, endpoint

class Input(msgspec.Struct):
    prompt: str

class Output(msgspec.Struct):
    text: str

@endpoint
def echo(ctx: RequestContext, payload: Input) -> Output:
    return Output(text=f"got: {payload.prompt}")

Run it locally, no orchestrator:

gen-worker run --payload '{"prompt": "hello"}'

cozyctl build / cozyctl deploy take it from here — the full path to a deployed, billed endpoint is tensorhub docs/writing-endpoints.md.

Adding a model

Hold state in a class: setup() runs once, every public method is one routable function. The worker downloads the binding, constructs the pipeline from the setup() annotation, and owns device placement + low-VRAM offload — endpoint code never touches .to("cuda") or offload config.

from diffusers import StableDiffusionXLPipeline
from gen_worker import HF, RequestContext, Resources, endpoint

@endpoint(
    model=HF("stabilityai/stable-diffusion-xl-base-1.0", dtype="bf16"),
    resources=Resources(gpu=True),
)
class Generate:
    def setup(self, pipeline: StableDiffusionXLPipeline) -> None:
        self.pipeline = pipeline

    def generate(self, ctx: RequestContext, payload: Input) -> Output:
        view = ctx.for_request(self.pipeline, seed=42)
        image = view(payload.prompt, generator=view.generator).images[0]
        return Output(text=ctx.save_image(image).ref)

Resources declares only what the endpoint CANNOT run without (gpu, gpu_count, libraries, vcpus) plus requires= — the function scope of the one requirement vocabulary, "sm89+, vram80g" style, minimum and recommended. A minimum gates a config WRITE, never execution: the worker runs on whatever machine it is handed, degraded and complaining. Handlers are exactly (self, ctx, payload); per-request state (sampler, seed, scheduler) lives in a ctx.for_request view over shared weights — never assigned onto the instance.

Bindings: HF(id, revision=, dtype=, subfolder=, files=, storage_dtype=), Hub(ref, release=, storage_dtype=), Civitai(id, version=), ModelScope(id, ...). The slot name comes from the models={} key or the setup() parameter — never a constructor argument. storage_dtype="fp8" keeps denoiser weights in fp8-E4M3 storage with per-layer upcast to the compute dtype (half the VRAM on any card); fp8-stored artifacts get the same treatment automatically. Quantization itself is ahead-of-time only — a conversion endpoint produces the artifact, never setup() (th#1803) — and the flavor axis is DELETED (§1.32(d), pgw#1148, finished by pgw#1319: the producer DECLARES a precision_class and nothing infers one from a label): selection within a tag group is tensor-layout-contract compatibility, declared per slot as Slot(layouts=…) (§1.33, pgw#1143). See docs/endpoint-authoring.md.

Curated checkpoint selection is a runtime payload argument: a handler declares model: SomeModelChoice (a ModelChoice enum of Model rows, each carrying a ModelRef binding + typed per-model defaults) and reads payload.model.defaults typed — one generate(model=) replaces N near-identical functions. model: SomeModelChoice | ModelRef opens BYOM. Streaming = an async-generator handler. Engine-hosted endpoints declare runtime="vllm" and get a booted, health-checked server subprocess injected into setup().

Slot(pipeline_cls, selected_by=, family=, default_checkpoint=) is the hub-resolved alternative to ModelChoice: the model SET lives in platform config, not code, and the COMPONENT TREE (pipeline.unet, pipeline.vae, ...) is derived from the pipeline class and published to the hub — parts are never declared as sibling slots. The per-model config SCHEMA derives from the handler's context annotation (ctx: RequestContext[SdxlDefaults], a gen_worker.families.GenerationDefaults vocabulary); the catalog owns the VALUES and ctx.defaults hands the resolved recipe to the handler typed. family="sdxl" explicitly keeps a non-root/defaultless model lane inside that architecture's binding gate; family="" explicitly marks a shared auxiliary as family-agnostic. Omitting it preserves the compatibility inference.

Full reference: docs/endpoint-authoring.md.

Public surface

The decorator + bindings: endpoint, Resources, Compile, HF, Hub, Civitai, ModelScope, ModelRef
Model selection: Model, ModelChoice, ModelDefaults, Slot, ResolvedSlot, gen_worker.families.GenerationDefaults
Compile envelope (the declared serving region — resolutions, text lengths, guidance, batch): Compile, CompileAxis, AxisClass, DynamicDim, pad_text_sequence; per-request views: ctx.for_request / gen_worker.view
Contexts: RequestContext (≤15 members) and JobContext, the one producer context — every @job body and every producer-kind @endpoint handler gets it, and what it may write comes from the declaration, not the kind
Errors: ValidationError, RetryableError, CanceledError, FatalError
Streaming: BatchItemDelta, IncrementalTokenDelta, Done, Error
Value types: Asset, ImageAsset, AudioAsset, VideoAsset
I/O codecs: gen_worker.io

The conversion ETL (hub ingest, dtype cast / quant, clone, Tensorhub publish) is gen_worker.convert (see docs/convert.md).

Local development

gen-worker run --payload '{"prompt": "hello"}'  # one-shot in-process
gen-worker run --list                            # describe functions (JSON)
gen-worker serve                                 # warm local server
gen-worker invoke <fn> prompt=hello              # client for serve
gen-worker prefetch                              # weights only, no GPU

stdout for results, stderr for events; exit 0 / 1 / 2 / 3 / 130 for success / user-exception / usage / model-resolution / SIGINT. Details: docs/local-dev.md; host contract: docs/host-integration.md.

Running tests

uv run --extra dev pytest

Plain uv run pytest would fall through to a global launcher — always pass --extra dev. Never pip install gen-worker globally: a stale ~/.local install silently shadows the working tree (tests/conftest.py hard-fails if gen_worker resolves outside src/).

Documentation

docs/endpoint-authoring.md — the @endpoint reference: bindings, variants, Resources, contexts, streaming, runtimes.
docs/models.md — the typed Model SDK: declare a model, bind a resolved instance to a handler parameter, generate typed bindings, run a handler hubless.
docs/local-dev.md — the CLI: run/serve/invoke/ prefetch, field=value grammar, --offline, exit codes.
docs/dockerfile.md — bring-your-own-Dockerfile contract.
docs/endpoint-envs.md — tenant envs/secrets.
docs/compile-cache.md — compiled cells: the graph digest vs the declared envelope, kernel lanes, JIT intake.

Examples

examples/marco-polo/ — minimal inference endpoint (sync, async, streaming)

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

gen_worker-0.125.0.tar.gz (2.4 MB view details)

Uploaded Aug 18, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

gen_worker-0.125.0-py3-none-any.whl (2.7 MB view details)

Uploaded Aug 18, 2026 Python 3

File details

Details for the file gen_worker-0.125.0.tar.gz.

File metadata

Download URL: gen_worker-0.125.0.tar.gz
Upload date: Aug 18, 2026
Size: 2.4 MB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for gen_worker-0.125.0.tar.gz
Algorithm	Hash digest
SHA256	`3d5f8e18cc7e158a9cdf68701805d2b68ed721383b8218443007810b0d526d6e`
MD5	`a2035aaa1354c487a36b8447ebcf7720`
BLAKE2b-256	`08e51cdcf35f9462f226004604e18d059ebf53b411af56f7950e954aa74f8da5`

See more details on using hashes here.

Provenance

The following attestation bundles were made for gen_worker-0.125.0.tar.gz:

Publisher: publish.yaml on cozy-creator/python-gen-worker

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: gen_worker-0.125.0.tar.gz
- Subject digest: 3d5f8e18cc7e158a9cdf68701805d2b68ed721383b8218443007810b0d526d6e
- Sigstore transparency entry: 2499979248
- Sigstore integration time: Aug 18, 2026
Source repository:
- Permalink: cozy-creator/python-gen-worker@81208895fbf17c84345ca06870abf56901ae9805
- Branch / Tag: refs/tags/v0.125.0
- Owner: https://github.com/cozy-creator
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: publish.yaml@81208895fbf17c84345ca06870abf56901ae9805
- Trigger Event: push