horde-image-utilities

Isolated heavy image capabilities for the AI Horde worker ecosystem.

Some image utilities a worker needs (background removal via rembg/onnxruntime, and later controlnet annotators and layer diffusion) carry heavy, conflict-prone native dependencies. Pulling those into the worker's main environment bloats it and risks dependency clashes. This package keeps them out of the way: it runs each capability as a small FastAPI service in its own virtual environment, and the worker talks to it over loopback HTTP using a dependency-free client.

It ships four cooperating pieces:

• Service: a FastAPI application exposing capability endpoints (for example POST /rembg/remove-background) plus an always-on operational surface under /ops.
• Client: a standard-library-only HTTP client (HordeImageUtilitiesClient) that the lean consumer uses to call the service. Its only runtime dependency is Pillow, already a base dependency.
• Launcher: CapabilityServerProcess starts, health-checks, and tears down the service subprocess, optionally under a different interpreter so the heavy dependencies live in a separate venv.
• Provisioning: build_provision_commands produces the uv commands that create a capability venv and install this package with the right extras.

Two-mode model

• Lean consumer environment: installs the base package only (Pillow, pydantic). It imports horde_image_utilities.client and horde_image_utilities.launcher to drive a capability process. No FastAPI, rembg, torch, or onnxruntime enter this environment.
• Capability environment: a separate venv that installs the server extra plus exactly one backend extra. It runs the actual model inference. Keeping this environment separate is what makes the dependency isolation real.

Install matrix

The base install has no heavy backends. Capabilities are opt-in extras:

The three rembg_* extras are mutually exclusive: exactly one accelerator backend per environment. This is enforced by a [tool.uv].conflicts declaration, so uv refuses to install more than one. The annotators extra has no accelerator split (no detector in scope uses onnxruntime), so no conflict entry is needed.

Do not co-install mediapipe with annotators in the same environment. mediapipe pulls in opencv-contrib-python (the GUI wheel) while annotators pulls in opencv-contrib-python-headless. Two opencv wheels sharing the cv2 namespace break cv2 (it imports as an empty namespace). Until this is resolved via a dependency override at deployment-pin compile time, keep mediapipe in its own environment. The mediapipe_face detector is future work and is not required by any worker capability.

Provision a capability venv (CUDA background removal shown):

uv venv .venv-rembg
uv pip install --python .venv-rembg "horde-image-utilities[server,rembg_cuda]"

Quickstart

Run the service:

# In the capability environment
python -m horde_image_utilities
# or the console script:
horde-image-utilities

Call it from the lean consumer environment:

from PIL import Image

from horde_image_utilities.client import HordeImageUtilitiesClient

client = HordeImageUtilitiesClient("http://127.0.0.1:7860")
if client.health():
    result = client.remove_background(Image.open("input.png"))
    result.save("output.png")

Or let the launcher own the subprocess lifecycle (point python_executable at the capability venv):

from horde_image_utilities.launcher import CapabilityServerProcess

with CapabilityServerProcess(python_executable="/path/to/.venv-rembg/bin/python") as server:
    memory_report = server.client.get_memory_report()

Configuration

Settings are read from the environment with the HIU_ prefix (see horde_image_utilities.config.ServiceSettings):

HIU_ALLOW_DOWNLOADS defaults to false: the service will not fetch missing models at runtime unless you opt in. Pre-download models, or set it to true.

Cache layout

When HIU_ISOLATE_MODEL_CACHE is enabled and AIWORKER_CACHE_HOME is set, model files are stored under an isolated, per-capability directory:

$AIWORKER_CACHE_HOME/horde/image-utilities/rembg/                     # rembg ONNX models
$AIWORKER_CACHE_HOME/horde/image-utilities/controlnet_aux/            # controlnet_aux base
$AIWORKER_CACHE_HOME/horde/image-utilities/controlnet_aux/annotators/ # annotator checkpoints (default)
$AIWORKER_CACHE_HOME/horde/image-utilities/vendor/                    # cloned upstream annotator node (default)

If AIWORKER_CACHE_HOME is not set, or isolation is disabled, each backend falls back to its own default (U2NET_HOME / ~/.u2net for rembg, the Hugging Face cache for controlnet). The service logs a warning when conflicting cache environment variables are detected.

Annotator checkpoints

The annotators capability resolves weight files under HIU_ANNOTATOR_MODEL_DIR (defaulting to the annotators/ directory shown above), laid out as <repo_id>/<filename> exactly as the worker pre-populates them (for example lllyasviel/Annotators/ControlNetHED.pth). Missing weights return HTTP 409 rather than triggering a download (downloads are off unless HIU_ALLOW_DOWNLOADS=true).

The transformers-hub control types are the exception: their weights load through the transformers library from the standard Hugging Face hub cache, so the deployment must point HF_HOME (or HUGGINGFACE_HUB_CACHE) at a cache pre-populated with the relevant model. These are normal and midas_depth (Intel/dpt-hybrid-midas), zoe_depth (Intel/zoedepth-nyu-kitti), depth_anything (LiheYoung/depth-anything-large-hf), and oneformer_ade20k / oneformer_coco (shi-labs/oneformer_*_swin_large).

Implemented control types span the classic set and the extended surface the AI-Horde annotation form and worker pre-annotation use:

• Classic: canny, scribble, hed, fakescribbles, mlsd, depth (LeReS), normal (MiDaS), openpose, seg (UniFormer).
• Weightless algorithmic: binary, standard_lineart, scribble_xdog, pyracanny, color, shuffle, recolor_luminance, recolor_intensity, tile, tile_ttplanet_guided, tile_ttplanet_simple.
• Annotator-layout weighted: lineart, lineart_anime, lineart_anime_denoise, pidinet, scribble_pidinet, teed, normal_bae, depth_anything_v2.
• Transformers-hub weighted: midas_depth, zoe_depth, depth_anything, oneformer_ade20k, oneformer_coco.

Per-detector parameters match the defaults hordelib passes through the comfyui_controlnet_aux AIO preprocessor (only the resolution knob is set; every other widget stays at its node default), so pre-annotation here is interchangeable with hordelib's in-graph annotation. Notably lineart runs with coarse disabled, mirroring hordelib, which drives it through the LineArtPreprocessor node without setting the coarse widget.

The seg runtime comes from the clone's bundled MMSegmentation/MMCV subtree, which runs on CPU without compiled ops given the pure-python MMCV config packages the annotators extra pulls in. The mediapipe_face control type is registered but not implemented and returns HTTP 501.

At startup a best-effort prefetch (enabled only when HIU_ALLOW_DOWNLOADS=true) warms the classic control types' weights before the extended set, so a fresh install can serve the legacy surface quickly while the extended weights trickle in.

Annotator detector source (clone at a pin)

The annotator detector algorithms are not vendored into this repository. On first use (and, best-effort, at service startup) the capability clones comfyui_controlnet_aux at a commit pinned in horde_image_utilities/annotators/upstream_manifest.json, prepends the clone's src to sys.path, and imports custom_controlnet_aux directly (the clone is not pip-installed). This mirrors how hordelib pins ComfyUI: the acquisition is idempotent and self-healing, performing no network access once the clone is already at the pin. Two runtime monkeypatches redirect the upstream weight loading through this package's resolver (so pre-populated checkpoints are used and missing files return HTTP 409) and make the LeReS detector load only its depth model, never the pix2pix "boost" model. See the repository NOTICE for the attribution and the exact patch sites.

Pre-bake the clone for an offline or image-baking step (git only; needs neither torch nor the annotator extra):

python -m horde_image_utilities.annotators

The /ops surface

The ops endpoints mount in every install, including lean ones with no heavy backend. They assume a trusted loopback caller (the worker that launched the process) and carry no authentication.

The client exposes typed methods for each: get_memory_report(), release_cache(), unload(capability=None), and shutdown().

A GET /health liveness endpoint is always available regardless of which backends are installed.

Development

uv sync
uv run ruff format . && uv run ruff check . --fix
uv run pytest

Tests that require a rembg backend are skipped automatically when it is not installed. The package version is derived from the release tag via hatch-vcs; there is no version string to bump in source.