hugpy-engine

Inference resolution for the Hugpy arms. It turns "a prompt and maybe a file" into a model and a task, and hands the call to central's /v1 relay. Central picks the worker, and that worker's wrapper does evict-to-fit, load, serving and the call log. The engine itself does not load models in fleet mode.

Distribution hugpy-engine, import hugpy_engine. Dependencies: hugpy-platform, abstract-essentials, pydantic>=2.

Status: LIVE. It is a library only: no service, no console script.

• Installed in /opt/hugpy-media/venv (0.3.7) and /opt/hugpy-video/venv (0.3.8).
• Imported by hugpy_media, hugpy_video and hugpy_oracle.
• Core's live basic mode does not use it. Only core's legacy config/prompt.py does, and that code is not live.

On 2026-10-15 the engine's last ties to hugpy-storage and hugpy-control are being removed, targeting 0.3.9 and later:

• vl_gguf_tasks is ported in from hugpy_storage.hugpy_marker.
• The hugpy_storage.model_paths.route_destination call is dropped; it was dead in fleet mode.
• hugpy_storage.console.downloader.model_status is replaced by registry availability from /v1/models.
• The guarded imports of the retired hugpy_control.settings.settings_store are deleted.

Check pyproject.toml for the current version. See docs/packages.md.

Public API

from hugpy_engine import (
    execute_prompt, execute_prompt_stream, execute_chat_stream,
    resolve, resolve_model_key,
    MODEL_REGISTRY, get_model_registry, refresh_registry,
    normalize_prompt_kwargs, infer_arg_name, runner_for,
)

Positional arguments to execute_prompt are inferred by type:

execute_prompt_stream is a single resolve, build and runner pass yielded as events (TokenEvent, DoneEvent, ErrorEvent). It accepts cancel_event. execute_chat_stream adds chat continuation on top of it.

Chat and vision run today. A task whose runner has no front-door endpoint yet resolves to a clean, named error (fail-as-data).

execute_prompt: infer, rank, assure

Full reference: docs/execute_prompt.md.

execute_prompt is not plain task routing. Its defining contract (operator): it "would infer programmatically what the user was looking to prompt, via file inspection, prompt type, and anything that necessarily indicated a model type — most assuredly to receive a response than most any other query method." Task bindings are only ONE input. The mechanic has three stages, all in the stdlib-only module hugpy_engine.resolvers.task_resolution, so that central can later serve the same resolution.

1. INFER. task_resolution.infer(args, kwargs) works out WHAT to prompt from several signals:

   • positional args: a bare prompt, a messages list, an existing file, a model-ish string, a data: URL or an http(s) URL
   • file type: the extension maps to image, audio, video, document, code or text
   • message parts: image_url, input_image, audio, data: URLs
   • explicit cues: task=, model_format=

   It derives the task only when the caller gave no explicit task and no explicit model_key:

   input	task
   image	image-text-to-text
   audio or video	asr
   pdf or doc	extract + text
   text	text-generation

   One-call use works: execute_prompt("describe this", "/path/img.png") needs no task or model. Inference only ever adds a task. It never overrides an explicit one, and a failure inside it never blocks the call.

2. RANK. task_resolution.resolve(...) returns an ORDERED candidate list, strongest first, by source precedence:

   explicit model_key → binding → configured default → capability match
   

   Capability matches are ranked **resident > available on a healthy worker

   other**. A row whose primary_task matches the task is preferred, then order is deterministic by id. "No model in the fleet lists task X" appears ONLY when no registry model can serve X. It is never a dead-end "bind one in Settings".

3. ASSURE. execute_prompt tries the candidates in order.

   • On a MODEL-SIDE failure (worker down, load refusal, timeout, 5xx, empty reply), it moves to the next candidate.
   • A CALLER error (4xx validation, bad args) is surfaced immediately. The same bad request would fail on every model, so it is never cascaded.
   • The first real answer wins and carries resolution metadata: {task, source, model_key, attempts:[...]}.
   • When every candidate fails, it returns a fail-as-data result (ok=False) with the full attempt list, never a bare exception that hides what was tried.

Registry and bindings source

The registry is built from basic core's GET /v1/models rows: id, hugpy_json.tasks, task_override, engine, resident, available, workers and resident_on. derive_model_config_row maps each row onto ModelConfig. If that fails, it falls back to the legacy GET /models, and then to a local hugpy.json disk scan of HUGPY_MODELS_HOME / LLAMA_CPP_MODEL_DIR. The scan is fail-soft and fails fast on an EIO models disk.

Bindings are read from GET /v1/task-bindings ({"defaults": {...}}), then the legacy GET /prompt/tasks. A 404 means "no bindings". Live basic core (0.4.40) serves neither route, so the binding stage is empty today and resolution relies on explicit keys, configured defaults and capability matches.

Both reads are TTL-cached and never raise on the resolve path.

Configuration

hugpy_engine.imports.constants names every environment variable, timeout and TTL the chain reads:

The same module holds the TASK_DEFAULTS, MEDIA_DEFAULTS and MEDIA_TYPE_TO_TASK tables.

The task plugin seam

hugpy_engine.tasks holds the task registry. Packages add tasks through the hugpy_engine.tasks entry-point group (a zero-argument callable), which tasks.load_entry_points() imports. Two packages register today:

• hugpy-media: media = "hugpy_media.plugin:register"
• hugpy-video: video = "hugpy_video.plugin:register"

task_endpoints.TASK_ROUTES maps tasks to the front-door endpoints.

Layout

hugpy_engine/
  dispatch/       execute_prompt / execute_prompt_stream / execute_chat_stream; per-(model, task) runner cache
  resolvers/      task_resolution (infer / rank, stdlib only), model_resolver (resolve), assure_model_key,
                  categories/ (task + framework registries)
  config/         models/ (MODEL_REGISTRY, models_config, defaults), main
  llama/runners/  fitevict_runner: the runner that sends a call to central's /v1
  schemas/        chat, event, metadata, model, runner, task schemas (pydantic)
  fit/, placement.py   placement types and null single-box defaults
  tasks.py, task_endpoints.py, task_deps.py, categories.py, name_match.py
  utils/          json_scavenge, no_think
  imports/        constants (every env name), functions, classes, init_imports
                  (docs/packages.md#the-imports-layout-convention)

See also:

• docs/execute_prompt.md
• docs/arms.md
• docs/packages.md