Solar Host

A multi-backend process manager for model inference servers with REST API and WebSocket log streaming.

Features

• Multi-Backend Support:
  • llama.cpp (llama-server) for GGUF models
  • HuggingFace AutoModelForCausalLM for text generation
  • HuggingFace AutoModelForSequenceClassification for classification
  • HuggingFace AutoModel for embeddings (last hidden state with mean pooling)
  • SGLang for high-throughput CUDA serving (NVIDIA hosts only, launched from its own virtualenv)
• Socket.IO control client - Connects to solar-control’s /hosts namespace for registration, heartbeat, and instance lifecycle (start/stop/restart, config updates). Supports pending-host and rejection events with post-approval sync.
• Robust instance lifecycle - Non-blocking process wait, state re-check after startup to avoid start/stop races, and full cleanup of log/state buffers on stop or delete.
• Log retention across failures (S-049/C2) - instance-addressable log files (instance id in the filename) survive process teardown; GET /instances/{id}/logs falls back to the on-disk file when the in-memory buffer is empty (even after the instance record is gone); delete_instance discards logs while stop retains them; retention bounded by log_file_retention_s (the newest file per (alias, instance_id) is always kept, so a multi-replica intent keeps a post-mortem for every replica).
• Structured start failures (C2) - a failed start answers 500 {detail, instance_id, exit_code, log_tail} so control can link the error to its process logs.
• Pull progress telemetry (S-051) - pull_model accepts a progress_cb; the parent poll loop emits throttled downloading events with bytes/speed plus exactly one terminal completed/failed event, pushed over the WS channel as pull_progress.
• Full resource snapshot on health push (S-050) - host_health carries the complete resource snapshot (byte-identical to GET /resources), alongside the legacy summary keys.
• Auto-assign ports starting from 3500
• Persistent configuration with auto-restart on boot
• Real-time log streaming via WebSocket
• REST API for instance management
• API key authentication

Installation

# Basic install (llama.cpp backend only)
pip install solar-host

# With HuggingFace backend support
pip install solar-host[huggingface]

# With NVIDIA GPU monitoring
pip install solar-host[nvidia]

# Everything
pip install solar-host[all]

# Development (editable install with test dependencies)
pip install -e ".[all,dev]"

Backend-Specific Requirements

For llama.cpp backend:

• Install llama-server and ensure it's in your PATH

For HuggingFace backends:

• Install with the huggingface extra: pip install solar-host[huggingface]

For the SGLang backend:

• NVIDIA CUDA host only — the host does not advertise the backend anywhere else, so solar-control never places an SGLang instance on a CPU or Apple host.
• SGLang lives in its own virtualenv (its dependency set conflicts with solar-host's), pointed at by SGLANG_VENV_PATH. Without that variable the runner falls back to a sglang on PATH; with neither, the backend is simply not advertised.

For the vLLM backend:

• NVIDIA CUDA host only, same advertisement rule as SGLang.
• vLLM lives in its own virtualenv, pointed at by VLLM_VENV_PATH; the runner prefers bin/vllm, then bin/python -m vllm.entrypoints.cli.main, then a vllm on PATH. First boot can take ten minutes or more on a cold JIT cache, which the ordinary INSTANCE_READY_TIMEOUT_S window covers.

Setup

1. Create .env file

Create a .env file in the solar-host/ directory:

API_KEY=your-secret-key-here
HOST=0.0.0.0
PORT=8001
MODELS_DIR=./models

# Solar-control connection (for Socket.IO registration and lifecycle)
SOLAR_CONTROL_URL=http://localhost:8000
SOLAR_CONTROL_API_KEY=your-solar-control-management-api-key

# SGLang backend (optional, NVIDIA hosts only)
SGLANG_VENV_PATH=/opt/venvs/sglang
SGLANG_PROMPT_CACHE_DIR=/var/cache/sglang

# vLLM backend (optional, NVIDIA hosts only)
VLLM_VENV_PATH=/opt/venvs/vllm

• API_KEY - Used by solar-control (and other callers) to access this host’s REST API.
• MODELS_DIR - Path to the models directory. Used for disk space reporting in the /health endpoint. Defaults to ./models.
• SOLAR_CONTROL_URL - Base URL of solar-control (HTTP; Socket.IO connects to the same origin).
• SOLAR_CONTROL_API_KEY - Management API key from solar-control. The host uses it to connect to the /hosts namespace; it must be approved via the management API or WebUI before it appears in the gateway pool.
• SGLANG_VENV_PATH - Virtualenv SGLang is installed into (the directory holding bin/sglang). The host launches the executable from it directly, setting VIRTUAL_ENV and prepending bin/ to PATH the way activate would. Leave empty to use a sglang on PATH; with neither, sglang is dropped from the advertised backends.
• SGLANG_PROMPT_CACHE_DIR - Root of SGLang's file-backed prompt cache. Each instance gets its own <root>/<alias>-<instance_id> subdirectory, exported as SGLANG_HICACHE_FILE_BACKEND_STORAGE_DIR. When it is empty the --hicache-storage-* flags are dropped with a warning (the in-memory hierarchical cache still works). The host deletes the subdirectory when the instance stops and sweeps leftovers at startup, so point it at a dedicated path. While a stop is in flight the dir is transiently visible as .trash-<uuid> until the background purge finishes, and the root must be dedicated to a single host — a second host's startup sweep would otherwise discard the first's live caches.
• VLLM_VENV_PATH - Virtualenv vLLM is installed into (the directory holding bin/vllm). The host launches the executable from it directly, setting VIRTUAL_ENV and prepending bin/ to PATH the way activate would. Leave empty to use a vllm on PATH; with neither, vllm is dropped from the advertised backends.
• GPU_TELEMETRY_OVERRIDE - (S-058, dev/test only) JSON array of {"index", "name", "total_gb", "used_gb"} entries that fakes per-device NVIDIA telemetry without a physical GPU. When set, GET /resources, the host_health push and every GPU capacity check read these devices instead of NVML — the integration suite uses it to exercise the whole S-058 chain on CPU-only CI runners. Unset on production hosts.

2. Start the server

# Start the server (reads HOST and PORT from .env)
solar-host

# Or with uvicorn directly (e.g. for --reload during development)
uvicorn solar_host.main:app --host 0.0.0.0 --port 8001 --reload

The server will:

• Create config.json automatically (if it doesn't exist)
• Create logs/ directory for instance logs
• Auto-restart any instances that were running before shutdown

3. Verify it's running

curl http://localhost:8001/health
# Should return: {"status":"healthy","service":"solar-host","version":"2.0.0","disk":{"total_gb":500,"used_gb":120,"available_gb":380}}

4. Access Swagger UI

Open your browser to: http://localhost:8001/docs

1. Click the "Authorize" button
2. Enter your API key from .env file
3. Click "Authorize" and then "Close"
4. Now you can use the interactive API documentation!

Backend Types

Solar Host supports five backend types:

Backend Type	Model Type	Endpoints Supported
llamacpp	GGUF models via llama-server	/v1/chat/completions, /v1/completions
huggingface_causal	HuggingFace AutoModelForCausalLM	/v1/chat/completions, /v1/completions
huggingface_classification	HuggingFace AutoModelForSequenceClassification	/v1/classify
huggingface_embedding	HuggingFace AutoModel (last hidden state)	/v1/embeddings
sglang	Safetensors models via sglang serve (NVIDIA only)	/v1/chat/completions, /v1/completions

GET / reports which of them this host can actually run as supported_backends, and the WS registration and health pushes carry the same list. sglang only appears there on an NVIDIA host where the executable resolves, which is how solar-control keeps SGLang replicas off hosts without it.

Managing Instances

Creating a llama.cpp Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "llamacpp",
      "model": "/path/to/model.gguf",
      "alias": "llama-3:8b",
      "threads": 4,
      "n_gpu_layers": 999,
      "temp": 0.7,
      "top_p": 0.9,
      "top_k": 40,
      "min_p": 0.05,
      "ctx_size": 8192,
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Creating a HuggingFace Causal LM Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "huggingface_causal",
      "model_id": "meta-llama/Llama-2-7b-chat-hf",
      "alias": "llama2-hf:7b",
      "device": "auto",
      "dtype": "auto",
      "max_length": 4096,
      "trust_remote_code": false,
      "use_flash_attention": true,
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Creating a HuggingFace Classification Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "huggingface_classification",
      "model_id": "distilbert-base-uncased-finetuned-sst-2-english",
      "alias": "sentiment:distilbert",
      "device": "auto",
      "dtype": "auto",
      "max_length": 512,
      "labels": ["negative", "positive"],
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Creating a HuggingFace Embedding Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "huggingface_embedding",
      "model_id": "sentence-transformers/all-MiniLM-L6-v2",
      "alias": "embed:minilm",
      "device": "auto",
      "dtype": "auto",
      "max_length": 512,
      "normalize_embeddings": true,
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Creating an SGLang Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "sglang",
      "model_path": "/path/to/model-directory",
      "alias": "deepseek-v4:flash",
      "tp_size": 8,
      "mem_fraction_static": 0.85,
      "context_length": 131072,
      "kv_cache_dtype": "fp8_e4m3",
      "enable_hierarchical_cache": true,
      "hicache_ratio": 30,
      "hicache_storage_backend": "file",
      "extra_args": ["--dist-init-addr", "10.0.0.1:5000"],
      "extra_env": {"SGLANG_DSV4_COMPRESS_STATE_DTYPE": "bf16"}
    }
  }'

--host, --port, --api-key and --served-model-name are not config fields: the host binds the port from its own allocator, authenticates with its own API key, and derives the served name from alias. extra_args is rejected when it tries to set one of them.

A colon in the alias is translated for SGLang, which reads alias:tag as base model plus LoRA adapter: deepseek-v4-flash:284b is served as deepseek-v4-flash-284b. The host reports that name as served_model_name on the instance (REST and instances_update), and solar-control rewrites the model field of each forwarded request to it while continuing to route on — and advertise — the alias. Clients keep using the alias.

Creating a vLLM Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "vllm",
      "model_path": "/path/to/model-directory",
      "alias": "glm-5.3-flash:320b",
      "tensor_parallel_size": 4,
      "gpu_memory_utilization": 0.9,
      "max_model_len": 1048576,
      "kv_cache_dtype": "fp8",
      "speculative_config": "{\"method\": \"mtp\", \"num_speculative_tokens\": 5}",
      "extra_env": {"VLLM_USE_FLASHINFER_MOE_FP4": "1"}
    }
  }'

--host, --port, --api-key, --model, --served-model-name, --enable-prompt-tokens-details and --disable-log-stats are not config fields: the host binds the port from its own allocator, authenticates with its own API key, and serves the alias verbatim — vLLM performs no : parsing on model names (registry lookup is exact-match), so glm-5.3-flash:320b is served under exactly that name and solar-control's request translation is a no-op for direct-alias requests to this backend; a request under a virtual model name is still translated to the served name. extra_args is rejected when it tries to set one of the host-managed flags.

Usage metrics: per-request token counts (prompt, cached split, generated) come from the vllm:* Prometheus counters at request end, and decode_tps is the engine's last ~10 s stats-window average seen while the request ran — a best-effort live number, not an exact per-request measurement, with the token counts the exact part.

Starting an Instance

curl -X POST http://localhost:8001/instances/{instance-id}/start \
  -H "X-API-Key: your-secret-key-here"

Viewing All Instances

curl http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here"

Stopping an Instance

curl -X POST http://localhost:8001/instances/{instance-id}/stop \
  -H "X-API-Key: your-secret-key-here"

API Endpoints

Instance Management

• POST /instances - Create new instance
• GET /instances - List all instances
• GET /instances/{id} - Get instance details
• PUT /instances/{id} - Update instance config
• DELETE /instances/{id} - Remove instance
• POST /instances/{id}/start - Start instance
• POST /instances/{id}/stop - Stop instance
• POST /instances/{id}/restart - Restart instance
• GET /instances/{id}/state - Get runtime state
• GET /instances/{id}/last-generation - Get last generation metrics

WebSocket

• WS /instances/{id}/logs - Stream logs with sequence numbers
• WS /instances/{id}/state - Stream runtime state updates

System

• GET /health - Health check
• GET /memory - GPU/RAM memory usage

Authentication

All requests require an X-API-Key header with your configured API key from the .env file.

Configuration Reference

llama.cpp Config Parameters

Parameter	Required	Default	Description
backend_type	No	"llamacpp"	Backend type identifier
model	Yes	-	Full path to the GGUF model file (or the pulled model directory when model_file is set)
model_file	No	-	Filename, relative path or * glob selecting the GGUF inside the model directory — see Selecting a GGUF by pattern
file_filters	No	-	HuggingFace download filters (allow_patterns) applied when the model is pulled, e.g. ["*UD-Q4_K_XL*", "mmproj-BF16.gguf"]
alias	Yes	-	Model alias (e.g., "llama-3:8b") used for routing
threads	No	1	Number of CPU threads to use
n_gpu_layers	No	999	Number of layers to offload to GPU (999 = all)
devices	No	-	Comma-separated devices to offload to (--device), e.g. "CUDA0,CUDA1" — see Multi-GPU hosts
split_mode	No	-	How to split across GPUs (--split-mode): "none", "layer", "row", "tensor"
tensor_split	No	-	Per-GPU proportions (--tensor-split), e.g. "3,1"
main_gpu	No	-	GPU index (--main-gpu) holding the model with split_mode "none", or the KV cache with "row"
temp	No	1.0	Sampling temperature (0.0-2.0)
top_p	No	1.0	Top-p sampling (0.0-1.0)
top_k	No	0	Top-k sampling (0 = disabled)
min_p	No	0.0	Min-p sampling (0.0-1.0)
ctx_size	No	131072	Context window size
chat_template_file	No	-	Path to Jinja chat template file
spec_type	No	-	Speculative decoding: "draft-mtp" or "draft-dspark" — see Speculative decoding
spec_draft_model	No	-	Draft GGUF for draft-dspark (--spec-draft-model); accepts the same patterns as model_file
spec_draft_n_max	No	-	Maximum draft tokens (--spec-draft-n-max)
spec_draft_conf_min	No	-	Truncate a drafted block below this predicted acceptance (--spec-draft-conf-min, draft-dspark only)
special	No	false	Enable llama-server --special flag
ot	No	-	Override tensor string (passed as -ot flag to llama-server)
model_type	No	"llm"	Model type: "llm", "embedding", or "reranker"
pooling	No	-	Pooling strategy for embedding models: "none", "mean", "cls", "last", "rank" (only valid when model_type is "embedding")
host	No	"0.0.0.0"	Host to bind to
port	No	auto	Port (auto-assigned if not specified)
api_key	Yes	-	API key for this instance

HuggingFace Causal LM Config Parameters

Parameter	Required	Default	Description
backend_type	Yes	-	Must be "huggingface_causal"
model_id	Yes	-	HuggingFace model ID or local path
alias	Yes	-	Model alias for routing
device	No	"auto"	Device: auto, cuda, mps, cpu
dtype	No	"auto"	Data type: auto, float16, bfloat16, float32
max_length	No	4096	Maximum sequence length
trust_remote_code	No	false	Trust remote code from HuggingFace
use_flash_attention	No	true	Use Flash Attention 2 if available
host	No	"0.0.0.0"	Host to bind to
port	No	auto	Port (auto-assigned if not specified)
api_key	Yes	-	API key for this instance

HuggingFace Classification Config Parameters

Parameter	Required	Default	Description
backend_type	Yes	-	Must be "huggingface_classification"
model_id	Yes	-	HuggingFace model ID or local path
alias	Yes	-	Model alias for routing
device	No	"auto"	Device: auto, cuda, mps, cpu
dtype	No	"auto"	Data type: auto, float16, bfloat16, float32
max_length	No	512	Maximum sequence length
labels	No	auto	Label names (auto-detected from model if not provided)
trust_remote_code	No	false	Trust remote code from HuggingFace
host	No	"0.0.0.0"	Host to bind to
port	No	auto	Port (auto-assigned if not specified)
api_key	Yes	-	API key for this instance

HuggingFace Embedding Config Parameters

Parameter	Required	Default	Description
backend_type	Yes	-	Must be "huggingface_embedding"
model_id	Yes	-	HuggingFace model ID or local path
alias	Yes	-	Model alias for routing
device	No	"auto"	Device: auto, cuda, mps, cpu
dtype	No	"auto"	Data type: auto, float16, bfloat16, float32
max_length	No	512	Maximum sequence length
normalize_embeddings	No	true	L2 normalize output embedding vectors
trust_remote_code	No	false	Trust remote code from HuggingFace
host	No	"0.0.0.0"	Host to bind to
port	No	auto	Port (auto-assigned if not specified)
api_key	Yes	-	API key for this instance

SGLang Config Parameters

Every optional flag defaults to "omitted", so SGLang's own default applies. The typed fields below cover the knobs that are stable across releases; anything else goes into extra_args / extra_env, which is what keeps this backend usable as SGLang's flag surface moves.

Parameter	Required	Default	Description
backend_type	Yes	-	Must be "sglang"
model_path	Yes	-	Local directory of the model weights (--model-path); derived from model_source when the instance is created through solar-control
alias	Yes	-	Model alias for routing; --served-model-name is derived from it, with any : replaced by -
tp_size	No	-	Tensor parallel size (--tp-size)
dp_size	No	-	Data parallel size (--dp-size)
context_length	No	-	Maximum context length (--context-length)
mem_fraction_static	No	-	GPU memory fraction for weights and the KV pool (--mem-fraction-static)
chunked_prefill_size	No	-	Prefill chunk in tokens (--chunked-prefill-size)
max_running_requests	No	-	Scheduler concurrency cap (--max-running-requests)
cuda_graph_max_bs	No	-	Largest batch size captured into a CUDA graph (--cuda-graph-max-bs)
cuda_graph_max_bs_decode	No	-	Largest decode batch size captured (--cuda-graph-max-bs-decode)
swa_full_tokens_ratio	No	-	Full-attention share for hybrid sliding-window models (--swa-full-tokens-ratio)
dtype	No	-	Weight dtype, e.g. "bfloat16" (--dtype)
quantization	No	-	Quantization method, e.g. "fp8" (--quantization)
kv_cache_dtype	No	-	KV cache dtype, e.g. "fp8_e4m3" (--kv-cache-dtype)
moe_runner_backend	No	-	MoE kernel backend, e.g. "flashinfer_mxfp4" (--moe-runner-backend)
speculative_algorithm	No	-	Speculative decoding algorithm, e.g. "EAGLE" (--speculative-algorithm)
trust_remote_code	No	false	Allow the model repo's own modelling code (--trust-remote-code)
enable_hierarchical_cache	No	false	Keep evicted prefixes in host memory (--enable-hierarchical-cache)
hicache_ratio	No	-	Host-to-device cache size ratio (--hicache-ratio)
hicache_mem_layout	No	-	Host cache layout, e.g. "page_first_direct" (--hicache-mem-layout)
hicache_io_backend	No	-	Host cache transfer backend, e.g. "direct" (--hicache-io-backend)
hicache_storage_backend	No	-	Persistent cache backend, e.g. "file" (--hicache-storage-backend); needs SGLANG_PROMPT_CACHE_DIR
hicache_storage_backend_extra_config	No	-	JSON object of storage options, e.g. {"max_size":"256G"} (--hicache-storage-backend-extra-config)
hicache_storage_prefetch_policy	No	-	Prefetch policy, e.g. "wait_complete" (--hicache-storage-prefetch-policy)
extra_args	No	-	Raw argv entries appended after the typed flags, so an entry here overrides one above it
extra_env	No	-	Extra environment variables for the SGLang process
host	No	"0.0.0.0"	Host to bind to
port	No	auto	Port (auto-assigned if not specified)

vLLM Config Parameters

Every optional flag defaults to "omitted", so vLLM's own default applies. The typed fields below cover the knobs that are stable across releases; anything else goes into extra_args / extra_env.

Parameter	Required	Default	Description
backend_type	Yes	-	Must be "vllm"
model_path	Yes	-	Local directory of the model weights (the positional model argument of vllm serve); derived from model_source when the instance is created through solar-control
alias	Yes	-	Model alias for routing; --served-model-name is the alias verbatim
tensor_parallel_size	No	-	Tensor parallel size (--tensor-parallel-size)
pipeline_parallel_size	No	-	Pipeline parallel size (--pipeline-parallel-size)
max_model_len	No	-	Maximum context length (--max-model-len)
gpu_memory_utilization	No	-	Fraction of GPU memory reserved for the engine (--gpu-memory-utilization)
max_num_seqs	No	-	Maximum concurrent sequences (--max-num-seqs)
max_num_batched_tokens	No	-	Maximum tokens per scheduler step (--max-num-batched-tokens)
dtype	No	-	Weight dtype, e.g. "bfloat16" (--dtype)
quantization	No	-	Quantization method, e.g. "fp8" (--quantization)
kv_cache_dtype	No	-	KV cache dtype, e.g. "fp8" (--kv-cache-dtype)
moe_backend	No	-	MoE kernel backend, e.g. "cutlass" (--moe-backend)
trust_remote_code	No	false	Allow the model repo's own modelling code (--trust-remote-code)
enforce_eager	No	false	Skip CUDA graph capture (--enforce-eager)
enable_prefix_caching	No	-	Prefix cache on/off (--enable-prefix-caching / --no-enable-prefix-caching); omitted leaves the engine default
speculative_config	No	-	JSON object string of speculative-decoding settings, e.g. {"method":"mtp","num_speculative_tokens":5} (--speculative-config)
tool_call_parser	No	-	Tool-call parser, e.g. "hermes" (--tool-call-parser)
reasoning_parser	No	-	Reasoning parser, e.g. "deepseek_r1" (--reasoning-parser)
enable_auto_tool_choice	No	false	Let the model choose tool calls (--enable-auto-tool-choice)
extra_args	No	-	Raw argv entries appended after the typed flags, so an entry here overrides one above it
extra_env	No	-	Extra environment variables for the vLLM process
host	No	"0.0.0.0"	Host to bind to
port	No	auto	Port (auto-assigned if not specified)

Device Options

Device	Description
auto	Automatically select best available (CUDA > MPS > CPU)
cuda	NVIDIA GPU (requires CUDA)
mps	Apple Silicon GPU (macOS)
cpu	CPU only

Selecting a GGUF by pattern

A pull resolves a model_source to a directory, but llama-server --model needs a file. model_file names it without hardcoding an absolute path:

{
  "backend_type": "llamacpp",
  "alias": "qwen3-vl:235b",
  "model_source": "huggingface://unsloth/Qwen3-VL-235B-Instruct-GGUF",
  "model_file": "*UD-Q4_K_XL*.gguf",
  "mmproj": "mmproj-BF16.gguf",
  "file_filters": ["*UD-Q4_K_XL*", "mmproj-BF16.gguf"]
}

The pattern is resolved against the pulled model directory: an absolute path is used as-is, then an exact relative path, then a glob at the directory root, and finally a recursive glob — so a bare filename or a broad pattern is still found inside a subfolder. Among multiple matches, the trailing shards of a split GGUF are dropped (llama-server loads them itself from ...-00001-of-000NN.gguf) and the largest remaining file wins. mmproj accepts the same patterns.

file_filters is a separate concern: it limits what gets downloaded (a file is kept when it matches any pattern), while model_file, mmproj and spec_draft_model decide which of the downloaded files each llama-server flag points at. Filters only apply to HuggingFace pulls — ORAS always pulls a Harbor artifact whole.

Multi-GPU hosts

n_gpu_layers decides how much of the model leaves the CPU; the four fields below decide where it goes on a host with several GPUs.

devices is the one to reach for when several instances share a host: it pins an instance to specific GPUs, so two models can run side by side without either of them seeing the other's VRAM. The names come from llama.cpp (llama-server --list-devices prints them, e.g. CUDA0, CUDA1), and leaving it empty offers every device to that instance:

{
  "backend_type": "llamacpp",
  "alias": "qwen3:32b",
  "model_source": "huggingface://org/Qwen3-32B-GGUF",
  "devices": "CUDA0,CUDA1",
  "split_mode": "row",
  "tensor_split": "3,1",
  "main_gpu": 0
}

split_mode chooses how the model is divided across whatever devices remain: "layer" (llama.cpp's default) pipelines layers and the KV cache, "row" splits weights by row and computes in parallel, "tensor" splits both and is still experimental, and "none" keeps the model on main_gpu alone.

tensor_split weights the division — "3,1" puts three quarters of the model on the first device — and is what unequal GPUs need. main_gpu selects the device that holds the model with split_mode: "none", or the KV cache and intermediate results with "row"; it does nothing for the other modes.

Scheduler-chosen devices (S-058)

Solar Control now bin-packs instances onto physical devices: it picks the exact gpu_ids, reserves them per instance, and the host enforces the choice when it starts the process (CUDA_VISIBLE_DEVICES in the given order, plus CUDA_DEVICE_ORDER=PCI_BUS_ID so the host's reported indices and CUDA's enumeration are the same space). The interfaces above (devices, main_gpu, tensor_split, tp_size) keep their meaning, but as positions within the chosen set — main_gpu: 0 is the most-free chosen GPU, never a physical id.

• GET /resources and the host_health push carry a gpus list — one entry per device with index, name, total_gb, used_gb, available_gb.
• POST /resources/reservations accepts gpu_count and gpu_ids; vram_gb is the per-GPU footprint. Without gpu_ids the host self-assigns the least-loaded devices, so legacy callers stay attributable per device.
• Instance records (and the WS instances_update payload) carry gpu_ids and vram_gb; the storage manifest shows the assigned devices per instance.
• At spawn the host re-verifies the chosen devices still have the requested capacity (a foreign process may have taken one since placement); a busy device fails the start fast with the device named, and the reconciler re-places on its next tick.
• GPU_TELEMETRY_OVERRIDE fakes the telemetry for tests (see .env section).

Speculative decoding

spec_type picks how the server drafts tokens ahead of the model. It only applies to model_type: "llm"; the flags are dropped for an embedding or reranker instance.

"draft-mtp" uses the served model's own Multi Token Prediction heads, so it needs nothing but a block size:

{
  "spec_type": "draft-mtp",
  "spec_draft_n_max": 2
}

"draft-dspark" drafts a whole block per forward pass with a separate DSpark model (a DFlash backbone plus a semi-autoregressive Markov head). It requires spec_draft_model, which is resolved exactly like mmproj — an absolute path, or a filename or glob looked up in the model directory:

{
  "backend_type": "llamacpp",
  "alias": "qwen3:4b",
  "model_source": "huggingface://org/Qwen3-4B-GGUF",
  "model_file": "*Q8_0*.gguf",
  "spec_type": "draft-dspark",
  "spec_draft_model": "*DSpark*.gguf",
  "spec_draft_n_max": 7,
  "file_filters": ["*Q8_0*", "*DSpark*"]
}

The drafter must be trained for the exact model being served (for example deepseek-ai/dspark_qwen3_4b_block7 for Qwen/Qwen3-4B), converted with convert_hf_to_gguf.py --target-model-dir so it inherits the target's tokenizer and embeddings. spec_draft_n_max is the block size and llama.cpp clamps it to the size the drafter was trained for — 7 for the published block7 checkpoints. The optional spec_draft_conf_min truncates each drafted block at the first position whose predicted acceptance falls below it, and only does anything for a drafter that carries a confidence head.

Since the drafter is a second file in the model directory, remember to widen file_filters to let it through on a HuggingFace pull.

Example Configurations

llama.cpp - Small Model

{
  "backend_type": "llamacpp",
  "model": "/models/llama-3-7b.gguf",
  "alias": "llama-3:7b",
  "threads": 4,
  "n_gpu_layers": 999,
  "temp": 0.7,
  "top_p": 0.9,
  "ctx_size": 8192,
  "api_key": "llama3-7b-key"
}

llama.cpp - Large Model with Custom Template

{
  "backend_type": "llamacpp",
  "model": "/models/gpt-oss-120b-F16.gguf",
  "alias": "gpt-oss:120b",
  "threads": 1,
  "n_gpu_layers": 999,
  "ctx_size": 131072,
  "chat_template_file": "/models/templates/harmony.jinja",
  "api_key": "gpt-oss-key"
}

HuggingFace - Text Generation

{
  "backend_type": "huggingface_causal",
  "model_id": "microsoft/phi-2",
  "alias": "phi-2:2.7b",
  "device": "cuda",
  "dtype": "float16",
  "max_length": 2048,
  "api_key": "phi2-key"
}

HuggingFace - Sentiment Classification

{
  "backend_type": "huggingface_classification",
  "model_id": "cardiffnlp/twitter-roberta-base-sentiment-latest",
  "alias": "sentiment:roberta",
  "device": "cuda",
  "max_length": 512,
  "labels": ["negative", "neutral", "positive"],
  "api_key": "sentiment-key"
}

HuggingFace - Embedding Model

{
  "backend_type": "huggingface_embedding",
  "model_id": "sentence-transformers/all-MiniLM-L6-v2",
  "alias": "embed:minilm",
  "device": "cuda",
  "max_length": 512,
  "normalize_embeddings": true,
  "api_key": "embed-key"
}

File Structure

solar-host/
├── .env                    # Configuration (not in git)
├── config.json             # Auto-generated instance storage (not in git)
├── logs/                   # Auto-generated log directory (not in git)
├── pyproject.toml          # Package metadata and dependencies
├── solar_host/
│   ├── backends/           # Backend runners
│   │   ├── base.py         # Abstract BackendRunner
│   │   ├── llamacpp.py     # llama.cpp runner
│   │   └── huggingface.py  # HuggingFace runner
│   ├── models/             # Pydantic models
│   │   ├── base.py         # Base models
│   │   ├── llamacpp.py     # llama.cpp config
│   │   └── huggingface.py  # HuggingFace configs
│   ├── servers/            # Standalone server processes
│   │   └── hf_server.py    # HuggingFace model server
│   ├── routes/             # API routes
│   ├── config.py           # Configuration management
│   ├── main.py             # FastAPI application
│   ├── models_manager.py   # Managed models directory and manifest
│   └── process_manager.py  # Process lifecycle management
├── tests/
└── README.md

Troubleshooting

Solar-host won't start

Error: "Address already in use"

• Another service is using port 8001
• Solution: Change PORT in .env or stop the other service

Error: "No module named 'solar_host'"

• The package is not installed
• Solution: pip install solar-host or pip install -e . for development

llama.cpp Instance fails to start

1. Verify llama-server is installed:

   which llama-server
   

2. Check model path:

   ls -lh /path/to/your/model.gguf
   

3. Check instance logs in logs/ directory

HuggingFace Instance fails to start

1. Verify dependencies:

   python -c "import torch; import transformers; print('OK')"
   

2. Check CUDA availability (if using GPU):

   python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}')"
   

3. Check MPS availability (macOS):

   python -c "import torch; print(f'MPS: {torch.backends.mps.is_available()}')"
   

4. Check instance logs in logs/ directory

Instance keeps retrying and failing

• Solar-host will retry starting an instance up to 2 times
• Check the error_message field:

  curl http://localhost:8001/instances/{instance-id} \
    -H "X-API-Key: your-key" | jq '.error_message'
  

Conda Environment

When running solar-host from a conda environment, HuggingFace server subprocesses automatically inherit the same environment. Just ensure all dependencies are installed in your conda environment:

conda activate your-env
pip install torch transformers accelerate

Integration with Solar Control

Solar-host connects to solar-control over Socket.IO (namespace /hosts). Set SOLAR_CONTROL_URL and SOLAR_CONTROL_API_KEY in .env. On startup the host registers and appears in solar-control’s pending list until approved.

Approve the host (via solar-control management API or WebUI):

# List pending hosts
curl http://your-control-server:8000/api/hosts/pending \
  -H "X-API-Key: your-management-api-key"

# Approve (use pending_id from the list)
curl -X POST http://your-control-server:8000/api/hosts/pending/{pending_id}/approve \
  -H "X-API-Key: your-management-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GPU Server 1",
    "url": "http://192.168.1.100:8001",
    "api_key": "your-solar-host-api-key"
  }'

Alternatively, create a host directly (no pending step) with POST /api/hosts and the same JSON body.

Once approved, instances are accessible through solar-control’s OpenAI-compatible gateway:

• /v1/chat/completions - Chat completion (llamacpp, huggingface_causal)
• /v1/completions - Text completion (llamacpp, huggingface_causal)
• /v1/classify - Classification (huggingface_classification)
• /v1/embeddings - Embeddings (huggingface_embedding)

GPU Execution

Solar Host supports NVIDIA GPU execution for step containers via the gpu field on each step definition. GPU access requires the NVIDIA Container Toolkit to be installed and the Docker daemon to have the nvidia runtime registered.

Job JSON examples

{ "gpu": { "count": 1 } }

One GPU — Docker picks the device. Recommended default on multi-GPU hosts.

{ "gpu": { "count": -1 } }

All available GPUs (explicit replacement for the legacy gpu: true boolean).

{ "gpu": { "device_ids": ["0"] } }

Pin a specific GPU by index or UUID.

Note: "gpu": {} (empty object) is a validation error. At least one of count or device_ids must be provided.

NVIDIA environment variables

When a step requests GPU access, the following environment variables are injected automatically (callers may override them via step.environment):

Variable	Default
NVIDIA_VISIBLE_DEVICES	all
NVIDIA_DRIVER_CAPABILITIES	compute,utility

Manual verification

To verify the NVIDIA Container Toolkit is working on a host without running a full job:

docker run --rm --gpus '"device=0"' nvidia/cuda:12.0-base nvidia-smi

Step Log Streaming

Solar Host captures stdout/stderr from every step container and makes them available in two complementary ways.

Data flow

Container stdout/stderr
        │
        ▼ (demuxed)
 JobStepExecutor._stream_logs()
        │
        ├──► JOBS_DIR/<job-id>/logs/<step>.log   (durable, combined)
        │
        └──► StepLogBuffer.append()
                  │
                  ├──► bounded in-memory deque  (latest 1000 lines)
                  └──► emit queue
                            │
                            ▼ (every 100 ms)
                  broadcast_step_log_batch()
                            │
                            ▼
                  Solar Control  →  "step_log" Socket.IO event

Event payload

Each step_log event carries an entries array. A normal log entry:

{
  "job_id":     "job-a1b2",
  "step_name":  "train",
  "step_index": 2,
  "stream":     "stdout",
  "seq":        42,
  "timestamp":  "2026-05-21T19:00:00.123456+00:00",
  "line":       "Epoch 1/10 loss=0.42"
}

The final entry for a step is a completion marker on the same event type:

{
  "job_id":     "job-a1b2",
  "step_name":  "train",
  "step_index": 2,
  "stream":     "stdout",
  "seq":        43,
  "timestamp":  "...",
  "line":       "",
  "completed":  true,
  "exit_code":  0
}

Using the same event shape for both regular lines and the completion marker means downstream consumers (S-032, Solar WebUI) only need to handle one event type.

Durability vs. real-time

	Durable	Real-time
Host log files	✅ JOBS_DIR/<job-id>/logs/<step>.log	❌
Socket.IO step_log	❌ best-effort, lost on disconnect	✅

The host-side log file is always written regardless of Solar Control connectivity. It is the authoritative record.

Reconnect behaviour

• SolarControlClient._emit is a no-op while disconnected — jobs never block or crash waiting for the WebSocket.
• The emit queue uses put_nowait with a 10 000-entry cap; entries are silently dropped when the queue is full (e.g. sustained disconnect + high-frequency logging).
• On reconnect, only new lines from that point forward are streamed; the durable log file can be read for historical lines.

Job Lifecycle Events (S-026)

Solar Host emits structured Socket.IO lifecycle events at every job and step state transition so Solar Control and downstream consumers (SuperNova, Solar WebUI) can monitor progress without polling.

Event catalog

Socket.IO event	Trigger	Key payload fields
job_started	After store.add(running) in JobExecutor.run_job	job_id, host_id, name, status, timestamp
step_started	After store.update_step(running) in JobStepExecutor	job_id, host_id, step_name, step_index, status, timestamp
step_completed	Successful container exit in _wait_and_record	job_id, host_id, step_name, step_index, status, timestamp, duration_s, exit_code
step_failed	Non-zero exit or start error in _wait_and_record / run	job_id, host_id, step_name, step_index, status, timestamp, duration_s, exit_code, error_summary
job_completed	All steps succeeded — _finalise_job completed path	job_id, host_id, status, timestamp, workspace_path, retention_deadline
job_failed	Any step failed or unexpected exception	job_id, host_id, status, timestamp, error_message
job_cancelled	Cancellation signal set — _finalise_job cancelled path	job_id, host_id, status, timestamp

host_id is the value received in registration_ack from Solar Control; it is null when the host has not yet registered.

retention_deadline is finished_at + retention_hours (ISO 8601 string).

error_summary for step_failed contains the last N lines of stderr captured in ContainerNonZeroExitError.last_stderr_lines.

Consistency guarantee

Each lifecycle event is emitted immediately after the matching JobStore mutation in the same coroutine. A consumer that receives a job_completed event can safely call GET /jobs/{id} and find the status already set to completed.

Fire-and-forget semantics

• Events are emitted directly by Socket.IO event name (e.g. sio.emit("job_started", data)) — no envelope, no batching.
• SolarControlClient.send_job_lifecycle is a no-op when disconnected; jobs never block waiting for the WebSocket.
• There is no retry queue for lifecycle events. State is always recoverable from GET /jobs/{id}.

Implementation details

solar_host/jobs/events.py        — payload builders + async emit_* functions
solar_host/ws_client.py          — send_job_lifecycle / broadcast_job_lifecycle
solar_host/jobs/executor.py      — job_started, job_completed, job_failed, job_cancelled
solar_host/jobs/step_executor.py — step_started, step_completed, step_failed

Backward Compatibility

Existing configurations without backend_type are automatically treated as llamacpp instances. No migration required.

Job REST API (S-027)

Intended callers: Solar Control only. These endpoints are not part of the public API surface.

Solar Host exposes three endpoints for submitting and managing containerised multi-step jobs. All endpoints require the X-API-Key header (same key as every other endpoint).

Endpoints

Method	Path	Description
POST	/jobs	Submit a new job for background execution
GET	/jobs/{job_id}	Inspect the current state of a job
DELETE	/jobs/{job_id}	Cancel a running job and delete its workspace

POST /jobs — Submit a job

Request body (JobDefinition shape):

{
  "job_id": "my-job-001",
  "name": "Fine-tune run",
  "steps": [
    {
      "name": "train",
      "image": "pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime",
      "command": ["python", "train.py"],
      "environment": {"EPOCHS": "5"},
      "gpu": {"count": 1}
    }
  ],
  "submission_id": "ctrl-sub-abc123",
  "correlation_id": "workflow-xyz"
}

• submission_id and correlation_id are optional strings Solar Control can use to correlate host jobs with its own records.
• job_id must be a simple alphanumeric slug (no path traversal characters).

Response — 202 Accepted:

{
  "job_id": "my-job-001",
  "status": "running",
  "workspace_path": "/var/solar/jobs/my-job-001",
  "submission_id": "ctrl-sub-abc123",
  "correlation_id": "workflow-xyz"
}

Error codes:

Code	Cause
400	Invalid job_id, empty steps, or GPU validation error
409	A job with the same job_id already exists in the store
503	Docker daemon is unavailable (executor disabled)
507	Insufficient disk space

GET /jobs/{job_id} — Inspect a job

Response — 200 OK:

{
  "job_id": "my-job-001",
  "name": "Fine-tune run",
  "status": "running",
  "current_step_index": 0,
  "workspace_path": "/var/solar/jobs/my-job-001",
  "created_at": "2026-05-21T19:00:00.000000+00:00",
  "started_at": "2026-05-21T19:00:01.000000+00:00",
  "finished_at": null,
  "retention_hours": 24.0,
  "error_message": null,
  "submission_id": "ctrl-sub-abc123",
  "correlation_id": "workflow-xyz",
  "steps": [
    {
      "name": "train",
      "status": "running",
      "container_id": "abc123def456",
      "started_at": "2026-05-21T19:00:01.500000+00:00",
      "finished_at": null,
      "duration_s": null,
      "exit_code": null,
      "error_message": null,
      "log_file": "/var/solar/jobs/my-job-001/logs/train.log",
      "recent_logs": [
        {
          "seq": 1,
          "stream": "stdout",
          "line": "Epoch 1/5 loss=0.87",
          "timestamp": "2026-05-21T19:00:05.000000+00:00"
        }
      ]
    }
  ]
}

• log_file is the host-side path to the durable combined stdout/stderr log for that step.
• recent_logs is a tail of the in-memory log buffer (up to 100 entries). Real-time streaming is available via Socket.IO step_log events (S-025).

Error codes:

Code	Cause
404	Unknown job_id

DELETE /jobs/{job_id} — Cancel a job

Performs synchronous cancellation per S-021 §6.3:

1. Signals the cancel event and stops the active container.
2. Waits up to 10 s for the run_job task to reach a terminal state.
3. Deletes the workspace directory.
4. Removes the job from the in-memory store and log buffer.

Response — 200 OK:

{"detail": "cancelled", "job_id": "my-job-001"}

Error codes:

Code	Cause
404	Unknown job_id
409	Job is already in a terminal state (completed, failed, or cancelled) — the cleanup loop will remove it

Example curl commands

API_KEY="your-secret-key-here"
HOST="http://localhost:8001"

# Submit a job
curl -s -X POST "$HOST/jobs" \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "demo-001",
    "name": "Demo job",
    "steps": [
      {
        "name": "hello",
        "image": "alpine:3.19",
        "command": ["echo", "hello from Solar Host"]
      }
    ]
  }' | jq .

# Poll state
curl -s "$HOST/jobs/demo-001" -H "X-API-Key: $API_KEY" | jq .status

# Cancel (if still running)
curl -s -X DELETE "$HOST/jobs/demo-001" -H "X-API-Key: $API_KEY" | jq .

---

Resource Reservations (S-034)

Solar Host exposes a host-local reservation ledger so Solar Control can pre-claim VRAM/RAM/disk capacity before placing a training job, without over-committing a host. All state is in-memory only — reservations do not survive a host restart.

Accounting formula

For each dimension d (VRAM, RAM, disk):

reported_usage_d = system.usage_d + Σ_i max(reservation_i.reserved_d − reservation_i.actual_d ?? 0, 0)
available_d      = total_d − reported_usage_d

system.usage_d is the live measured system usage (pynvml for VRAM, psutil for RAM, shutil for disk). For a pending reservation actual is null so the full reserved amount counts as headroom. For a running reservation only the unconsumed headroom max(reserved − actual, 0) is added, so real consumption captured in system.usage is never double-counted.

A reservation is running iff the linked job_id exists in the local JobStore with status == running.

Endpoints

Method	Path	Success	Description
POST	/resources/reservations	201	Create a reservation
GET	/resources	200	Snapshot: capacity + reservation list
DELETE	/resources/reservations/{id}	200	Release a reservation

All endpoints require the X-API-Key header (inherited from global middleware).

POST /resources/reservations

Request body (ReservationRequest):

{
  "job_id": "job-abc123",
  "workload_type": "training",
  "vram_gb": 16.0,
  "ram_gb": 8.0,
  "disk_gb": 50.0,
  "ttl_seconds": 3600
}

• ttl_seconds or expires_at may be specified (not both). When neither is given, a default TTL (default_reservation_ttl_seconds, 24 h by default) is applied so abandoned reservations can't hold capacity forever. This is a default, not a cap — Solar Control may set a longer-lived reservation by supplying its own ttl_seconds/expires_at.
• vram_gb/ram_gb/disk_gb must be >= 0; ttl_seconds (when given) must be > 0.
• Returns 201 ReservationView on success.
• Returns 409 {"error": "capacity_exceeded", "dimension": "vram", "requested_gb": …, "available_gb": …} when the request would exceed available capacity.
• Returns 422 for invalid request body.

GET /resources

Returns a ResourceSnapshot with per-dimension availability and the full reservation list including per-job actual usage (for running reservations):

{
  "memory_type": "VRAM",
  "vram": {"total_gb": 24.0, "system_used_gb": 8.0, "reserved_headroom_gb": 4.0, "reported_used_gb": 12.0, "available_gb": 12.0},
  "ram":  {"total_gb": 64.0, "system_used_gb": 12.0, "reserved_headroom_gb": 4.0, "reported_used_gb": 16.0, "available_gb": 48.0},
  "disk": {"total_gb": 500.0, "system_used_gb": 100.0, "reserved_headroom_gb": 20.0, "reported_used_gb": 120.0, "available_gb": 380.0},
  "reservations": [
    {
      "id": "res-<hex>",
      "job_id": "job-abc123",
      "workload_type": "training",
      "status": "pending",
      "vram_gb": 16.0,
      "ram_gb": 8.0,
      "actual_vram_gb": null,
      "actual_ram_gb": null,
      "expires_at": "2024-01-01T12:00:00Z"
    }
  ]
}

DELETE /resources/reservations/{id}

• Returns 200 {"detail": "released", "id": "<id>"} on success.
• Returns 404 for unknown IDs.
• Returns 409 when the linked job is currently running (running reservations must not be released while the job holds real capacity).

Background loops

Two background tasks are started alongside the existing health_report_loop:

Loop	Interval	Action
resource_usage_poll_loop	10 s	Refreshes per-job actual VRAM/RAM/disk for running reservations
reservation_cleanup_loop	60 s	Removes expired non-running reservations

host_health payload extension

The host_health Socket.IO event now includes a reservations block:

{
  "reservations": {
    "active_count": 2,
    "vram": {"total_gb": 24.0, "system_used_gb": 8.0, "reserved_headroom_gb": 4.0, "reported_used_gb": 12.0, "available_gb": 12.0},
    "ram":  {...},
    "disk": {...}
  }
}

Per-reservation details are omitted from the event (decision O4); use GET /resources to enumerate reservations.

Example curl commands

API_KEY="your-secret-key-here"
HOST="http://localhost:8001"

# Create a reservation
curl -s -X POST "$HOST/resources/reservations" \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"job_id": "job-001", "workload_type": "training", "vram_gb": 16.0, "ram_gb": 8.0, "ttl_seconds": 3600}' | jq .

# Check current availability
curl -s "$HOST/resources" -H "X-API-Key: $API_KEY" | jq '{memory_type, vram: .vram.available_gb, ram: .ram.available_gb}'

# Release the reservation
curl -s -X DELETE "$HOST/resources/reservations/res-<id>" -H "X-API-Key: $API_KEY" | jq .