Solar Host

A multi-backend process manager for model inference servers with REST API and WebSocket log streaming.

Features

• Multi-Backend Support:
  • llama.cpp (llama-server) for GGUF models
  • HuggingFace AutoModelForCausalLM for text generation
  • HuggingFace AutoModelForSequenceClassification for classification
  • HuggingFace AutoModel for embeddings (last hidden state with mean pooling)
• Socket.IO control client - Connects to solar-control’s /hosts namespace for registration, heartbeat, and instance lifecycle (start/stop/restart, config updates). Supports pending-host and rejection events with post-approval sync.
• Robust instance lifecycle - Non-blocking process wait, state re-check after startup to avoid start/stop races, and full cleanup of log/state buffers on stop or delete.
• Log retention across failures (S-049/C2) - instance-addressable log files (instance id in the filename) survive process teardown; GET /instances/{id}/logs falls back to the on-disk file when the in-memory buffer is empty (even after the instance record is gone); delete_instance discards logs while stop retains them; retention bounded by log_file_retention_s (the newest file per (alias, instance_id) is always kept, so a multi-replica intent keeps a post-mortem for every replica).
• Structured start failures (C2) - a failed start answers 500 {detail, instance_id, exit_code, log_tail} so control can link the error to its process logs.
• Pull progress telemetry (S-051) - pull_model accepts a progress_cb; the parent poll loop emits throttled downloading events with bytes/speed plus exactly one terminal completed/failed event, pushed over the WS channel as pull_progress.
• Full resource snapshot on health push (S-050) - host_health carries the complete resource snapshot (byte-identical to GET /resources), alongside the legacy summary keys.
• Auto-assign ports starting from 3500
• Persistent configuration with auto-restart on boot
• Real-time log streaming via WebSocket
• REST API for instance management
• API key authentication

Installation

# Basic install (llama.cpp backend only)
pip install solar-host

# With HuggingFace backend support
pip install solar-host[huggingface]

# With NVIDIA GPU monitoring
pip install solar-host[nvidia]

# Everything
pip install solar-host[all]

# Development (editable install with test dependencies)
pip install -e ".[all,dev]"

Backend-Specific Requirements

For llama.cpp backend:

• Install llama-server and ensure it's in your PATH

For HuggingFace backends:

• Install with the huggingface extra: pip install solar-host[huggingface]

Setup

1. Create .env file

Create a .env file in the solar-host/ directory:

API_KEY=your-secret-key-here
HOST=0.0.0.0
PORT=8001
MODELS_DIR=./models

# Solar-control connection (for Socket.IO registration and lifecycle)
SOLAR_CONTROL_URL=http://localhost:8000
SOLAR_CONTROL_API_KEY=your-solar-control-management-api-key

• API_KEY - Used by solar-control (and other callers) to access this host’s REST API.
• MODELS_DIR - Path to the models directory. Used for disk space reporting in the /health endpoint. Defaults to ./models.
• SOLAR_CONTROL_URL - Base URL of solar-control (HTTP; Socket.IO connects to the same origin).
• SOLAR_CONTROL_API_KEY - Management API key from solar-control. The host uses it to connect to the /hosts namespace; it must be approved via the management API or WebUI before it appears in the gateway pool.

2. Start the server

# Start the server (reads HOST and PORT from .env)
solar-host

# Or with uvicorn directly (e.g. for --reload during development)
uvicorn solar_host.main:app --host 0.0.0.0 --port 8001 --reload

The server will:

• Create config.json automatically (if it doesn't exist)
• Create logs/ directory for instance logs
• Auto-restart any instances that were running before shutdown

3. Verify it's running

curl http://localhost:8001/health
# Should return: {"status":"healthy","service":"solar-host","version":"2.0.0","disk":{"total_gb":500,"used_gb":120,"available_gb":380}}

4. Access Swagger UI

Open your browser to: http://localhost:8001/docs

1. Click the "Authorize" button
2. Enter your API key from .env file
3. Click "Authorize" and then "Close"
4. Now you can use the interactive API documentation!

Backend Types

Solar Host supports four backend types:

Managing Instances

Creating a llama.cpp Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "llamacpp",
      "model": "/path/to/model.gguf",
      "alias": "llama-3:8b",
      "threads": 4,
      "n_gpu_layers": 999,
      "temp": 0.7,
      "top_p": 0.9,
      "top_k": 40,
      "min_p": 0.05,
      "ctx_size": 8192,
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Creating a HuggingFace Causal LM Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "huggingface_causal",
      "model_id": "meta-llama/Llama-2-7b-chat-hf",
      "alias": "llama2-hf:7b",
      "device": "auto",
      "dtype": "auto",
      "max_length": 4096,
      "trust_remote_code": false,
      "use_flash_attention": true,
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Creating a HuggingFace Classification Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "huggingface_classification",
      "model_id": "distilbert-base-uncased-finetuned-sst-2-english",
      "alias": "sentiment:distilbert",
      "device": "auto",
      "dtype": "auto",
      "max_length": 512,
      "labels": ["negative", "positive"],
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Creating a HuggingFace Embedding Instance

curl -X POST http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here" \
  -H "Content-Type: application/json" \
  -d '{
    "config": {
      "backend_type": "huggingface_embedding",
      "model_id": "sentence-transformers/all-MiniLM-L6-v2",
      "alias": "embed:minilm",
      "device": "auto",
      "dtype": "auto",
      "max_length": 512,
      "normalize_embeddings": true,
      "host": "0.0.0.0",
      "api_key": "instance-key"
    }
  }'

Starting an Instance

curl -X POST http://localhost:8001/instances/{instance-id}/start \
  -H "X-API-Key: your-secret-key-here"

Viewing All Instances

curl http://localhost:8001/instances \
  -H "X-API-Key: your-secret-key-here"

Stopping an Instance

curl -X POST http://localhost:8001/instances/{instance-id}/stop \
  -H "X-API-Key: your-secret-key-here"

API Endpoints

Instance Management

• POST /instances - Create new instance
• GET /instances - List all instances
• GET /instances/{id} - Get instance details
• PUT /instances/{id} - Update instance config
• DELETE /instances/{id} - Remove instance
• POST /instances/{id}/start - Start instance
• POST /instances/{id}/stop - Stop instance
• POST /instances/{id}/restart - Restart instance
• GET /instances/{id}/state - Get runtime state
• GET /instances/{id}/last-generation - Get last generation metrics

WebSocket

• WS /instances/{id}/logs - Stream logs with sequence numbers
• WS /instances/{id}/state - Stream runtime state updates

System

• GET /health - Health check
• GET /memory - GPU/RAM memory usage

Authentication

All requests require an X-API-Key header with your configured API key from the .env file.

Configuration Reference

llama.cpp Config Parameters

HuggingFace Causal LM Config Parameters

HuggingFace Classification Config Parameters

HuggingFace Embedding Config Parameters

Device Options

Selecting a GGUF by pattern

A pull resolves a model_source to a directory, but llama-server --model needs a file. model_file names it without hardcoding an absolute path:

{
  "backend_type": "llamacpp",
  "alias": "qwen3-vl:235b",
  "model_source": "huggingface://unsloth/Qwen3-VL-235B-Instruct-GGUF",
  "model_file": "*UD-Q4_K_XL*.gguf",
  "mmproj": "mmproj-BF16.gguf",
  "file_filters": ["*UD-Q4_K_XL*", "mmproj-BF16.gguf"]
}

The pattern is resolved against the pulled model directory: an absolute path is used as-is, then an exact relative path, then a glob at the directory root, and finally a recursive glob — so a bare filename or a broad pattern is still found inside a subfolder. Among multiple matches, the trailing shards of a split GGUF are dropped (llama-server loads them itself from ...-00001-of-000NN.gguf) and the largest remaining file wins. mmproj accepts the same patterns.

file_filters is a separate concern: it limits what gets downloaded (a file is kept when it matches any pattern), while model_file, mmproj and spec_draft_model decide which of the downloaded files each llama-server flag points at. Filters only apply to HuggingFace pulls — ORAS always pulls a Harbor artifact whole.

Multi-GPU hosts

n_gpu_layers decides how much of the model leaves the CPU; the four fields below decide where it goes on a host with several GPUs.

devices is the one to reach for when several instances share a host: it pins an instance to specific GPUs, so two models can run side by side without either of them seeing the other's VRAM. The names come from llama.cpp (llama-server --list-devices prints them, e.g. CUDA0, CUDA1), and leaving it empty offers every device to that instance:

{
  "backend_type": "llamacpp",
  "alias": "qwen3:32b",
  "model_source": "huggingface://org/Qwen3-32B-GGUF",
  "devices": "CUDA0,CUDA1",
  "split_mode": "row",
  "tensor_split": "3,1",
  "main_gpu": 0
}

split_mode chooses how the model is divided across whatever devices remain: "layer" (llama.cpp's default) pipelines layers and the KV cache, "row" splits weights by row and computes in parallel, "tensor" splits both and is still experimental, and "none" keeps the model on main_gpu alone.

tensor_split weights the division — "3,1" puts three quarters of the model on the first device — and is what unequal GPUs need. main_gpu selects the device that holds the model with split_mode: "none", or the KV cache and intermediate results with "row"; it does nothing for the other modes.

Speculative decoding

spec_type picks how the server drafts tokens ahead of the model. It only applies to model_type: "llm"; the flags are dropped for an embedding or reranker instance.

"draft-mtp" uses the served model's own Multi Token Prediction heads, so it needs nothing but a block size:

{
  "spec_type": "draft-mtp",
  "spec_draft_n_max": 2
}

"draft-dspark" drafts a whole block per forward pass with a separate DSpark model (a DFlash backbone plus a semi-autoregressive Markov head). It requires spec_draft_model, which is resolved exactly like mmproj — an absolute path, or a filename or glob looked up in the model directory:

{
  "backend_type": "llamacpp",
  "alias": "qwen3:4b",
  "model_source": "huggingface://org/Qwen3-4B-GGUF",
  "model_file": "*Q8_0*.gguf",
  "spec_type": "draft-dspark",
  "spec_draft_model": "*DSpark*.gguf",
  "spec_draft_n_max": 7,
  "file_filters": ["*Q8_0*", "*DSpark*"]
}

The drafter must be trained for the exact model being served (for example deepseek-ai/dspark_qwen3_4b_block7 for Qwen/Qwen3-4B), converted with convert_hf_to_gguf.py --target-model-dir so it inherits the target's tokenizer and embeddings. spec_draft_n_max is the block size and llama.cpp clamps it to the size the drafter was trained for — 7 for the published block7 checkpoints. The optional spec_draft_conf_min truncates each drafted block at the first position whose predicted acceptance falls below it, and only does anything for a drafter that carries a confidence head.

Since the drafter is a second file in the model directory, remember to widen file_filters to let it through on a HuggingFace pull.

Example Configurations

llama.cpp - Small Model

{
  "backend_type": "llamacpp",
  "model": "/models/llama-3-7b.gguf",
  "alias": "llama-3:7b",
  "threads": 4,
  "n_gpu_layers": 999,
  "temp": 0.7,
  "top_p": 0.9,
  "ctx_size": 8192,
  "api_key": "llama3-7b-key"
}

llama.cpp - Large Model with Custom Template

{
  "backend_type": "llamacpp",
  "model": "/models/gpt-oss-120b-F16.gguf",
  "alias": "gpt-oss:120b",
  "threads": 1,
  "n_gpu_layers": 999,
  "ctx_size": 131072,
  "chat_template_file": "/models/templates/harmony.jinja",
  "api_key": "gpt-oss-key"
}

HuggingFace - Text Generation

{
  "backend_type": "huggingface_causal",
  "model_id": "microsoft/phi-2",
  "alias": "phi-2:2.7b",
  "device": "cuda",
  "dtype": "float16",
  "max_length": 2048,
  "api_key": "phi2-key"
}

HuggingFace - Sentiment Classification

{
  "backend_type": "huggingface_classification",
  "model_id": "cardiffnlp/twitter-roberta-base-sentiment-latest",
  "alias": "sentiment:roberta",
  "device": "cuda",
  "max_length": 512,
  "labels": ["negative", "neutral", "positive"],
  "api_key": "sentiment-key"
}

HuggingFace - Embedding Model

{
  "backend_type": "huggingface_embedding",
  "model_id": "sentence-transformers/all-MiniLM-L6-v2",
  "alias": "embed:minilm",
  "device": "cuda",
  "max_length": 512,
  "normalize_embeddings": true,
  "api_key": "embed-key"
}

File Structure

solar-host/
├── .env                    # Configuration (not in git)
├── config.json             # Auto-generated instance storage (not in git)
├── logs/                   # Auto-generated log directory (not in git)
├── pyproject.toml          # Package metadata and dependencies
├── solar_host/
│   ├── backends/           # Backend runners
│   │   ├── base.py         # Abstract BackendRunner
│   │   ├── llamacpp.py     # llama.cpp runner
│   │   └── huggingface.py  # HuggingFace runner
│   ├── models/             # Pydantic models
│   │   ├── base.py         # Base models
│   │   ├── llamacpp.py     # llama.cpp config
│   │   └── huggingface.py  # HuggingFace configs
│   ├── servers/            # Standalone server processes
│   │   └── hf_server.py    # HuggingFace model server
│   ├── routes/             # API routes
│   ├── config.py           # Configuration management
│   ├── main.py             # FastAPI application
│   ├── models_manager.py   # Managed models directory and manifest
│   └── process_manager.py  # Process lifecycle management
├── tests/
└── README.md

Troubleshooting

Solar-host won't start

Error: "Address already in use"

• Another service is using port 8001
• Solution: Change PORT in .env or stop the other service

Error: "No module named 'solar_host'"

• The package is not installed
• Solution: pip install solar-host or pip install -e . for development

llama.cpp Instance fails to start

1. Verify llama-server is installed:

   which llama-server
   

2. Check model path:

   ls -lh /path/to/your/model.gguf
   

3. Check instance logs in logs/ directory

HuggingFace Instance fails to start

1. Verify dependencies:

   python -c "import torch; import transformers; print('OK')"
   

2. Check CUDA availability (if using GPU):

   python -c "import torch; print(f'CUDA: {torch.cuda.is_available()}')"
   

3. Check MPS availability (macOS):

   python -c "import torch; print(f'MPS: {torch.backends.mps.is_available()}')"
   

4. Check instance logs in logs/ directory

Instance keeps retrying and failing

• Solar-host will retry starting an instance up to 2 times
• Check the error_message field:

  curl http://localhost:8001/instances/{instance-id} \
    -H "X-API-Key: your-key" | jq '.error_message'
  

Conda Environment

When running solar-host from a conda environment, HuggingFace server subprocesses automatically inherit the same environment. Just ensure all dependencies are installed in your conda environment:

conda activate your-env
pip install torch transformers accelerate

Integration with Solar Control

Solar-host connects to solar-control over Socket.IO (namespace /hosts). Set SOLAR_CONTROL_URL and SOLAR_CONTROL_API_KEY in .env. On startup the host registers and appears in solar-control’s pending list until approved.

Approve the host (via solar-control management API or WebUI):

# List pending hosts
curl http://your-control-server:8000/api/hosts/pending \
  -H "X-API-Key: your-management-api-key"

# Approve (use pending_id from the list)
curl -X POST http://your-control-server:8000/api/hosts/pending/{pending_id}/approve \
  -H "X-API-Key: your-management-api-key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "GPU Server 1",
    "url": "http://192.168.1.100:8001",
    "api_key": "your-solar-host-api-key"
  }'

Alternatively, create a host directly (no pending step) with POST /api/hosts and the same JSON body.

Once approved, instances are accessible through solar-control’s OpenAI-compatible gateway:

• /v1/chat/completions - Chat completion (llamacpp, huggingface_causal)
• /v1/completions - Text completion (llamacpp, huggingface_causal)
• /v1/classify - Classification (huggingface_classification)
• /v1/embeddings - Embeddings (huggingface_embedding)

GPU Execution

Solar Host supports NVIDIA GPU execution for step containers via the gpu field on each step definition. GPU access requires the NVIDIA Container Toolkit to be installed and the Docker daemon to have the nvidia runtime registered.

Job JSON examples

{ "gpu": { "count": 1 } }

One GPU — Docker picks the device. Recommended default on multi-GPU hosts.

{ "gpu": { "count": -1 } }

All available GPUs (explicit replacement for the legacy gpu: true boolean).

{ "gpu": { "device_ids": ["0"] } }

Pin a specific GPU by index or UUID.

Note: "gpu": {} (empty object) is a validation error. At least one of count or device_ids must be provided.

NVIDIA environment variables

When a step requests GPU access, the following environment variables are injected automatically (callers may override them via step.environment):

Manual verification

To verify the NVIDIA Container Toolkit is working on a host without running a full job:

docker run --rm --gpus '"device=0"' nvidia/cuda:12.0-base nvidia-smi

Step Log Streaming

Solar Host captures stdout/stderr from every step container and makes them available in two complementary ways.

Data flow

Container stdout/stderr
        │
        ▼ (demuxed)
 JobStepExecutor._stream_logs()
        │
        ├──► JOBS_DIR/<job-id>/logs/<step>.log   (durable, combined)
        │
        └──► StepLogBuffer.append()
                  │
                  ├──► bounded in-memory deque  (latest 1000 lines)
                  └──► emit queue
                            │
                            ▼ (every 100 ms)
                  broadcast_step_log_batch()
                            │
                            ▼
                  Solar Control  →  "step_log" Socket.IO event

Event payload

Each step_log event carries an entries array. A normal log entry:

{
  "job_id":     "job-a1b2",
  "step_name":  "train",
  "step_index": 2,
  "stream":     "stdout",
  "seq":        42,
  "timestamp":  "2026-05-21T19:00:00.123456+00:00",
  "line":       "Epoch 1/10 loss=0.42"
}

The final entry for a step is a completion marker on the same event type:

{
  "job_id":     "job-a1b2",
  "step_name":  "train",
  "step_index": 2,
  "stream":     "stdout",
  "seq":        43,
  "timestamp":  "...",
  "line":       "",
  "completed":  true,
  "exit_code":  0
}

Using the same event shape for both regular lines and the completion marker means downstream consumers (S-032, Solar WebUI) only need to handle one event type.

Durability vs. real-time

The host-side log file is always written regardless of Solar Control connectivity. It is the authoritative record.

Reconnect behaviour

• SolarControlClient._emit is a no-op while disconnected — jobs never block or crash waiting for the WebSocket.
• The emit queue uses put_nowait with a 10 000-entry cap; entries are silently dropped when the queue is full (e.g. sustained disconnect + high-frequency logging).
• On reconnect, only new lines from that point forward are streamed; the durable log file can be read for historical lines.

Job Lifecycle Events (S-026)

Solar Host emits structured Socket.IO lifecycle events at every job and step state transition so Solar Control and downstream consumers (SuperNova, Solar WebUI) can monitor progress without polling.

Event catalog

host_id is the value received in registration_ack from Solar Control; it is null when the host has not yet registered.

retention_deadline is finished_at + retention_hours (ISO 8601 string).

error_summary for step_failed contains the last N lines of stderr captured in ContainerNonZeroExitError.last_stderr_lines.

Consistency guarantee

Each lifecycle event is emitted immediately after the matching JobStore mutation in the same coroutine. A consumer that receives a job_completed event can safely call GET /jobs/{id} and find the status already set to completed.

Fire-and-forget semantics

• Events are emitted directly by Socket.IO event name (e.g. sio.emit("job_started", data)) — no envelope, no batching.
• SolarControlClient.send_job_lifecycle is a no-op when disconnected; jobs never block waiting for the WebSocket.
• There is no retry queue for lifecycle events. State is always recoverable from GET /jobs/{id}.

Implementation details

solar_host/jobs/events.py        — payload builders + async emit_* functions
solar_host/ws_client.py          — send_job_lifecycle / broadcast_job_lifecycle
solar_host/jobs/executor.py      — job_started, job_completed, job_failed, job_cancelled
solar_host/jobs/step_executor.py — step_started, step_completed, step_failed

Backward Compatibility

Existing configurations without backend_type are automatically treated as llamacpp instances. No migration required.

Job REST API (S-027)

Intended callers: Solar Control only. These endpoints are not part of the public API surface.

Solar Host exposes three endpoints for submitting and managing containerised multi-step jobs. All endpoints require the X-API-Key header (same key as every other endpoint).

Endpoints

POST /jobs — Submit a job

Request body (JobDefinition shape):

{
  "job_id": "my-job-001",
  "name": "Fine-tune run",
  "steps": [
    {
      "name": "train",
      "image": "pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime",
      "command": ["python", "train.py"],
      "environment": {"EPOCHS": "5"},
      "gpu": {"count": 1}
    }
  ],
  "submission_id": "ctrl-sub-abc123",
  "correlation_id": "workflow-xyz"
}

• submission_id and correlation_id are optional strings Solar Control can use to correlate host jobs with its own records.
• job_id must be a simple alphanumeric slug (no path traversal characters).

Response — 202 Accepted:

{
  "job_id": "my-job-001",
  "status": "running",
  "workspace_path": "/var/solar/jobs/my-job-001",
  "submission_id": "ctrl-sub-abc123",
  "correlation_id": "workflow-xyz"
}

Error codes:

GET /jobs/{job_id} — Inspect a job

Response — 200 OK:

{
  "job_id": "my-job-001",
  "name": "Fine-tune run",
  "status": "running",
  "current_step_index": 0,
  "workspace_path": "/var/solar/jobs/my-job-001",
  "created_at": "2026-05-21T19:00:00.000000+00:00",
  "started_at": "2026-05-21T19:00:01.000000+00:00",
  "finished_at": null,
  "retention_hours": 24.0,
  "error_message": null,
  "submission_id": "ctrl-sub-abc123",
  "correlation_id": "workflow-xyz",
  "steps": [
    {
      "name": "train",
      "status": "running",
      "container_id": "abc123def456",
      "started_at": "2026-05-21T19:00:01.500000+00:00",
      "finished_at": null,
      "duration_s": null,
      "exit_code": null,
      "error_message": null,
      "log_file": "/var/solar/jobs/my-job-001/logs/train.log",
      "recent_logs": [
        {
          "seq": 1,
          "stream": "stdout",
          "line": "Epoch 1/5 loss=0.87",
          "timestamp": "2026-05-21T19:00:05.000000+00:00"
        }
      ]
    }
  ]
}

• log_file is the host-side path to the durable combined stdout/stderr log for that step.
• recent_logs is a tail of the in-memory log buffer (up to 100 entries). Real-time streaming is available via Socket.IO step_log events (S-025).

Error codes:

DELETE /jobs/{job_id} — Cancel a job

Performs synchronous cancellation per S-021 §6.3:

1. Signals the cancel event and stops the active container.
2. Waits up to 10 s for the run_job task to reach a terminal state.
3. Deletes the workspace directory.
4. Removes the job from the in-memory store and log buffer.

Response — 200 OK:

{"detail": "cancelled", "job_id": "my-job-001"}

Error codes:

Example curl commands

API_KEY="your-secret-key-here"
HOST="http://localhost:8001"

# Submit a job
curl -s -X POST "$HOST/jobs" \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "job_id": "demo-001",
    "name": "Demo job",
    "steps": [
      {
        "name": "hello",
        "image": "alpine:3.19",
        "command": ["echo", "hello from Solar Host"]
      }
    ]
  }' | jq .

# Poll state
curl -s "$HOST/jobs/demo-001" -H "X-API-Key: $API_KEY" | jq .status

# Cancel (if still running)
curl -s -X DELETE "$HOST/jobs/demo-001" -H "X-API-Key: $API_KEY" | jq .

Resource Reservations (S-034)

Solar Host exposes a host-local reservation ledger so Solar Control can pre-claim VRAM/RAM/disk capacity before placing a training job, without over-committing a host. All state is in-memory only — reservations do not survive a host restart.

Accounting formula

For each dimension d (VRAM, RAM, disk):

reported_usage_d = system.usage_d + Σ_i max(reservation_i.reserved_d − reservation_i.actual_d ?? 0, 0)
available_d      = total_d − reported_usage_d

system.usage_d is the live measured system usage (pynvml for VRAM, psutil for RAM, shutil for disk). For a pending reservation actual is null so the full reserved amount counts as headroom. For a running reservation only the unconsumed headroom max(reserved − actual, 0) is added, so real consumption captured in system.usage is never double-counted.

A reservation is running iff the linked job_id exists in the local JobStore with status == running.

Endpoints

All endpoints require the X-API-Key header (inherited from global middleware).

POST /resources/reservations

Request body (ReservationRequest):

{
  "job_id": "job-abc123",
  "workload_type": "training",
  "vram_gb": 16.0,
  "ram_gb": 8.0,
  "disk_gb": 50.0,
  "ttl_seconds": 3600
}

• ttl_seconds or expires_at may be specified (not both). When neither is given, a default TTL (default_reservation_ttl_seconds, 24 h by default) is applied so abandoned reservations can't hold capacity forever. This is a default, not a cap — Solar Control may set a longer-lived reservation by supplying its own ttl_seconds/expires_at.
• vram_gb/ram_gb/disk_gb must be >= 0; ttl_seconds (when given) must be > 0.
• Returns 201 ReservationView on success.
• Returns 409 {"error": "capacity_exceeded", "dimension": "vram", "requested_gb": …, "available_gb": …} when the request would exceed available capacity.
• Returns 422 for invalid request body.

GET /resources

Returns a ResourceSnapshot with per-dimension availability and the full reservation list including per-job actual usage (for running reservations):

{
  "memory_type": "VRAM",
  "vram": {"total_gb": 24.0, "system_used_gb": 8.0, "reserved_headroom_gb": 4.0, "reported_used_gb": 12.0, "available_gb": 12.0},
  "ram":  {"total_gb": 64.0, "system_used_gb": 12.0, "reserved_headroom_gb": 4.0, "reported_used_gb": 16.0, "available_gb": 48.0},
  "disk": {"total_gb": 500.0, "system_used_gb": 100.0, "reserved_headroom_gb": 20.0, "reported_used_gb": 120.0, "available_gb": 380.0},
  "reservations": [
    {
      "id": "res-<hex>",
      "job_id": "job-abc123",
      "workload_type": "training",
      "status": "pending",
      "vram_gb": 16.0,
      "ram_gb": 8.0,
      "actual_vram_gb": null,
      "actual_ram_gb": null,
      "expires_at": "2024-01-01T12:00:00Z"
    }
  ]
}

DELETE /resources/reservations/{id}

• Returns 200 {"detail": "released", "id": "<id>"} on success.
• Returns 404 for unknown IDs.
• Returns 409 when the linked job is currently running (running reservations must not be released while the job holds real capacity).

Background loops

Two background tasks are started alongside the existing health_report_loop:

host_health payload extension

The host_health Socket.IO event now includes a reservations block:

{
  "reservations": {
    "active_count": 2,
    "vram": {"total_gb": 24.0, "system_used_gb": 8.0, "reserved_headroom_gb": 4.0, "reported_used_gb": 12.0, "available_gb": 12.0},
    "ram":  {...},
    "disk": {...}
  }
}

Per-reservation details are omitted from the event (decision O4); use GET /resources to enumerate reservations.

Example curl commands

API_KEY="your-secret-key-here"
HOST="http://localhost:8001"

# Create a reservation
curl -s -X POST "$HOST/resources/reservations" \
  -H "X-API-Key: $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"job_id": "job-001", "workload_type": "training", "vram_gb": 16.0, "ram_gb": 8.0, "ttl_seconds": 3600}' | jq .

# Check current availability
curl -s "$HOST/resources" -H "X-API-Key: $API_KEY" | jq '{memory_type, vram: .vram.available_gb, ram: .ram.available_gb}'

# Release the reservation
curl -s -X DELETE "$HOST/resources/reservations/res-<id>" -H "X-API-Key: $API_KEY" | jq .