SiliconScavenger

A heterogeneous inference scheduler for CPU + Intel iGPU + NVIDIA dGPU — Ollama-compatible REST API, Python SDK, and CLI.

Developed by JBAC EdTech (Anubhav Choudhery & Jai Ansh Singh Bindra).

---

What it does

SiliconScavenger routes LLM inference requests across all three compute devices on your machine — CPU, Intel integrated GPU, and NVIDIA dGPU — concurrently, with:

• A lock-free C++ scheduler (Python never sits in the per-token hot path)
• An Ollama-compatible REST API (/api/generate, /api/chat, /api/tags, /api/embeddings, …)
• An OpenAI-compatible REST API (/v1/chat/completions, /v1/embeddings, …)
• Tool calling and JSON structured output on both API surfaces
• Session affinity — multi-turn chats pin to the same device for KV-cache reuse
• A live web dashboard showing per-device utilisation, queue depth, and session count
• A CLI mirroring Ollama's UX (scavenger serve, scavenger pull, scavenger create, …)

Measured performance (live run, Aug 2026)

Against an unmodified Ollama 0.30.10 install, same machine, same model (qwen2.5:3b Q4_K_M), 24-request concurrent trace:

System	TTFT p50	TTFT p95
SiliconScavenger CPU+iGPU	2 538 ms	46 388 ms
Ollama stock (CPU)	63 047 ms	89 219 ms
Ollama iGPU-enabled	71 203 ms	88 156 ms

24.8× faster p50 TTFT vs Ollama stock. iGPU was at 95–98 % utilisation throughout the Scavenger run — GPU-bound, not CPU-bound.

3-device simulation (CPU + iGPU + NVIDIA RTX 3060): +94.6 % p95 TTFT improvement when dGPU absorbs 56 % of requests.

---

Hardware requirements

Device	Backend	Required
x86 CPU	OpenVINO CPU	Always (minimum)
Intel iGPU (12th Gen Core+)	OpenVINO GPU + llama.cpp Vulkan	Recommended
NVIDIA dGPU (Ampere/Ada)	llama.cpp CUDA (sm_86+)	Optional

Minimum tested: Windows 11, Python 3.12, Intel Core i7-12650H.
Full dGPU support validated on NVIDIA RTX 3060 Laptop GPU (CUDA 12.6, driver 581.95).

---

Installation

Quickstart — server mode

pip install "siliconscavenger[server]"
scavenger serve
# Open http://127.0.0.1:11434/dashboard/

With audio/ASR support

pip install "siliconscavenger[server,audio]"

All extras

pip install "siliconscavenger[all]"

Note on the C++ extension: pip install triggers a CMake build of the native _scavenger_core pybind11 extension. You need:

• Visual Studio 2022 Build Tools (Windows) or GCC 12+ / Clang 15+ (Linux)
• CMake ≥ 3.26
• For iGPU: OpenVINO 2024.x runtime
• For dGPU: CUDA Toolkit 12.x + nvcc in PATH

---

CLI reference

All commands are available as scavenger <subcommand>:

Usage: scavenger {serve,run,pull,list,version,create,import-ollama}

scavenger serve — start the inference server

# Mock echo backend (no model needed, useful for API testing):
scavenger serve

# Real inference with a GGUF model:
scavenger serve --model path/to/qwen2.5-3b-instruct-q4_k_m.gguf

# Custom host/port:
scavenger serve --host 0.0.0.0 --port 8080

# Pre-flight device check (smoke-tests all registered slots, exits 1 if any fail):
scavenger serve --check-devices

# All options:
scavenger serve --help

scavenger list — list registered models

scavenger list
# NAME                SIZE    MODIFIED
# qwen2.5:3b          1840.5M 2026-08-02T20:46:09Z
# my-coder:v1         1840.5M 2026-08-02T20:46:38Z

scavenger pull — register a local GGUF model

scavenger pull /path/to/model.gguf
scavenger pull /path/to/model.gguf --name mymodel:v1

scavenger create — create a model from a Modelfile

# Modelfile syntax is a subset of Ollama's:
cat > Modelfile <<'EOF'
FROM qwen2.5:3b
SYSTEM You are a helpful coding assistant.
PARAMETER temperature 0.5
PARAMETER top_p 0.9
EOF

scavenger create my-coder:v1 -f Modelfile

scavenger import-ollama — import a model Ollama already downloaded

# Reads directly from ~/.ollama/models/blobs — no re-download:
scavenger import-ollama qwen2.5:3b
scavenger import-ollama qwen3:8b

scavenger version

scavenger version
# 0.7.0

---

REST API

The server starts on http://127.0.0.1:11434 by default (same port as Ollama).

Ollama-compatible endpoints

POST /api/generate — single-turn completion

curl http://localhost:11434/api/generate \
  -d '{"model":"qwen2.5:3b","prompt":"Why is the sky blue?","stream":false}'

{
  "model": "qwen2.5:3b",
  "response": "The sky appears blue because...",
  "done": true,
  "eval_count": 120,
  "routed_device": "igpu"
}

Streaming ("stream": true or omitted) sends newline-delimited JSON chunks, matching Ollama's wire format exactly.

JSON mode: add "format": "json" to enforce structured output.

POST /api/chat — multi-turn chat with session affinity

curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5:3b",
    "stream": false,
    "messages": [{"role":"user","content":"Hello! What can you do?"}]
  }'

{
  "message": {"role": "assistant", "content": "I can help with..."},
  "done": true,
  "session_id": "a1b2c3d4...",
  "routed_device": "igpu"
}

Session pinning: pass "session_id" from the previous response to pin subsequent turns to the same device:

curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5:3b",
    "stream": false,
    "session_id": "a1b2c3d4...",
    "messages": [{"role":"user","content":"Tell me more."}]
  }'

Tool calling:

curl http://localhost:11434/api/chat \
  -d '{
    "model": "qwen2.5:3b",
    "stream": false,
    "messages": [{"role":"user","content":"What time is it in Tokyo?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_time",
        "description": "Get current time for a city",
        "parameters": {
          "type": "object",
          "properties": {"city": {"type": "string"}},
          "required": ["city"]
        }
      }
    }]
  }'

POST /api/embeddings

curl http://localhost:11434/api/embeddings \
  -d '{"model":"qwen2.5:3b","prompt":"The quick brown fox"}'

GET /api/tags — list models (Ollama-compatible)

curl http://localhost:11434/api/tags

GET /api/ps — running models + active sessions

curl http://localhost:11434/api/ps

POST /api/create — create model from Modelfile (streaming)

curl http://localhost:11434/api/create \
  -d '{"name":"my-model:v1","modelfile":"FROM qwen2.5:3b\nSYSTEM You are helpful."}'

POST /api/pull, DELETE /api/delete, POST /api/show, POST /api/copy

Ollama-compatible model management endpoints.

---

OpenAI-compatible endpoints

Drop-in replacement for the OpenAI Python client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="not-needed",
)

response = client.chat.completions.create(
    model="qwen2.5:3b",
    messages=[{"role": "user", "content": "Explain gradient descent simply."}],
)
print(response.choices[0].message.content)

POST /v1/chat/completions

# Tool calling
response = client.chat.completions.create(
    model="qwen2.5:3b",
    messages=[{"role": "user", "content": "Get the weather in London"}],
    tools=[{
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get weather for a location",
            "parameters": {
                "type": "object",
                "properties": {"location": {"type": "string"}},
                "required": ["location"],
            },
        }
    }],
)

# JSON structured output
response = client.chat.completions.create(
    model="qwen2.5:3b",
    messages=[{"role": "user", "content": "List 5 colors as JSON"}],
    response_format={"type": "json_object"},
)

POST /v1/embeddings

response = client.embeddings.create(
    model="qwen2.5:3b",
    input="The quick brown fox",
)
print(response.data[0].embedding[:5])

GET /v1/models

curl http://localhost:11434/v1/models

---

Monitoring & admin endpoints

GET /health — overall health

curl http://localhost:11434/health
# {"status": "ok", "scheduler": "running", "mock": false}

GET /health/devices — per-device status

curl http://localhost:11434/health/devices
# {
#   "devices": {
#     "cpu":  {"degraded": false, "queue_depth": 0},
#     "igpu": {"degraded": false, "queue_depth": 2},
#     "dgpu": {"degraded": false, "queue_depth": 0}
#   }
# }

GET /telemetry — live utilisation

curl http://localhost:11434/telemetry
# {
#   "cpu":  {"util_pct": 42.0, "vram_used_mb": null},
#   "igpu": {"util_pct": 87.0, "vram_used_mb": 1240},
#   "dgpu": {"util_pct": 65.0, "vram_used_mb": 3072}
# }

POST /admin/recover-device/{device} — clear degraded flag

curl -X POST http://localhost:11434/admin/recover-device/igpu

---

Audio / ASR (requires [audio] extra)

pip install "siliconscavenger[server,audio]"
export SCAVENGER_WHISPER_MODEL_PATH=/path/to/whisper-base

# OpenAI-compatible multipart upload:
curl http://localhost:11434/v1/audio/transcriptions \
  -F file=@recording.wav \
  -F model=whisper-1

# Native JSON endpoint:
curl http://localhost:11434/api/audio \
  -d "{\"audio\": \"$(base64 -w0 recording.wav)\", \"language\": \"en\"}"

---

Python SDK

The siliconscavenger package exposes the C++ scheduler directly:

import siliconscavenger as ss

print(ss.version())          # "0.7.0"
print(ss.has_real_backends()) # True if built with real GGUF backends

# Create a scheduler (real inference requires --model path or env var)
scheduler = ss.Scheduler()

# Enqueue a task (CPU inference token array or embedding)
import numpy as np
task_id = scheduler.enqueue_task(np.array([1.0, 2.0, 3.0], dtype=np.float32))

# Retrieve result (blocks until done)
result = scheduler.wait_for_result(task_id)

# Live telemetry
tel = scheduler.get_telemetry()
print(tel["cpu"]["util_pct"])   # CPU utilisation %
print(tel["igpu"]["util_pct"])  # iGPU utilisation %
print(tel["dgpu"]["util_pct"])  # dGPU utilisation % (when CUDA build)

Session management (via the API)

import httpx

BASE = "http://localhost:11434"

# Start a session
r = httpx.post(f"{BASE}/api/chat", json={
    "model": "qwen2.5:3b",
    "stream": False,
    "messages": [{"role": "user", "content": "Hello"}],
})
session_id = r.json()["session_id"]
device = r.json()["routed_device"]   # "cpu" | "igpu" | "dgpu"

# Continue (pinned to same device for KV-cache reuse)
r2 = httpx.post(f"{BASE}/api/chat", json={
    "model": "qwen2.5:3b",
    "stream": False,
    "session_id": session_id,
    "messages": [{"role": "user", "content": "Tell me more"}],
})

---

Dashboard

Open http://127.0.0.1:11434/dashboard/ after scavenger serve:

• Device cards: CPU / iGPU / dGPU with live utilisation gauges and queue depth
• 3-lane timeline: per-device request history
• Session strip: active session count
• Links to /api/ps and /health/devices for raw data

---

Environment variables

Variable	Default	Effect
SCAVENGER_LLAMA_MODEL_PATH	—	Path to GGUF model; activates real inference
SCAVENGER_PREFER_DGPU	0	Set to 1 to bias first-turn routing to dGPU
SCAVENGER_WHISPER_MODEL_PATH	—	Path to OpenVINO Whisper model for ASR
SCAVENGER_KV_STATE_DIR	~/.scavenger/kv	Directory for KV session state files

---

Architecture

┌────────────────────────────────────────────────┐
│              Client Layer                       │
│  CLI  │  Ollama REST  │  OpenAI REST  │  UI     │
└────────────────┬───────────────────────────────┘
                 │  Python (FastAPI / asyncio)
        ┌────────▼─────────┐
        │  pybind11 bridge  │
        └────────┬──────────┘
                 │  C++17 native
        ┌────────▼──────────────────┐
        │  Scavenger Core            │
        │  lock-free task queue      │
        │  completion-time scoring   │
        │  3-device hw poller        │
        └────┬─────────┬────────┬───┘
             │         │        │
    ┌─────────▼─┐  ┌──▼──────┐  ┌▼────────────┐
    │ OpenVINO  │  │ Vulkan  │  │ CUDA/llama  │
    │ CPU+iGPU  │  │ iGPU    │  │ dGPU sm_86  │
    └───────────┘  └─────────┘  └─────────────┘

Python is never in the per-token hot path. The native C++ core owns the scheduler, queues, and memory manager.

---

License

MIT — Copyright (c) 2026 JBAC EdTech (Anubhav Choudhery & Jai Ansh Singh Bindra)