npuserver 1.2.0

A high-performance Flask server tailored for running Large Language Models (LLMs) locally on Intel NPUs using OpenVINO GenAI.

NPU Server (npuserver)

A high-performance Python library and Flask backend tailored specifically for running Large Language Models (LLMs) locally on Intel NPUs using OpenVINO GenAI.

This server provides an OpenAI-compatible API for seamless integration with existing tools, robust NPU memory management, and dynamic on-the-fly hardware compilation of Hugging Face models into optimized NPU blobs.

Core Features

• 🚀 OpenAI-Compatible API: Seamlessly integrate with any existing LLM tooling (like LangChain, AutoGen, or custom frontends) using the standard /v1/chat/completions endpoint. Fully supports real-time Server-Sent Event (SSE) streaming.
• 🧠 Strict Memory Management: The Intel NPU has limited, highly specialized memory. This server gives you complete explicit control over it. Load and unload models programmatically while aggressively garbage-collecting to prevent NPU memory leaks.
• ⚡ On-The-Fly Compilation: If a downloaded Hugging Face model hasn't been compiled for the NPU, the server intelligently intercepts the load request and dynamically compiles an optimized OpenVINO .blob before serving it.
• 🚫 No Background Downloads: To prevent runaway bandwidth usage and unexpected latency, the server strictly enforces that models must be downloaded locally before it attempts to load or compile them.
• 🌐 Static Model Registry: Ships with a fully decoupled, static HTML dashboard (serve/index.html) designed for easy hosting on GitHub Pages to cleanly display your available models without exposing backend connections.

---

📦 Installation

Ensure you have Python installed and your Intel NPU drivers configured properly on Windows.

# 1. Clone the repository
git clone https://github.com/durgasai299792458/npuserver.git
cd npuserver

# 2. Setup a virtual environment
python -m venv venv
venv\Scripts\activate

# 3. Install the package locally
pip install -e .

Required Core Dependencies: openvino-genai, flask, huggingface-hub

---

🛠️ Usage Guide

1. Starting the Server

The server runs on Flask. You can spin it up programmatically using the library:

from npuserver import run_server

# Starts the NPU backend on port 8080
run_server(port=8080)

2. Using the Python Client Library

You can remotely control the server's NPU memory directly from your Python scripts using the built-in client functions, without needing to write raw HTTP requests manually.

import npuserver

# 1. Fetch a list of all available models
models = npuserver.get_models_status(api_base_url="http://localhost:8080")
for m in models:
    print(f"{m['name']} is {m['status']}")

# 2. Explicitly load a model into NPU memory
print("Loading model...")
npuserver.load_model("durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i")

# 3. Completely wipe the NPU memory and free resources
npuserver.unload_model()

# 4. Delete only compiled OpenVINO NPU cache files for a model
npuserver.delete_compiled("durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i")

# 5. Completely delete compiled files AND downloaded HuggingFace snapshots
npuserver.delete("durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i")

3. Downloading Models Manually

Because the server strictly refuses to download gigabytes of data in the background, you must ensure the model exists in the cache first. You can download compatible models (like Qwen2.5-3B-OpenVINO-INT4-npu or gemma-4-E2B-OpenVINO-INT4) directly using Hugging Face utilities, or the server will read them if they are cached natively.

---

📡 HTTP API Reference

If you are building your own frontend or using standard REST clients (like curl or Postman), use these endpoints:

🧠 Model Memory Management

POST /load

Loads a specific model into NPU memory. If another model is currently active, it is safely ejected and garbage collected first. If the requested model is downloaded but uncompiled, it halts to compile it first.

curl -X POST http://localhost:8080/load \
     -H "Content-Type: application/json" \
     -d '{"model": "durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i"}'

POST /unload

Explicitly unloads the currently active model and triggers Python garbage collection to flush the NPU VRAM immediately.

curl -X POST http://localhost:8080/unload

GET /models

Returns a consolidated JSON array of all models (active, compiled, and available remotely on your GitHub Pages registry).

GET /health

Returns the server's status (e.g., whether it is idle or currently processing a generation task).

POST /delete or POST /v1/models/delete

Deletes a model's compiled cache files and/or downloaded Hugging Face weights from the server.

curl -X POST http://localhost:8080/delete \
     -H "Content-Type: application/json" \
     -d '{"model": "durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i", "compiled_only": true}'

DELETE /v1/models/<model_name>

Deletes a model's files from disk. Setting query parameter ?compiled_only=true deletes only the compiled blob folder.

curl -X DELETE http://localhost:8080/v1/models/durgasai299792458/Qwen3-4B-OpenVINO-INT4-npu-i?compiled_only=false

💬 Inference

POST /v1/chat/completions

A standard OpenAI chat endpoint. Strict Execution Note: You MUST call /load before sending a chat completion request. The server will reject the request with a 400 Bad Request if the NPU memory is empty.

curl -X POST http://localhost:8080/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "messages": [
         {"role": "system", "content": "You are a helpful AI assistant running on an Intel NPU."},
         {"role": "user", "content": "Write a short poem about microprocessors."}
       ],
       "max_tokens": 2048,
       "temperature": 0.7,
       "stream": true
     }'

---

📂 Cache & File Structure

By default, npuserver neatly organizes heavy model files in your user cache directory so your code repository stays clean and lightweight.

On Windows, models are stored at: C:\Users\<username>\.cache\npuserver\

• ...\hf\: Stores the raw weights and safetensors downloaded directly from the Hugging Face hub.
• ...\compiled\: Stores the optimized .blob files successfully compiled by OpenVINO GenAI.

Troubleshooting Tip: If a hardware compilation ever fails, gets interrupted, or becomes corrupted, simply navigate to the compiled\ directory, delete the specific model's folder, and the server will safely attempt to re-compile it from scratch on your next /load request!