GPUStack Higress Plugins
Higress Proxy-Wasm plugins for GPUStack, providing AI API traffic processing, observability, and enhanced gateway features.
Overview
This repository contains custom Higress Proxy-Wasm plugins designed for GPUStack, distributed as a Python package that includes pre-compiled Wasm plugins and a built-in HTTP file server for serving them.
Installation
pip install gpustack-higress-plugins
Requirements: Python >= 3.10
Available Plugins
-
gpustack-token-usage - Collects and injects token usage statistics into AI API responses. For streaming responses: time to first token, time per output token, and tokens per second. For non-streaming responses: tokens per second only. Supports real client IP injection and path-based filtering.
-
gpustack-set-header-pre-route - Automatically injects the route name and model name into HTTP request headers before routing, based on configurable path suffixes or prefixes.
-
gpustack-ip-acl - Per-consumer source-IP blacklist/whitelist, mirroring the GPUStack Enterprise API key IP ACL semantics at the gateway. Rules are keyed by the Higress consumer identity (
x-mse-consumer, injected bygpustack-ext-auth);deniedCidrsalways take precedence overallowedCidrs, consumer keys support*wildcards, and invalid client IPs fail closed. Supports per-route overrides via HigressmatchRules. Deploy atAUTHN/350, immediately after ext-auth. -
gpustack-lb - The single binary of the LB plugin framework (
mode: context/mode: finisher), replacing model-mapper: publishes candidate instances, then picks one by weighted-sum ranking and steers the route viax-higress-target-cluster. -
gpustack-lb-session-affinity - LB capability plugin: keeps one session's requests on the same instance.
-
gpustack-lb-least-load - LB capability plugin: sends each request to the least-loaded instance.
-
gpustack-lb-decision-service - LB capability plugin: for task requests, asks a Jev-compatible system-1 decision service which candidate model should serve the task, and contributes a confidence-weighted rank entry (per-model probability scores) to the LB finisher.
See each plugin's README.md and example.yaml under extensions/ for full configuration and deployment details.
Filter-Chain Ordering
Plugins are positioned by phase (bucket order: AUTHN → … → UNSPECIFIED; buckets beat raw priority numbers) and priority (descending within a phase). The intended chain — all rejection points ahead of any scheduling-state work:
AUTHN 900 model-router → 810 transformer (strips spoofed identity headers)
→ 360 gpustack-ext-auth (injects trusted x-mse-consumer; 401)
→ 350 gpustack-ip-acl (403)
→ 340-325 LB band (context → session-affinity / prefix / least-load → decision-service (328) → finisher)
UNSPECIFIED 600 gpustack-rate-limit (429) → 400 gpustack-token-usage
→ 100 ai-proxy → router
When changing any plugin's position, re-check its ordering constraints (documented in each plugin's README) and verify the live filter chain via Envoy config_dump after rollout.
Usage
Start Plugin Server
# Start the built-in HTTP file server
gpustack-plugins start --port 8080
# Or with custom host
gpustack-plugins start --port 8080 --host 0.0.0.0
The server will be available at http://localhost:8080.
API Endpoints
# Health check
curl http://localhost:8080/
# Download a plugin
curl http://localhost:8080/wasm-plugins/gpustack-token-usage/1.0.0/plugin.wasm -o plugin.wasm
# Get metadata
curl http://localhost:8080/wasm-plugins/gpustack-token-usage/1.0.0/metadata.txt
Python API
from gpustack_higress_plugins import create_app, router
# Embed in an existing FastAPI app
app.include_router(router)
# Or create a standalone app
app = create_app()
Configure Higress WasmPlugin
apiVersion: extensions.higress.io/v1alpha1
kind: WasmPlugin
metadata:
name: gpustack-token-usage
namespace: higress-system
spec:
url: http://plugin-server:8080/wasm-plugins/gpustack-token-usage/1.0.0/plugin.wasm
defaultConfig:
realIPToHeader: x-gpustack-real-ip
Development
Prerequisites
- Go 1.24+
- Python 3.10+
- oras (
brew install oras) — required for fetching remote plugins
Build Plugins
# Install Python dependencies
make dev
# Build all plugins (local + remote, requires oras)
make build
# Build only local Go plugins (no oras required)
make -C extensions build-all
# Build specific plugin
make -C extensions build PLUGIN_NAME=gpustack-token-usage
If
orasis not installed,make buildwill build local plugins only and print a warning.
Run Tests
# Test Go plugins
make test
# Test single plugin
make -C extensions test PLUGIN_NAME=gpustack-token-usage
Check Wheel Contents
make verify-whl
Reports each expected plugin (from extensions/*/VERSION and remote_plugins.yaml) as ✓ present, ✗ missing, or version mismatch, and checks that manifest.json is included.
Deployment
Kubernetes (recommended)
Deploy the plugin server as a separate service and reference it from WasmPlugin resources:
# Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: gpustack-higress-plugins
spec:
template:
spec:
containers:
- name: plugins
image: gpustack/higress-plugins:latest
ports:
- containerPort: 8080
livenessProbe:
httpGet:
path: /
port: 8080
readinessProbe:
httpGet:
path: /
port: 8080
Docker Image
# Build Docker image
make image
# Build with custom Go proxy
GOPROXY=https://goproxy.cn,direct make image
# Run standalone
docker run -p 8080:8080 gpustack/higress-plugins:latest
Project Structure
gpustack-higress-plugins/
├── extensions/ # Go plugin source code
│ ├── gpustack-token-usage/
│ │ ├── main.go
│ │ ├── go.mod
│ │ └── VERSION
│ ├── gpustack-set-header-pre-route/
│ ├── gpustack-ip-acl/ # Per-consumer IP blacklist/whitelist
│ ├── gpustack-lb/ # LB framework (context/finisher roles)
│ ├── remote_plugins.yaml # Remote OCI plugin config
│ └── Makefile
├── gpustack_higress_plugins/ # Python package
│ ├── __init__.py
│ ├── main.py # CLI + FastAPI app factory
│ ├── server.py # /wasm-plugins router
│ ├── plugins/ # Compiled .wasm files (generated)
│ └── manifest.json # Plugin index (generated)
├── scripts/ # Build scripts
│ ├── generate_manifest.py
│ ├── generate_metadata.py
│ └── fetch_remote_plugins.py
├── Dockerfile
├── pyproject.toml
└── Makefile
Versioning
- Package version follows Semantic Versioning (MAJOR.MINOR.PATCH)
- Each plugin has its own version in
extensions/<name>/VERSION - Package version is set from the git tag at release time (placeholder
0.0.0in development) - RC releases (e.g.
0.2.0rc1) are published to TestPyPI; stable releases go to PyPI
License
Apache License 2.0
Metadata
Release files for gpustack-higress-plugins 0.5.2.post2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| gpustack_higress_plugins-0.5.2.post2-py3-none-any.whl | Python 3 | none | any | Details for gpustack_higress_plugins-0.5.2.post2-py3-none-any.whl |
Release files / gpustack_higress_plugins-0.5.2.post2-py3-none-any.whl
| Download URL | gpustack_higress_plugins-0.5.2.post2-py3-none-any.whl |
|---|---|
| Size | 26.6 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e75aa63175cc7a71012d8492d582164b3b981dc318a2f009d7aa312942a1f57d
|
|
BLAKE2b-256 checksum How to use checksums |
011b068ee69f58c9ccf004a20a6306e83b49070a705e9ccb97671239c1b30e3e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"22.04","id":"jammy","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Security for gpustack-higress-plugins 0.5.2.post2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.