Vipera

A local AI gateway that plans before it runs.

One .gguf in, an OpenAI-compatible HTTP server out — on hardware where the model does not quite fit.

The problem it solves

A 14B model on a 4 GB card looks like a CPU-only model. It is not: 17 of its 40 layers fit in VRAM at 8K context, and putting them there is the difference between 6.6 and 0.2 tokens/second.

Vipera measures the machine, budgets VRAM and RAM per layer, and hands llama.cpp a plan it can justify:

vipera system doctor        # what the machine actually has
vipera serve start          # plan, load, serve
curl localhost:8686/health  # {"status": "ok", "model": "..."}

Measured on a GTX 1650 4 GB, Hermes-4-14B Q3_K_M at 8K context:

The whole plan is printed before the model loads, and every flag in it is traceable to a measurement: planner.py carries the numbers in comments, docs/benchmarks.md carries the runs.

Install and run

pip install "vipera-ai[llama,gguf,nvidia]"   # llama-cpp-python builds for your platform
vipera model search                          # fetch a GGUF from Hugging Face
vipera serve start                           # → http://127.0.0.1:8686/v1

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8686/v1", api_key="none")
stream = client.chat.completions.create(
    model="qwen2.5-7b-instruct",  # the alias from GET /v1/models; omit to use the loaded model
    messages=[{"role": "user", "content": "Explain MoE in one paragraph."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Verified clients: openai-python, LangChain, LlamaIndex, Continue, the OpenAI Playground.

Should you use it?

Feature-by-feature with sources: docs/comparison.md.

How it decides

flowchart LR
    subgraph Client
        C1["openai-python"]
        C2["curl"]
    end

    subgraph Vipera["Vipera Gateway"]
        S["FastAPI server"]
        E["Engine"]
        P["Planner"]
        H["Hardware probe"]
        R["Registry"]
    end

    M[("GGUF file")]

    C1 & C2 -->|"HTTP / SSE"| S
    S -->|"chat()"| E
    E -->|"reads"| M
    P -->|"flags"| E
    H --> P
    R --> P

Concurrency model, fallback ladder, and known gaps: docs/architecture.md. The terminal's visual rules: docs/design/visual-system.md.

CLI

Every command, flag, and default: docs/cli.md.

HTTP API

curl http://127.0.0.1:8686/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "user", "content": "Hello"}]}'

Schemas, SSE framing, and Python/JavaScript examples: docs/http-api.md.

Configuration

Config lives at ~/.vipera/config.json. Change it with vipera system config or edit the JSON. The keys that change behavior most: profile (auto, cpu, gpu, lowmem, honored by serve, chat, and bench), ctx, kv_cache (f16, q8_0, q4_0), spec_type (draft-free n-gram speculation, on by default), default_model. Invalid values are repaired on load; vipera system config refuses to write them.

The optimization layer (shaping, tuning, both on) trims a request that is larger than the loaded context instead of letting llama.cpp cut the system prompt, and reuses the layer split a previous vipera bench measured on this machine. /health reports prefix-cache reuse. Details: docs/architecture.md.

Full table: docs/configuration.md.

What the numbers look like

Two findings from docs/benchmarks.md, both on consumer hardware:

• Quantize with an importance matrix, not blindly. Q3_K_M + imatrix is 18.7% smaller than Q4_K_M (7.32 vs 9.00 GB), loads 25% faster, prefills 27% faster, and lands within the eval-set error on perplexity (6.9805 vs 7.0056, a 0.11σ gap). Say "same quality, 19% lighter", not "better".
• kv_cache: q8_0 halves KV memory for about 3% decode throughput. q5_1 and iq4_nl cost 14–18%: use them only when KV size blocks the context you need.

Contributing

git clone https://github.com/vipera-iso/vipera
cd vipera
python -m venv .venv && .venv/bin/pip install -e ".[dev,llama,gguf]"
.venv/bin/pre-commit install
.venv/bin/pytest -q

CONTRIBUTING.md (dev setup, PR flow, demo GIF) · SECURITY.md (vulnerability reporting) · CHANGELOG.md (release history) · Code of conduct

MIT © 2026 Vipera — see LICENSE.