VaporRAM 💨

VaporRAM is a local inference server, CLI and web dashboard for google/gemma-4-E4B-it, packaged for consumer hardware.

Its goal is to run the model under a 1.5 GB RAM ceiling by streaming transformer layers directly from NVMe SSD storage. That streaming engine — unbuffered O_DIRECT reads with kernel prefetch hints and an int8 quantised KV cache — is implemented in pure C under c/.

Current status (alpha): token generation runs through llama.cpp, which memory-maps the full GGUF file. The C layer streamer is built but not yet wired into the token path, so the RAM ceiling is not yet achieved — measured RSS with the Q4_K_M weights is roughly 6 GB. The dashboard reports real measured RSS, so you can see this for yourself. Connecting the streaming engine to generation is the primary remaining work.

Key Features

• Live Memory Telemetry: The dashboard and /v1/stats report the engine's actual measured RSS, host RAM and KV-cache projections — no estimated or placeholder figures.
• Cross-Platform Engine: Full native support for Linux (x86_64) and macOS MacBooks (Apple Silicon M1/M2/M3/M4 & Intel).
• Sequential Layer Pipeline (SLP): Zero-copy unbuffered O_DIRECT NVMe SSD layer streaming with asynchronous POSIX kernel prefetching hints (POSIX_FADV_WILLNEED).
• AVX2 SIMD & ARM NEON Kernels: Hand-written matrix-vector kernels in the C engine, with an OpenMP build (make -C c). Benchmark them on your own hardware with ./vapor bench.
• int8 Quantized KV Cache: Compresses Key & Value attention states with per-token scale factors (implemented in c/kv_cache.c; used by the C engine path).
• OpenAI-Compatible API: Built-in HTTP server supporting /v1/chat/completions, /v1/responses, /v1/models, and /health.
• Web UI & Interactive CLI: Includes an interactive terminal chat mode (vapor chat) and a web dashboard (vapor web).

Hardware & System Requirements

Installation

Option 1: Install via PyPI (Recommended)

pip install vapor-ram

Option 2: Prebuilt Release

Download and extract the latest prebuilt binary tarball:

mkdir vapor-ram && cd vapor-ram
tar xzf vapor-ram-v1.0.1-linux-x86_64.tar.gz

Option 3: Build from Source

Clone the repository and compile using make:

git clone https://github.com/sudsarkar13/vapor-ram.git
cd vapor-ram
make -C c

Usage Guide

The project includes a CLI launcher called vapor (./vapor or python3 vapor).

1. System Diagnostics & Resource Planning

Run diagnostics to check system capabilities and memory budget compliance:

# Run hardware diagnostic checks
./vapor doctor

# View RAM ceiling budget breakdown (< 1.5 GB)
./vapor plan

2. Interactive Terminal Chat

Launch an interactive chat session:

./vapor chat --preset coder

3. One-Shot Prompt Generation

Execute a quick single prompt generation from the command line:

./vapor run "Explain quantum computing in simple terms."

4. OpenAI-Compatible API Server

Start an HTTP server supporting OpenAI endpoints (/v1/chat/completions):

./vapor serve --host 0.0.0.0 --port 8000

Query the API using curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it",
    "messages": [{"role": "user", "content": "Hello! What can you do?"}]
  }'

5. Web Interface

Start the server and automatically launch the Web UI in your default browser:

./vapor web

vapor web binds 127.0.0.1 — the dashboard is reachable from this machine only. Add --share to open it to the rest of the network.

6. Sharing the Model Across Devices

vapor serve binds every interface, so any device on the same network can talk to the model. Because that also means anyone on that network can, the server requires an API key whenever it is not bound to loopback. The key is generated on first use and stored in ~/.vapor-ram/api_key (mode 0600), so clients configured once keep working across restarts.

./vapor serve            # shared on the LAN, key required
./vapor share            # print the URL, key and ready-to-paste clients

vapor share prints something like:

  ➜  Shared on LAN   : http://192.168.1.24:8000
  ➜  API for clients : http://192.168.1.24:8000/v1
  ➜  API key         : vr_8ZqK1sT4bN0pWm7xVhQrLdY2
  ➜  One-tap link    : http://192.168.1.24:8000/?key=vr_8ZqK1sT4bN0pWm7xVhQrLdY2

Clients may present the key three ways — pick whichever your client supports:

Any OpenAI client works by pointing base_url at the shared address:

from openai import OpenAI
client = OpenAI(base_url="http://192.168.1.24:8000/v1", api_key="vr_...")

Opening the one-tap link on another device loads the dashboard, stores the key in that browser, and strips it from the address bar. Visiting the bare host instead prompts for the key.

Key management

Access from outside the network

The LAN address only works locally. To reach the model from anywhere, put a TLS tunnel in front of the server rather than forwarding the port on your router — plain HTTP would send the API key across the internet in cleartext:

cloudflared tunnel --url http://localhost:8000
tailscale serve 8000            # private to your tailnet
ssh -R 8000:localhost:8000 user@your-vps

Then use the https:// address the tunnel prints as the base URL, with the same API key.

Configuration & Preset Flags

You can customize execution using presets or flags:

Performance

Measured on a Ryzen 7 5700U (8 physical cores / 16 threads, 15 GB RAM) with gemma-4-E4B-it-Q4_K_M.gguf, 32 decoded tokens per run:

Two things follow:

• Thread count dominates. One thread per hyperthread is ~3.4x slower than one per physical core, because llama.cpp's kernels already saturate each core's vector units. VaporRAM now defaults to physical cores; 4, 6 and 8 are within noise of each other, so there is little to gain by tuning further. VAPOR_N_THREADS overrides it.
• Context size costs memory, not speed. Decode throughput is unchanged between 4096 and 16384, but the KV cache grows ~1.3 GB. Pick the smallest window your conversations need — on a 16 GB machine, 16384 leaves little room for a browser and an editor, and once the host starts swapping, throughput falls much further than these figures suggest.

The weights are preloaded when the server starts, so the first message does not pay the ~5-9s load on top of its own generation.

Stopping the Server

CTRL+C stops the server, including while the engine is mid-generation. CTRL+\ works too. If neither reaches the process — some terminals do not deliver the signal to the foreground process — stop it from anywhere with:

./vapor stop                 # same path as the dashboard's Stop button

You can also type q and press Enter in the server's own terminal, or use the PID printed in the startup banner. Run with VAPOR_DEBUG_SIGNALS=1 to see which shutdown path is armed and whether the terminal can generate signals at all.

Project Structure

• c/vapor_engine: Compiled C SIMD inference engine binary.
• vapor: Main Python CLI frontend launcher.
• doctor.py: Installation and hardware diagnostic script.
• openai_server.py: OpenAI-compatible HTTP API server implementation.
• resource_plan.py: Dynamic memory budget calculation planner.
• version.py: Engine version information.
• web/: Frontend dashboard UI static assets.
• docs/: GitHub Pages documentation website and screenshot guides.

License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.