VaporRAM 💨

VaporRAM is a local inference server, CLI and web dashboard for google/gemma-4-E4B-it, packaged for consumer hardware.

It provides an OpenAI-compatible HTTP API, a terminal chat client, network sharing behind an API key, and a dashboard that reports only measured values. Token generation runs on llama.cpp.

Its research goal is to run the model under a 1.5 GB RAM ceiling by streaming transformer layers from NVMe rather than keeping them resident. The streaming reader — unbuffered O_DIRECT reads with aligned double buffers and kernel prefetch hints — is implemented in pure C under c/, and now reads the real byte ranges of each block resolved from the GGUF tensor directory. It is a measurement path, not the token path.

Current status (beta): the server, CLI, sharing and dashboard are feature-complete and tested. The RAM ceiling is not met — llama.cpp memory-maps the full GGUF, so measured RSS with the Q4_K_M weights is 6.8–8.1 GB depending on context size.

That is now a measured trade rather than an open question. At ~990 MB/s under O_DIRECT, streaming all 42 blocks for every token would cost ~2.5 s/token (~0.4 tok/s), against 6.8 tok/s with the weights resident — roughly 17x. Reaching the ceiling means paying that, and writing a full inference engine to do it. The dashboard measures both numbers on your own hardware; treat the ceiling as a goal the project has costed, not a feature that is nearly done.

Key Features

• Live Memory Telemetry: The dashboard and /v1/stats report the engine's actual measured RSS, host RAM and KV-cache projections — no estimated or placeholder figures.
• Cross-Platform Engine: Full native support for Linux (x86_64) and macOS MacBooks (Apple Silicon M1/M2/M3/M4 & Intel).
• Sequential Layer Pipeline (SLP): Zero-copy unbuffered O_DIRECT NVMe SSD layer streaming with asynchronous POSIX kernel prefetching hints (POSIX_FADV_WILLNEED).
• AVX2 SIMD & ARM NEON Kernels: Hand-written matrix-vector kernels in the C engine, with an OpenMP build (make -C c). Benchmark them on your own hardware with ./vapor bench.
• int8 Quantized KV Cache: Compresses Key & Value attention states with per-token scale factors (implemented in c/kv_cache.c; used by the C engine path).
• OpenAI-Compatible API: Built-in HTTP server supporting /v1/chat/completions, /v1/responses, /v1/models, and /health.
• Web UI & Interactive CLI: Includes an interactive terminal chat mode (vapor chat) and a web dashboard (vapor web).

Hardware & System Requirements

Installation

Option 1: Install via PyPI (Recommended)

pip install vapor-ram

Option 2: Prebuilt Release

Download and extract the latest prebuilt binary tarball:

mkdir vapor-ram && cd vapor-ram
tar xzf vapor-ram-v1.0.1-linux-x86_64.tar.gz

Option 3: Build from Source

Clone the repository and compile using make:

git clone https://github.com/sudsarkar13/vapor-ram.git
cd vapor-ram
make -C c

Usage Guide

The project includes a CLI launcher called vapor (./vapor or python3 vapor).

1. System Diagnostics & Resource Planning

Run diagnostics to check system capabilities and memory budget compliance:

# Run hardware diagnostic checks
./vapor doctor

# View RAM ceiling budget breakdown (< 1.5 GB)
./vapor plan

2. Interactive Terminal Chat

Launch an interactive chat session:

./vapor chat --preset coder

3. One-Shot Prompt Generation

Execute a quick single prompt generation from the command line:

./vapor run "Explain quantum computing in simple terms."

4. OpenAI-Compatible API Server

Start an HTTP server supporting OpenAI endpoints (/v1/chat/completions):

./vapor serve --host 0.0.0.0 --port 8000

Query the API using curl:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it",
    "messages": [{"role": "user", "content": "Hello! What can you do?"}]
  }'

5. Web Interface

Start the server and automatically launch the Web UI in your default browser:

./vapor web

vapor web binds 127.0.0.1 — the dashboard is reachable from this machine only. Add --share to open it to the rest of the network.

6. Sharing the Model Across Devices

vapor serve binds every interface, so any device on the same network can talk to the model. Because that also means anyone on that network can, the server requires an API key whenever it is not bound to loopback. The key is generated on first use and stored in ~/.vapor-ram/api_key (mode 0600), so clients configured once keep working across restarts.

./vapor serve            # shared on the LAN, key required
./vapor share            # print the URL, key and ready-to-paste clients

vapor share prints something like:

  ➜  Shared on LAN   : http://192.168.1.24:8000
  ➜  API for clients : http://192.168.1.24:8000/v1
  ➜  API key         : vr_8ZqK1sT4bN0pWm7xVhQrLdY2
  ➜  One-tap link    : http://192.168.1.24:8000/?key=vr_8ZqK1sT4bN0pWm7xVhQrLdY2

Clients may present the key three ways — pick whichever your client supports:

Any OpenAI client works by pointing base_url at the shared address:

from openai import OpenAI
client = OpenAI(base_url="http://192.168.1.24:8000/v1", api_key="vr_...")

Opening the one-tap link on another device loads the dashboard, stores the key in that browser, and strips it from the address bar. Visiting the bare host instead prompts for the key.

Key management

Access from outside the network

The LAN address only works locally. To reach the model from anywhere, put a TLS tunnel in front of the server rather than forwarding the port on your router — plain HTTP would send the API key across the internet in cleartext:

cloudflared tunnel --url http://localhost:8000
tailscale serve 8000            # private to your tailnet
ssh -R 8000:localhost:8000 user@your-vps

Then use the https:// address the tunnel prints as the base URL, with the same API key.

Configuration & Preset Flags

You can customize execution using presets or flags:

Weight Layout & Streaming

The Brain Cortex tab reads the GGUF tensor directory directly, so every figure it shows is a value in the file rather than an estimate. For gemma-4-E4B-it-Q4_K_M:

Press Measure to stream those real ranges through O_DIRECT, bypassing the page cache. On an NVMe SSD this reports ~990 MB/s and ~59 ms per block.

Performance

Measured on a Ryzen 7 5700U (8 physical cores / 16 threads, 15 GB RAM) with gemma-4-E4B-it-Q4_K_M.gguf, 32 decoded tokens per run:

Two things follow:

• Thread count dominates. One thread per hyperthread is ~3.4x slower than one per physical core, because llama.cpp's kernels already saturate each core's vector units. VaporRAM now defaults to physical cores; 4, 6 and 8 are within noise of each other, so there is little to gain by tuning further. VAPOR_N_THREADS overrides it.
• Context size costs memory, not speed. Decode throughput is unchanged between 4096 and 16384, but the KV cache grows ~1.3 GB. Pick the smallest window your conversations need — on a 16 GB machine, 16384 leaves little room for a browser and an editor, and once the host starts swapping, throughput falls much further than these figures suggest.

The weights are preloaded when the server starts, so the first message does not pay the ~5-9s load on top of its own generation.

Stopping the Server

CTRL+C stops the server, including while the engine is mid-generation. CTRL+\ works too. If neither reaches the process — some terminals do not deliver the signal to the foreground process — stop it from anywhere with:

./vapor stop                 # same path as the dashboard's Stop button

You can also type q and press Enter in the server's own terminal, or use the PID printed in the startup banner. Run with VAPOR_DEBUG_SIGNALS=1 to see which shutdown path is armed and whether the terminal can generate signals at all.

Project Structure

• c/vapor_engine: Streaming inspector. Given a GGUF and a plan of byte ranges, streams them through the O_DIRECT reader and reports measured per-block timings as JSON. It does not generate tokens.
• vapor_ram/gguf.py: GGUF container parser — tensor directory, shapes, quantisation types and exact byte ranges.
• vapor_ram/cortex.py: Resolves real layer ranges from the GGUF and drives the inspector.
• vapor: Main Python CLI frontend launcher.
• doctor.py: Installation and hardware diagnostic script.
• openai_server.py: OpenAI-compatible HTTP API server implementation.
• resource_plan.py: Dynamic memory budget calculation planner.
• version.py: Engine version information.
• web/: Frontend dashboard UI static assets.
• docs/: GitHub Pages documentation website and screenshot guides.

License

This project is licensed under the Apache 2.0 License. See the LICENSE file for details.