VaporRAM 💨

VaporRAM is a local inference server, CLI and web dashboard for google/gemma-4-E4B-it, packaged for consumer hardware.

It gives you an OpenAI-compatible HTTP API, a terminal chat client, network sharing behind an API key, and a dashboard that shows only values it has measured. Token generation runs on llama.cpp.

The project's research goal is to run the model under a 1.5 GB RAM ceiling by streaming transformer blocks from NVMe instead of keeping them resident. That goal is not met, and this README says what is true instead.

What is actually true

Every figure below was measured on the machine described under Performance, with gemma-4-E4B-it-Q4_K_M.gguf. Reproduce them yourself with vapor bench, vapor doctor and the dashboard's Profiling tab.

If streaming ever did serve generation, this is what it would cost — also measured, on this machine's NVMe:

974.7 MB/s under O_DIRECT, 60.0 ms per block (52.3–85.5 ms). Streaming all 42 blocks for every token = 2.54 s/token (0.39 tok/s), against ~5 tok/s with the weights resident: roughly 13x slower.

That is the price of the 1.5 GB ceiling. It is a costed research goal, not a feature that is nearly finished.

Key Features

• Image input — the model can see. Attach images in the dashboard (or paste a screenshot), or send OpenAI content parts over the API. Needs the multimodal projector, a separate ~990 MB download.
• OpenAI-compatible API — /v1/chat/completions (streaming and not), /v1/responses, /v1/models, /health. Point any OpenAI SDK at it by changing base_url. Responses carry a real usage block.
• Reasoning you can read — gemma-4-E4B-it reasons natively. The dashboard shows the thought process as it streams, in a block you can expand and read, with four effort levels.
• Network sharing behind an API key — one command exposes the model to your LAN; the key is generated on first use and stored outside the repo at ~/.vapor-ram/api_key (mode 0600).
• Measured telemetry — the dashboard reports the server's real RSS, host RAM, the GGUF's actual tensor layout, and the throughput of your last generation. If a figure cannot be measured, it is not shown.
• GGUF introspection — a from-scratch parser reads the tensor directory, so the Brain Cortex tab shows byte ranges from the file rather than estimates.
• Streaming measurement path — a C reader streams the model's real block ranges with O_DIRECT and reports per-block timings, so the streaming design can be costed on your own SSD.
• Cross-platform — Linux (x86_64) and macOS (Apple Silicon and Intel). Tested in CI on both.
• Terminal client — vapor chat for an interactive session, vapor run for one-shot prompts.

Hardware & System Requirements

The 1.5 GB target does not describe what the software needs today. These are the real requirements for running it as it currently works:

On a 16 GB machine, n_ctx 16384 leaves little headroom. Once the host starts swapping, throughput falls much further than the figures below.

Installation

Option 1: PyPI (recommended)

pip install vapor-ram

Option 2: Prebuilt release

Download the tarball for your platform from the latest release:

tar xzf vapor-ram-v<version>-linux-x86_64.tar.gz
cd vapor-ram-v<version>

Pre-release builds (alpha, beta, rc) are published alongside stable and are listed on the releases page. Nothing is gated — take whichever you want.

Option 3: From source

git clone https://github.com/sudsarkar13/vapor-ram.git
cd vapor-ram
pip install -e .
make -C c          # optional: builds the streaming inspector and SIMD microbenchmark

Then get the weights

The model is not bundled — it is 4.98 GB, downloaded separately:

vapor download

Usage Guide

Installed from PyPI the command is vapor; from a checkout, ./vapor.

1. Diagnostics

vapor doctor        # hardware, dependencies, weights
vapor plan          # memory budget against the RAM ceiling target
vapor inspect       # GGUF tensor layout

2. Terminal chat

vapor chat --preset coder

3. One-shot prompt

vapor run "Explain quantum computing in simple terms."

4. OpenAI-compatible server

vapor serve --host 0.0.0.0 --port 8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "google/gemma-4-E4B-it",
    "messages": [{"role": "user", "content": "Hello! What can you do?"}]
  }'

5. Web dashboard

vapor web

vapor web binds 127.0.0.1 — reachable from this machine only. Add --share to open it to the network.

6. Sharing across devices

vapor serve binds every interface, so any device on the same network can talk to the model. Because that also means anyone on that network can, the server requires an API key whenever it is not bound to loopback. The key is generated on first use and stored in ~/.vapor-ram/api_key (mode 0600), so clients configured once keep working across restarts.

vapor serve            # shared on the LAN, key required
vapor share            # print the URL, key and ready-to-paste clients

vapor share prints something like:

  ➜  Shared on LAN   : http://192.168.1.24:8000
  ➜  API for clients : http://192.168.1.24:8000/v1
  ➜  API key         : vr_8ZqK1sT4bN0pWm7xVhQrLdY2
  ➜  One-tap link    : http://192.168.1.24:8000/?key=vr_8ZqK1sT4bN0pWm7xVhQrLdY2

Clients may present the key three ways — pick whichever your client supports:

Any OpenAI client works by pointing base_url at the shared address:

from openai import OpenAI
client = OpenAI(base_url="http://192.168.1.24:8000/v1", api_key="vr_...")

Opening the one-tap link on another device loads the dashboard, stores the key in that browser, and strips it from the address bar. Visiting the bare host instead prompts for the key.

Key management

Access from outside the network

The LAN address only works locally. To reach the model from anywhere, put a TLS tunnel in front of the server rather than forwarding the port on your router — plain HTTP would send the API key across the internet in cleartext:

cloudflared tunnel --url http://localhost:8000
tailscale serve 8000            # private to your tailnet
ssh -R 8000:localhost:8000 user@your-vps

Then use the https:// address the tunnel prints as the base URL, with the same API key.

Configuration & Commands

Settings live in vapor.json, written by the server whenever you change something in the dashboard or CLI. vapor.example.json is the tracked reference copy; vapor.json itself is deliberately not tracked in git so the repository cannot overwrite your settings. Set VAPOR_CONFIG_PATH to keep several independent configurations.

Reasoning

gemma-4-E4B-it supports reasoning natively: its chat template takes an enable_thinking flag, injects <|think|> at the top of the system turn, and emits the thought process inside <|channel>thought … <channel|> before the answer. VaporRAM enables it by default when the active model's template supports it, detected by reading the template out of the GGUF rather than assuming.

In the dashboard, reasoning appears above each reply in a Thinking block that animates while it streams and is open by default, so the working-out can be read as it arrives. Collapsing it is opt-in.

Four effort levels are available. The model's template has no effort parameter — it takes only a boolean — so these are VaporRAM's own: each level contributes a depth instruction placed with the thinking token at the top of the system turn, plus a reasoning-token budget shown in the UI.

vapor serve --no-think              # reasoning off for this server
vapor serve --think-level low       # or pick a level
vapor run --think "..."             # reasoning for one prompt
vapor chat                          # /think toggles mid-session

Per request, send {"thinking": false} or {"reasoning_effort": "medium"}. Over the API, reasoning streams on its own delta.reasoning_content field, so a client that does not know about it renders the answer alone rather than mixing the two.

Two things worth knowing. Reasoning shares the max_tokens budget with the answer, and on a hard question it can consume all of it — VaporRAM detects that and says so rather than returning an empty reply. And the model decides whether a question warrants reasoning: simple prompts are answered directly.

Images

gemma-4-E4B-it is multimodal, but the GGUF is a text-only conversion — none of its 720 tensors are vision or audio. The vision and audio towers live in a separate projector file:

vapor download --mmproj        # ~990 MB, fetched separately from the weights

With it installed, the dashboard grows an attach button beside the composer: multi-select, removable previews, and pasting a screenshot straight into the input. An image on its own is a valid message.

Over the API, send standard OpenAI content parts:

{"messages": [{"role": "user", "content": [
  {"type": "text", "text": "What is in this image?"},
  {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
]}]}

Ask what the server can accept before uploading anything large:

curl -s localhost:8000/health | jq .multimodal
# {"ready": true, "projector": "mmproj-F16.gguf",
#  "accepts": ["image", "audio", "video"]}

Sending media to a server without the projector is refused with a 400, not answered — the model would otherwise receive an image token with no image behind it and describe something that was never there.

Measured cost of enabling it on the reference machine: 0.44 GB of RSS (6.06 GB with, 5.61 GB with --no-mmproj, A/B under identical conditions). Processing an image did not raise it further. --no-mmproj on serve and web leaves an installed projector unused; VAPOR_MMPROJ overrides its location.

Audio and video are not wired up yet. The projector carries the audio tower and the control tokens are in place, but audio needs input decoding and resampling first. Sending either is refused rather than mishandled.

Weight Layout

The Brain Cortex tab reads the GGUF tensor directory directly, so every figure it shows is a value in the file rather than an estimate. The parser is validated by construction: the last tensor must end at exactly the file size, and for this model it does — byte 4,977,171,584 of a 4,977,171,584-byte file.

For gemma-4-E4B-it-Q4_K_M.gguf:

Weights by quantisation type, as bytes in the file:

Performance

Reference machine: AMD Ryzen 7 5700U (8 physical cores / 16 threads), 15 GB RAM, NVMe SSD, Linux. Weights gemma-4-E4B-it-Q4_K_M.gguf. Reproduce with vapor bench and the dashboard's Profiling tab.

Memory and startup

Doubling the context window costs 1.45 GB and does not change decode speed. Pick the smallest window your conversations actually need.

Because llama.cpp memory-maps the weights, RSS is not a fixed number: under memory pressure the kernel evicts mapped pages and RSS falls without the server misbehaving. The figures above are peaks with memory available.

Throughput

256-token generations, temperature 0:

4.0–5.3 tok/s across every configuration. Run-to-run variation on an idle laptop is larger than the difference between reasoning levels, so treat these as one range rather than a ranking. Reasoning does not slow generation down; it produces more tokens, so a reply takes longer to finish.

Thread count is the setting that matters. llama.cpp's kernels already saturate each core's vector units, so one thread per hyperthread is substantially slower than one per physical core. VaporRAM defaults to physical cores; VAPOR_N_THREADS overrides it.

Weights are preloaded at startup, so the first message does not pay the load cost on top of its own generation.

Streaming (vapor bench)

Streaming all 42 blocks through O_DIRECT, bypassing the page cache:

That last row is the cost of the 1.5 GB ceiling on this hardware: roughly 13x slower than the ~5 tok/s achieved with the weights resident. Reaching the ceiling means paying it, and writing a full inference engine to do it.

SIMD microbenchmark (vapor bench --simd)

A single-threaded dot product at the model's hidden size (2,560 floats), 200,000 iterations:

This measures the CPU, not the model. Token generation runs on llama.cpp and does not use this kernel.

Stopping the Server

CTRL+C stops the server, including mid-generation. CTRL+\ works too. If neither reaches the process — some terminals do not deliver the signal to the foreground process group — stop it from anywhere with:

vapor stop                 # same path as the dashboard's Stop button

You can also type q and press Enter in the server's own terminal, or use the PID printed in the startup banner. Run with VAPOR_DEBUG_SIGNALS=1 to see which shutdown path is armed and whether the terminal can generate signals at all.

Project Structure

License

Apache 2.0. See LICENSE.