RAMP — Resource-Aware Model Proxy

Your local LLM should get out of the way when you need the RAM back.

RAMP is a daemon that sits in front of Ollama or llama.cpp and swaps your model for a smaller one when memory gets tight, then back when it frees up.

• Open Chrome with 40 tabs → your 7B quietly becomes a 3B.
• Close them → a minute later the 7B is back.
• You never pick a model size again, and nothing gets OOM-killed.

It speaks the OpenAI API, so your existing tools work unchanged.

Install

pip install ramp-llm

(pipx install ramp-llm to keep it isolated, or uvx ramp-llm to run it without installing anything.)

ramp doctor    # can this machine run it? if not, what to fix
ramp           # start it

That's the setup. No config file — RAMP finds your models, measures your RAM and GPU, and builds the ladder itself.

Commands

RAMP runs in the background, so you inspect it with separate commands rather than by watching a terminal.

Add -v to any of them for more detail; every command has --help.

How greedy should it be? By default RAMP keeps a slice of RAM free for the rest of your machine. To use more of it:

ramp --aggressive     # leave only ~500 MB free, climb back in ~20s
ramp --profile safe   # the opposite: keep plenty free, move slowly

Point your tools at it

Change one line:

- client = OpenAI(base_url="http://localhost:11434/v1")   # straight to Ollama
+ client = OpenAI(base_url="http://localhost:8090/v1")    # through RAMP

That's the whole integration — verified against the official openai Python SDK, including streaming and model listing. Ollama's native /api/* routes are proxied too, so tools written against Ollama work as well.

Or change nothing at all:

ramp --transparent

RAMP takes the port your model server already uses and moves that server one port over, so every tool you have routes through it with no config. Works with Ollama, llama.cpp and LM Studio. It asks first, proves the relocated server healthy before touching the original, rolls back on any failure, and puts everything back when it stops.

Try it without any models

ramp demo        # terminal 1
ramp watch       # terminal 2 - leave this visible
ramp stress      # terminal 3 - fill memory and watch the tier drop

Nothing is downloaded. DEMO.md walks through it, plus a second walkthrough using real models.

With real models

Install Ollama, pull two models of different sizes, and start:

ollama pull qwen2.5:1.5b
ollama pull qwen2.5:0.5b
ramp

RAMP builds the ladder from whatever you have. For llama.cpp instead, see examples/ramp.yaml; to hand-tune anything, run ramp init and edit the file it writes.

How it works

RAMP watches RAM, VRAM and free disk every few seconds and moves between models on its own:

• Drops fast when memory gets tight — hesitating is what freezes machines.
• Climbs back slowly, only once memory has been comfortably free for a while, so it can't flap up and down.
• Never mid-request: in-flight generations finish, new ones wait ~2s.
• Low disk blocks upgrades rather than causing downgrades, since a smaller model frees no disk.

A swap costs about 2 seconds (measured), and RAMP itself uses about 65 MB — which it reports, so you can check rather than trust.

📖 docs/CONCEPTS.md explains the mechanics properly: what actually consumes memory when an LLM runs, why VRAM pressure silently becomes RAM pressure, and the control theory behind the policy.

Monitoring

An elastic daemon lives or dies on one number: how often it actually swaps.

ramp status                        # human readable
curl localhost:8090/ramp/metrics   # Prometheus

Under 2 swaps/hour means RAMP is invisible, which is the goal. Above ~12 means churn worth tuning. docs/MONITORING.md covers what to watch and how to tune it; alert rules are in examples/monitoring/.

docker build -t ramp .
docker run --rm -p 8090:8090 \
  --add-host=host.docker.internal:host-gateway \
  -e RAMP_OLLAMA_URL=http://host.docker.internal:11434 \
  ramp

Why this doesn't already exist

Everyone running a local model has made the same bad trade: pick a big model and let the machine choke, or pick a small one and pay for the worst case all day. Existing tools make you choose once, up front, forever.

• Ollama / LM Studio pick a quantization once, at load time (ollama#14674).
• llama-swap swaps on the client's requested model, never on system state.
• FlexQuant / LSAQ / Voltron proved elastic execution works academically; none shipped a usable daemon.

RAMP is the missing controller: policy-driven, damped against thrashing, and measured — it reports its own swap rate so you can prove it isn't.

Prior art & acknowledgements

RAMP didn't invent elastic inference; it productizes an idea others established. Credit where it's due.

Research — FlexQuant (the closest statement of this exact problem), Any-Precision LLM (one weight file servable at several bit-widths — on the roadmap because of it), LSAQ (memory budget as the primary input), Voltron (scaling precision mid-generation), MoBiQuant, and PowerInfer / AirLLM for the complementary problem of fitting oversized models.

Tools — llama-swap is the direct inspiration for the shape of the solution: a transparent proxy in front of local inference servers. RAMP differs in what triggers a swap, but that architecture is its idea. Ollama and LM Studio proved local tooling has to be zero-configuration to get adopted.

Built on llama.cpp, Ollama, psutil, FastAPI, httpx and uvicorn. RAMP is a controller — it deliberately owns none of the hard parts of inference.

Contributing

See CONTRIBUTING.md. You need no model files to develop: the mock backend runs the whole daemon end to end in seconds.

Real /ramp/metrics output from your own machine is especially welcome — whether the tuning defaults are right is an empirical question, and more data settles it.

pytest -q                        # 155 tests
ruff check src tests scripts

License

MIT — see LICENSE.