DiskLLM

Run a 7B LLM on 258MB RAM. Stock llama.cpp needs 6000MB. DiskLLM uses your SSD instead.

DiskLLM is an Ollama-style Python CLI and patched llama.cpp backend that keeps the KV cache on SSD through mmap instead of allocating the full cache in private RAM. It launches an OpenAI-compatible llama-server, downloads GGUF models from Ollama or HuggingFace, and persists context sessions across restarts.

Results

Quick Start

pip install diskllm
diskllm pull qwen2.5:7b
diskllm run qwen2.5:7b --session demo --headless
diskllm chat qwen2.5:7b --session demo --no-start

The server exposes an OpenAI-compatible API at http://127.0.0.1:8080/v1.

How It Works

Stock llama.cpp allocates KV tensors in RAM. At long context, that KV allocation can dominate system memory even when model weights are memory-mapped.

DiskLLM adds a patched llama.cpp --kv-backend ssd mode. The backend creates a CPU-addressable mmap file under ~/.diskllm/kv_cache, keeps only the active attention window hot, and lets the OS page the rest through NVMe storage. The patch is wired through dense KV, SWA, hybrid, and iSWA cache paths.

diskllm run always starts the patched server with:

-ngl 0
--kv-backend ssd
--kv-path ~/.diskllm/kv_cache
--kv-window 2048
--host 0.0.0.0
--port 8080
--no-repack

When --session NAME is used, DiskLLM also passes --slot-save-path ~/.diskllm/sessions and uses llama-server slot save/restore APIs to persist context as ~/.diskllm/sessions/<name>.mmap.

Read more in docs/how-it-works.md.

Benchmarks

Benchmarks were run CPU-only on Windows 11 with an Intel i5-12450H, 16GB RAM, and NVMe SSD.

See docs/benchmarks.md for the full tables and hardware notes.

Model Registry

qwen2.5:3b   bartowski/Qwen2.5-3B-Instruct-GGUF Q4_K_M
qwen2.5:7b   bartowski/Qwen2.5-7B-Instruct-GGUF Q4_K_M
qwen2.5:14b  bartowski/Qwen2.5-14B-Instruct-GGUF Q4_K_M
lfm2.5:8b    LiquidAI/LFM2.5-8B-A1B-GGUF Q4_K_M

DiskLLM can pull models from Ollama or HuggingFace. By default, diskllm pull tries Ollama first when the ollama CLI is installed, then falls back to HuggingFace. Force a source with:

diskllm pull qwen2.5:7b --source ollama
diskllm pull qwen2.5:7b --source huggingface

For HuggingFace GGUF repos, pass the repo name directly. DiskLLM lists repo files and picks Q4_K_M by default:

diskllm pull bartowski/Qwen2.5-7B-Instruct-GGUF
diskllm pull bartowski/Qwen2.5-7B-Instruct-GGUF:Q5_K_M

Storage

~/.diskllm/models/      downloaded GGUF files
~/.diskllm/kv_cache/    SSD KV mmap files
~/.diskllm/sessions/    persistent slot KV sessions
~/.diskllm/config.json  settings and installed model index
~/.diskllm/logs/        background server logs
~/.diskllm/bin/         patched llama-server binary

Override the home directory with DISKLLM_HOME.

Patched llama.cpp

The llama.cpp/ directory is tracked as a git submodule containing the patched fork. DiskLLM finds the patched server binary in this order:

1. ~/.diskllm/bin/llama-server.exe
2. DISKLLM_LLAMA_SERVER
3. llama_server in ~/.diskllm/config.json
4. bundled package path diskllm/bin/windows/llama-server.exe
5. development checkout path llama.cpp/build/bin/llama-server.exe

If a binary is found outside ~/.diskllm, DiskLLM copies it into ~/.diskllm/bin and runs that copy.

OpenAI Client

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="diskllm")
reply = client.chat.completions.create(
    model="diskllm",
    messages=[{"role": "user", "content": "What is DiskLLM?"}],
)
print(reply.choices[0].message.content)

More examples are in examples/.

Contributing

Contributions are welcome. Start with CONTRIBUTING.md, keep changes small, and include benchmark or test evidence for runtime changes.

License

DiskLLM is released under the MIT License. See LICENSE.