mdl

A small CLI for running one local llama.cpp server at a time from a config file, instead of pasting flag soup into your shell.

llama-server -m /srv/models/Ornith-1.5-35B-A3B-Q4_K_M.gguf -ngl 99 \
  --n-cpu-moe 24 -c 65536 -fa on --cache-type-k q8_0 \
  --cache-type-v q8_0 -np 1 --port 8080

mdl run ornith

mdl.py is a single file, Python 3.11+ (needs tomllib), standard library only. Runs on Linux, macOS and Windows.

mdl_ui.py adds an optional terminal dashboard (mdl ui). It is the only part that needs a dependency — Textual — and the CLI never imports it, so every command but ui stays dependency-free.

Install

pipx install llama-mdl          # or: pip install llama-mdl
pipx install "llama-mdl[ui]"    # with the terminal dashboard

The package is llama-mdl; the command it installs is mdl. (Plain mdl on PyPI is an unrelated project.) Nothing but the dashboard has a dependency, and that is Textual.

Or run it straight from a clone - it is two files and a standard library:

git clone https://github.com/diverseau/llama-mdl ~/src/mdl
python ~/src/mdl/mdl.py --help

Then create a starter config:

mdl init

That writes ~/.config/mdl/models.toml, finds llama-server on your PATH if it is there, and tells you what to edit.

Config

~/.config/mdl/models.toml. One table per model; the table name is what you pass to mdl run.

# Optional. Defaults to "llama-server" on $PATH.
# $MDL_LLAMA_SERVER overrides this.
llama_server = "/opt/llama.cpp/build/bin/llama-server"

[ornith]
model = "/srv/models/Ornith-1.5-35B-A3B-Q4_K_M.gguf"
ngl = 99
n_cpu_moe = 24
ctx = 65536
flash_attn = true
kv_type = "q8_0"
parallel = 1
port = 8080

[qwen-small]
model = "/srv/models/Qwen3-8B-Q5_K_M.gguf"
ngl = 99
ctx = 16384
port = 8080

On Windows, write paths with forward slashes (C:/models/foo.gguf) or double the backslashes, since TOML treats \ as an escape character.

Keys

Two top-level keys sit outside the model tables: llama_server (above) and ready_timeout, the seconds run waits for /health before giving up. It defaults to 300, which a 70B on a slow disk can exceed.

Anything else in a model table is an error, so a typo like flash_atn tells you instead of silently doing nothing.

Commands

mdl run <name>   Start <name> in the background, tail its log until the
                 server answers /health, and exit. The server keeps running
                 after mdl exits.
mdl stop         SIGTERM the running server, SIGKILL after 10s, clean up.
mdl ps [--json]  name, pid, port and uptime, or "nothing running".
                 --json prints the state as JSON (null when idle) for
                 scripts and status bars.
mdl list         The models defined in the config.
mdl add <gguf>   Append an entry for a .gguf to the config, with sane
                 defaults. Takes an optional name and port.
mdl check        Validate every model in the config without launching
                 anything. Exits non-zero if it finds a problem.
mdl init         Write a starter config, if you do not have one.
mdl --version    The version, for bug reports.
mdl ui           The dashboard. Bare `mdl` opens it too.
mdl logs [-f]    Print the running server's log; -f follows it.
                 Takes a model name to read a stopped one's log.

Without textual installed, mdl ui fails with one line and bare mdl prints the usage string, exactly as it always did.

$ mdl list
ornith      /srv/models/Ornith-1.5-35B-A3B-Q4_K_M.gguf
qwen-small  /srv/models/Qwen3-8B-Q5_K_M.gguf

$ mdl run ornith
starting ornith (pid 48812), log /home/leon/.local/state/mdl/ornith.log
load_tensors: offloaded 43/43 layers to GPU
llama_context: n_ctx = 65536
main: server is listening on http://127.0.0.1:8080
ready: ornith on http://127.0.0.1:8080 (pid 48812)

$ mdl ps
ornith  pid 48812  port 8080  up 1h04m

$ mdl stop
stopped ornith (pid 48812)

add and check are the two that save the most time:

$ mdl add ~/models/Qwen3-8B-Q5_K_M.gguf
added [qwen3-8b-q5-k-m] to /home/leon/.config/mdl/models.toml
  Qwen3-8B-Q5_K_M.gguf (5.4G, 37 layers)
  run it with: mdl run qwen3-8b-q5-k-m

$ mdl check
ornith      ok
qwen-small  model file not found
mdl: 1 problem(s) found

add only appends, and check never launches anything, so both are safe to run against a config you care about.

The UI

mdl ui (or just mdl) opens a dashboard over the same config and the same state file. Anything you do in it is visible to the CLI and vice versa.

Idle, it lists your models with a status dot, shows the selected model's parameters, and previews the exact llama-server command it would run. e edits those parameters and saves them back to models.toml, leaving your comments and layout alone. Running, it swaps in live telemetry: VRAM, KV-cache use, a tokens/sec sparkline, busy slots, and a colour-coded log tail.

 key          does
 up/down, j k select a model
 enter, r     run the selected model
 s            stop the running server
 R            restart
 e            edit ngl / ctx / kv_type / port, saved to models.toml
 c            copy the llama-server command
 p            prompt the running model without leaving the UI
 l            focus the log, / filters it
 g            reload the config
 ?            help
 q            quit the UI - the server keeps running

Quitting never stops a server; s is the only thing that does.

The telemetry panels need llama.cpp's metrics endpoint, so add --metrics to a model's args to light them up:

args = ["--metrics"]

Without it the dashboard still works, and those panels say metrics off rather than failing. While a model is loading they say loading instead, since nothing is listening yet.

Talking to the model

p opens a chat with whatever is running, without leaving the UI.

It keeps the conversation, so follow-up questions have context; ctrl+l starts a fresh one. Reasoning is shown dimmed and timed separately, whether the server hands it back in its own field or inline as <think> tags. esc interrupts a running reply - it closes the socket rather than waiting for the next token - and closes the pane once nothing is streaming.

The rate is the server's own tok/s when it reports timings, and ours otherwise. ttft is time to first token, which is the number that tells you whether a long context is hurting.

Animation

The wordmark drifts its gradient by default. Set ui_fx = "off" at the top level of the config to paint it flat, or pass mdl ui --no-fx for a one-off.

Files

~/.config/mdl/models.toml        your config
~/.local/state/mdl/state.json    name, pid, port and start time of the server
~/.local/state/mdl/<name>.log    server stdout+stderr, rotated on each run
~/.local/state/mdl/<name>.log.1  the previous run, and .2 before that
~/.local/state/mdl/ui-marks.json which models the UI has seen start or fail

$XDG_CONFIG_HOME and $XDG_STATE_HOME are honoured if set. On Windows the same layout lives under %USERPROFILE%.

Behaviour notes

• One server at a time. run while something is up is an error telling you to stop first.
• Readiness is an HTTP probe, not log scraping. run polls /health on the configured port. llama.cpp has reworded its startup line between builds; this contract has not.
• Obvious mistakes fail before launch. A missing model file, a missing binary or a busy port is one line in milliseconds, not a failed model load.
• Stale state self-heals. If the pid in state.json is gone (crash, reboot, kill -9) the file is removed and ps reports nothing running.
• If the server exits during startup, run reports its exit status, removes the state file, and exits 1. The log has the reason.
• If it does not report ready in time, run exits 1 but leaves the server running, since it may still be loading. Check the log, or mdl stop. Raise ready_timeout if 300s is genuinely not enough.
• Pid reuse is guarded against. The state file records the OS process creation time, so a recycled pid is not mistaken for your server. macOS has no cheap way to read that, so it falls back to the pid alone.
• stop signals the process tree, not just the pid. If your llama_server is a wrapper script, killing the wrapper alone would orphan the real server and leave the port held.
• The last few logs are kept. <name>.log shuffles along to .1 and .2 on each run, so the crash you were not watching is still there.
• Errors are one line on stderr and a non-zero exit. No tracebacks.

Tests

python tests/run.py           # fast: no real model needed
python tests/run.py --live    # also drives a real model through the UI

The fast suites run against a temp config and a fake llama-server, so they never touch ~/.config/mdl. The POSIX process semantics (detaching, orphan self-heal, SIGTERM escalating to SIGKILL) need Linux:

docker run --rm -v "$PWD:/repo:ro" python:3.12-slim \
    sh -c 'cp -r /repo /w && cd /w && python3 tests/integration_posix.py'

Non-goals

These are deliberate, and issues asking for them will be closed with a link here. mdl starts one server, stops it, and tells you what is running.

• No daemon. Nothing runs in the background except the server itself.
• No multiple concurrent servers. These models are measured in gigabytes of VRAM; running two is usually a mistake, and refusing is a feature.
• No model downloading. Use huggingface-cli, or your browser.
• No hot-swap or auto-unload. mdl stop && mdl run other is two words.
• No web UI. llama-server already ships one.

If you want these, llama-swap and Ollama are good and solve different problems.

Contributing

See CONTRIBUTING.md. Short version: open an issue first, keep mdl.py free of dependencies, and run the tests.

python tests/run.py
ruff check .

CI runs the suites on Linux, macOS and Windows across Python 3.11-3.13, the pinned Textual floor and the current release, Ruff, the POSIX process suite, and a packaging check on both the wheel and the sdist.

Security issues go through SECURITY.md, privately, rather than the public tracker.

License

MIT. See LICENSE.