LMRunner

LMRunner is a lightweight CLI for starting and managing local LLM inference endpoints backed by llama.cpp.

Installation

From source

git clone https://github.com/jschw/LMRunner.git
cd LMRunner
python -m pip install -e .

With pip

If you want to compile llama-server by yourself or just download the binaries, just do:

python -m pip install --upgrade lmrunner

You will have to set the path according in llm_server_config.json . Standard path of llama.cpp is /lmrunner/Llamacpp/llama.cpp/build/bin/llama-server .

If you want to install the python-bindings for llama.cpp (easier but may not up-to-date):

python -m pip install --upgrade lmrunner[llamacppbindings]

Run

./lmrunner

Commands

Commands are entered inside the interactive prompt and always start with /.