llm-llamacpp

A plugin for LLM providing access to models running on a llama.cpp server.

Installation

Install this plugin in the same environment as LLM:

llm install llm-llamacpp-plugin

Setup

Building the Server

First, you need to have a llama.cpp server running. You can start one using the built server binary:

# Download and build llama.cpp if you haven't already
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build the server
make

# Start the server with your model
./llama-server -m models/your-model.gguf -c 4096

The server will start on http://localhost:8080 by default.

Usage

Once the plugin is installed and your llama.cpp server is running, you can use it like any other LLM model:

llm -m llamacpp "Your prompt here"

Using a different server URL

If your llama.cpp server is running on a different host or port, you can set the LLM_LLAMACPP_SERVER environment variable:

export LLM_LLAMACPP_SERVER=http://your-server:port

in windows

setx LLM_LLAMACPP_SERVER http://your-server:port

Model Management (Router Mode)

The plugin supports both single-model mode and router mode, where you can dynamically load, unload, and switch between models.

llama-server can serve multiple models with router mode.

CLI Commands

Use the llm llamacpp command to manage models:

# List available models
llm llamacpp list

# Load a specific model
llm llamacpp load <model-id>

# Unload a model
llm llamacpp unload <model-id>

# Switch to a different model
llm llamacpp switch <model-id>

# Show model information
llm llamacpp info <model-id>

Using Multiple Models

In router mode, you can switch between models dynamically:

# Load model 1
llm llamacpp load model1

# Use it
llm -m llamacpp "Your prompt here"

# Switch to model 2
llm llamacpp switch model2

# Use it
llm -m llamacpp "Your prompt here"

Per-Project Configuration

You can configure the server URL per project using .llm/llama-server.json:

{
  "url": "http://your-server:port"
}

Conversations

You can use conversations just like with other models:

llm -m llamacpp "First message"
llm -c "Follow-up question"

JSON Schema

You can request JSON output using LLM's schema feature:

llm -m llamacpp-tools "Generate a person" --schema '{"name": "string", "age": "integer"}'

Vision Models

If you're running a vision-capable llama.cpp model with multimodal support, the plugin can handle image attachments.

llm -m llamacpp-vision -a /tmp/screen.png "What is the user doing in this screenshot"

Embedding Models

The plugin also supports embedding models running on llama.cpp server. To use embeddings:

# Start the server with embedding support
./build/bin/server -m models/embedding-model.gguf --embedding

Then use it with LLM:

# Get embeddings for text
llm embed -m llamacpp-embed "Hello world"

# Get embeddings for multiple items
llm embed -m llamacpp-embed "First text" "Second text"

Development

To install the plugin for development:

cd llm-llamacpp
pip install -e .

Run the tests:

pytest

License

Apache 2.0