llm-llamacpp-plugin 0.1.2

LLM plugin providing access to llama.cpp server models

llm-llamacpp

A plugin for LLM providing access to models running on a llama.cpp server.

Installation

Install this plugin in the same environment as LLM:

llm install llm-llamacpp-plugin

Setup

First, you need to have a llama.cpp server running. You can start one using the llama.cpp server binary:

# Download and build llama.cpp if you haven't already
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

# Build the server
make

# Start the server with your model
./llama-server -m models/your-model.gguf -c 4096

The server will start on http://localhost:8080 by default.

Usage

Once the plugin is installed and your llama.cpp server is running, you can use it like any other LLM model:

llm -m llamacpp "Your prompt here"

Using a different server URL

If your llama.cpp server is running on a different host or port, you can set the LLM_LLAMACPP_SERVER environment variable:

export LLM_LLAMACPP_SERVER=http://your-server:port

in windows

setx LLM_LLAMACPP_SERVER http://your-server:port

Conversations

You can use conversations just like with other models:

llm -m llamacpp "First message"
llm -c "Follow-up question"

JSON Schema

You can request JSON output using LLM's schema feature:

llm -m llamacpp-tools "Generate a person" --schema '{"name": "string", "age": "integer"}'

Vision Models

If you're running a vision-capable llama.cpp model with multimodal support, the plugin can handle image attachments.

llm -m llamacpp-vision -a /tmp/screen.png "What is the user doing in this screenshot"

Embedding Models

The plugin also supports embedding models running on llama.cpp server. To use embeddings:

# Start the server with embedding support
./llama-server -m models/embedding-model.gguf --embedding

Then use it with LLM:

# Get embeddings for text
llm embed -m llamacpp-embed "Hello world"

# Get embeddings for multiple items
llm embed -m llamacpp-embed "First text" "Second text"

Development

To install the plugin for development:

cd llm-llamacpp
pip install -e .

Run the tests:

pytest

License

Apache 2.0