llm-ollama

LLM plugin providing access to models running on an Ollama server.

Installation

Install this plugin in the same environment as LLM.

llm install llm-ollama

Quickstart

By default, the plugin connects to a local Ollama server. Ensure the server is running and has some models available. Alternatively, the plugin can connect to a remote or managed Ollama cloud server; see Connecting to Ollama server for configuration instructions.

The plugin automatically discovers all models available on the server and registers them with LLM. To see the list of models and their capabilities, run:

llm ollama models

model                             digest          capabilities
gemma3:latest                     c0494fe00251    completion, vision
gpt-oss:120b-cloud                569662207105    completion, tools, thinking
mxbai-embed-large:latest          468836162de7    embedding
qwen3:4b                          2bfd38a7daaf    completion, tools, thinking

Once registered, models are available for prompting, chatting, and embedding under their Ollama names prefixed with ollama/. Assuming you have gemma3:latest available, you can run a prompt using:

llm -m ollama/gemma3:latest 'How much is 2+2?'

The plugin automatically creates shorter aliases for models that have :latest in the name, so the previous command is equivalent to running:

llm -m ollama/gemma3 'How much is 2+2?'

To start an interactive chat session instead of a one-shot prompt, run:

llm chat -m ollama/gemma3

Chatting with ollama/gemma3:latest
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
Type '!edit' to open your default editor and modify the prompt
Type '!fragment <my_fragment> [<another_fragment> ...]' to insert one or more fragments
>

Features

Attachments

Models that report the vision capability accept images, and models that report audio accept WAV and MP3 audio, using the LLM attachments option:

llm -m ollama/llava "Describe this image" -a https://static.simonwillison.net/static/2024/pelicans.jpg
llm -m ollama/gemma4:e2b "Transcribe this recording" -a recording.mp3

Tools

Ollama models with tools support can make use of LLM tools passed to them:

llm -m ollama/llama3.2 -T llm_time 'What is the time?' --td

The plugin also registers ollama_web_search and ollama_web_fetch tools that wrap the web search API provided by ollama.com. These tools augment models with the latest information to reduce hallucinations and improve accuracy. To use these tools, an API key must be configured (see Ollama cloud).

Embeddings

The plugin supports LLM embeddings with models that report the embedding capability, such as mxbai-embed-large:

llm embed -m ollama/mxbai-embed-large -i README.md

By default, the input will be truncated from the end to fit within the context length. This behavior can be changed by setting OLLAMA_EMBED_TRUNCATE=no environment variable. In such cases, embedding operation will fail if the context length is exceeded.

Reasoning

Ollama models with thinking support stream their reasoning trace alongside the final answer. The plugin surfaces it as a separate reasoning part on the response so it can be displayed, persisted, and round-tripped through follow-up turns.

Two knobs control reasoning behavior:

• -o think false disables the reasoning step entirely; the model skips thinking and goes straight to the answer.
• --hide-reasoning (or hide_reasoning=True in the Python API) keeps the reasoning step but suppresses the trace from the visible output. The model still thinks; only the trace is hidden.

JSON schemas

Ollama's built-in support for structured outputs can be accessed through LLM schemas, for example:

llm -m ollama/llama3.2 --schema "name, age int, one_sentence_bio" "invent a cool dog"

Async models

The plugin registers async LLM models suitable for use with Python asyncio.

To utilize an async model, retrieve it using llm.get_async_model() function instead of llm.get_model() and then await the response:

import asyncio, llm


async def run():
    model = llm.get_async_model("ollama/llama3.2:latest")
    response = model.prompt("A short poem about tea")
    print(await response.text())


asyncio.run(run())

Model aliases

The same Ollama model may be referenced by several names with different tags. For example, in the following list, there is a single unique model with three different names:

ollama list
NAME                    ID              SIZE    MODIFIED
stable-code:3b          aa5ab8afb862    1.6 GB  9 hours ago
stable-code:code        aa5ab8afb862    1.6 GB  9 seconds ago
stable-code:latest      aa5ab8afb862    1.6 GB  14 seconds ago

In such cases, the plugin will register a single model and create additional aliases. Continuing the previous example, this is what LLM will have:

llm models
...

Ollama: ollama/stable-code:3b (aliases: ollama/stable-code:code, ollama/stable-code:latest, ollama/stable-code)

Model options

All models accept Ollama modelfile parameters as options. Use the -o name value syntax to specify them, for example:

• -o temperature 0.8: set the temperature of the model
• -o num_ctx 256000: set the size of the context window used to generate the next token

See the referenced page for the complete list with descriptions and default values.

Additionally, the -o flag supports plugin-specific options:

• -o json_object 1 forces the model to reply with a valid JSON object. Note that your prompt must mention JSON for this to work;
• -o think false disables the intermediate reasoning step for thinking-capable models. See also the Reasoning feature section.

Connecting to Ollama server

The plugin connects to an Ollama server to list and run models. Three deployment scenarios are supported: a local server, a self-hosted remote server, and Ollama's hosted cloud service.

Local server

By default, the plugin connects to a local Ollama server at localhost:11434. If your local server runs on a non-default port, set OLLAMA_HOST:

export OLLAMA_HOST=http://localhost:8080

Remote server

To connect to a self-hosted Ollama server on your network or in the cloud, set OLLAMA_HOST to its address:

export OLLAMA_HOST=https://192.168.1.13:11434

Authentication

If the server is protected with Basic Authentication, include the credentials in the URL:

export OLLAMA_HOST=https://username:password@192.168.1.13:11434

Special characters in usernames or passwords must be URL-encoded:

# For username "user@domain" and password "p@ssw0rd"
export OLLAMA_HOST=https://user%40domain:p%40ssw0rd@192.168.1.13:11434

If the server is behind a reverse proxy that requires custom headers, use the OLLAMA_HEADERS environment variable with a comma-separated list of key=value pairs:

# JWT token auth (e.g. Open-WebUI's Ollama endpoint)
export OLLAMA_HEADERS='Authorization=Bearer mytoken,User-Agent=custom-client'

# Cloudflare Tunnel with a Service Token
export OLLAMA_HEADERS='CF-Access-Client-Id=abcdef.access,CF-Access-Client-Secret=123456789'

Ollama cloud

Ollama cloud is a hosted service that lets you run models without installing or operating a local server. It offers a range of open models and does not require any local setup beyond configuring the plugin.

To use it, point the plugin at the cloud endpoint and provide your API key:

export OLLAMA_HOST=https://ollama.com
llm keys set ollama # paste your API key when prompted

The API key is stored securely by llm and used automatically. Alternatively, you can set it as an environment variable:

export OLLAMA_API_KEY=your-api-key

A key can also be supplied per call with --key, which overrides both of the above:

llm -m ollama/gpt-oss:120b-cloud --key your-api-key "How much is 2+2?"

Development

Setup

To set up this plugin locally, first checkout the code. Then create a new virtual environment and install the dependencies. If you are using uv:

cd llm-ollama
uv sync --extra test --extra lint

Otherwise, if you prefer using standard tools:

cd llm-ollama
python3 -m venv .venv
pip install -e '.[test,lint]'

Code layout

llm_ollama/
  __init__.py     re-exports the hooks, nothing else
  registry.py     model hooks, discovery, capability cache
  commands.py     the `llm ollama models` command
  tools.py        the plugin's own tools
  auth.py         client creation from OLLAMA_HOST, OLLAMA_HEADERS and the API key
  cache.py        on-disk cache decorator
  models/
    base.py       BaseMixin
    chat.py       chat models, conversion of any llm.Tool for a chat request
    embed.py      embedding model

• A module owns its @llm.hookimpl functions, and __init__.py re-exports them.
• Imports run one way: __init__ → registry, commands, tools; commands → registry; registry → models, auth, cache; models → auth; tools → auth. Nothing imports from __init__.
• Inside models/, family modules import from base.py and never from each other.
• A name that another module of the package imports is public, with no leading underscore. Tests may import private names.

Each concrete model class lists a mixin of ours first, so that it overrides __str__ and get_key, followed by one llm base class:

BaseMixin                      base.py; __str__, get_key
├─ ChatMixin                   chat.py; options, attachment types, message building
│  ├─ OllamaChatModel          + llm.KeyModel
│  └─ AsyncOllamaChatModel     + llm.AsyncKeyModel
└─ OllamaEmbeddingModel        embed.py; + llm.EmbeddingModel

Testing and linting

To test or lint the code, first activate the environment:

source .venv/bin/activate

The environment includes llm; it will pick up the local version of the plugin, which is useful for manual testing.

To run automated unit and integration tests:

python -m pytest

Unit tests live in tests/unit and mock the Ollama server. Integration tests live in tests/integration and need a running Ollama server. They are grouped into kinds, one per model capability they test, named as Ollama reports it:

In Ollama's terms, completion means the model generates text; it covers the chat API, which the plugin uses.

The tests of each kind will be:

• Enabled automatically if the Ollama server has the model and the model reports the capabilities it must support;
• Skipped, with the reason, otherwise;
• Force-enabled with --<kind> (but fail if they cannot be enabled);
• Force-disabled with --no-<kind>.

To test with another model, pass --<kind>-model, for example --thinking-model qwen3:14b. The thinking model must be able to turn thinking off: gpt-oss reasons even with think set to false, so it fails the thinking tests.

Unit tests mirror the package one to one: the tests for llm_ollama/models/chat.py are in tests/unit/models/test_chat.py. Integration tests go through llm.get_model() or llm.get_embedding_model() and exercise several modules at once, so each is named after its kind instead: test_<kind>.py. Each integration file marks its kind once with pytestmark and gets the model ID from the model_id fixture. To add a kind, add a row to MODEL_KINDS in tests/conftest.py, which generates its marker and options.

To lint and format the code:

python -m ruff check --fix .
python -m ruff format .