llm-omlx

llm-omlx is a plugin for LLM that discovers and uses text, vision, and embedding models served by oMLX. It uses oMLX's OpenAI-compatible API while presenting every model through LLM's public interfaces.

Installation

Install the plugin into the same environment as LLM:

llm install llm-omlx

Point it at one oMLX server. When OMLX_API_BASE is unset, the plugin reads the host and port from the local oMLX settings.json (under OMLX_BASE_PATH, default ~/.omlx), so a server managed by the oMLX app on the same machine works with no configuration. The API key is optional when the server allows unauthenticated requests:

export OMLX_API_BASE=http://127.0.0.1:8000/v1  # optional when oMLX runs locally
export OMLX_API_KEY=your-key  # optional

You can also store a key under LLM's omlx alias or provide --key for an individual command. Explicit keys take precedence over the stored alias and environment variable.

Usage

Refresh discovery and list the canonical omlx/ model IDs:

llm omlx models --refresh
llm models | grep '^omlx/'

Run text and vision prompts, or create an embedding:

llm -m omlx/your-model "Hello"
llm -m omlx/your-vlm -a image.png "Describe this image"
llm embed -m omlx/your-embedding-model -c "Embed this"

Model IDs come directly from the configured server and are always exposed as omlx/<provider-id>.

Discovery and offline behavior

The plugin first reads GET /v1/models/status. Models classified by oMLX as llm, vlm, or embedding are registered as text, vision, or embedding models respectively. Helper, hidden, reranker, audio, MarkItDown, and unknown special models are excluded.

If the status endpoint returns 404 or 405, discovery falls back to GET /v1/models. In that fallback catalogue, IDs containing embed (case-insensitive) are classified as embeddings; all other IDs become text-only chat models. Vision capability is never guessed from a model name.

A sanitized catalogue is cached in LLM's user directory for the configured API base. Fresh cache entries avoid repeated discovery. A matching stale cache can keep model listing usable when the server is offline; llm omlx models --refresh attempts the server first and uses the matching stale cache only if refresh fails. No model filesystem paths or raw status documents are persisted.

Model capabilities

Generation models support streaming, reasoning parts, conversations, tools, and structured JSON schemas. Actual tool calling and schema enforcement still depend on the selected model, its chat template, and the oMLX grammar configuration.

Only models classified as vlm accept attachments. Supported image MIME types are:

• image/png
• image/jpeg
• image/gif
• image/webp

Images may come from LLM-supported files, URLs, or bytes. Audio and video are not supported. Embedding models accept text only; binary and oMLX multimodal embedding items are not supported.

Configuration

Development

The project requires Python 3.10 or later and uses uv:

uv sync --locked --all-groups
uv lock --check
uv run ruff check .
uv run ruff format --check .
uv run pyright
uv run pytest
uv build
uv run twine check dist/*

Live acceptance tests are skipped by default. With an oMLX server running:

OMLX_API_BASE=http://127.0.0.1:8000/v1 \
OMLX_API_KEY=your-key \
uv run pytest --run-live -m live -q

See CONTRIBUTING.md for the contribution workflow.

License

Apache-2.0. See LICENSE.