local-embed

Standalone local embedding server — loads one GGUF (llama-cpp) or HF transformer embedding model once and exposes it as an OpenAI-compatible HTTP service on a single port: POST /v1/embeddings, the Models API (GET /v1/models, GET /v1/models/{id}), plus FastMCP tools.

Two backends: gguf loads a local .gguf file (llama-cpp-python); transformer loads an HF repo id (sentence-transformers). The server never downloads weights — you pre-download them (Model weights below).

Built for slife (its memdb and memfiles plugins call it over HTTP, so a model is loaded once per process tree); it is fully standalone — any OpenAI-compatible client works.

                POST http://127.0.0.1:8000/v1/embeddings
   slife memdb ─────────────────────────┐
   slife memfiles ──────────────────────┤
   any OpenAI client ───────────────────┤
                                        ▼
                            ┌────────────────────────────┐
                            │  local-embed               │
                            │  /v1/embeddings  (HTTP)    │
                            │  /v1/models      (HTTP)    │
                            │  /v1/models/{id} (HTTP)    │
                            │  /health         (HTTP)    │
                            │  /mcp  (FastMCP tools)     │
                            │  ONE loaded model          │
                            └────────────────────────────┘

Install

Requires uv (it manages the Python 3.13+ runtime). The transformer backend installs the same everywhere:

uv tool install 'local-embed[transformer]'   # sentence-transformers

gguf (llama-cpp-python) is the platform-sensitive one: PyPI ships Linux/macOS CPU wheels but no Windows wheels, so on Windows a plain install silently compiles from source (needs MSVC + CMake). The upstream project's prebuilt-wheel indexes (https://abetlen.github.io/llama-cpp-python/whl/<cpu|cuXXX|metal>) are the supported one-click path — pick the row for your platform:

When the index is an extra, uv uses its wheel only when PyPI has no compatible wheel for the platform (the Windows / Metal cases); on Linux PyPI does ship a CPU wheel, so to get the CUDA build you must use the CUDA index. Note the CUDA Linux wheels need glibc ≥ 2.35 (built for manylinux_2_35) — on an older distro uv falls back to a source build. If the gguf backend is missing, the server errors at startup with the exact command for your platform (the Windows CPU one on Windows).

Core deps are fastmcp + starlette; the model backends are optional extras. Installing does not fetch a model — get the weights first.

Model weights

The server never downloads models; both backends fail at load time when their weights are missing (gguf needs the gguf_path file, transformer needs the repo in the HF cache).

GGUF

.gguf files are, in general, community conversions — upstream releases are PyTorch weights, not GGUF — so there is no single authoritative source: conversions are scattered across Hugging Face, ModelScope, Ollama, llama.cpp community uploads and various project sites. Pick any source you trust; the transport doesn't matter, local-embed only needs the file on disk.

HF is the most common host and its CLI pulls a single file:

uv tool install "huggingface-hub[cli]"      # provides `hf`
hf download <owner>/<repo> <model>.gguf --local-dir D:\models\bge-m3
# one-off, no permanent tool install:
uvx --from huggingface-hub hf download <owner>/<repo> <model>.gguf --local-dir D:\models\bge-m3

Any source also works from a browser or wget/curl — on HF, every file is fetchable from https://huggingface.co/<owner>/<repo>/resolve/main/<model>.gguf.

Prefer a high-fidelity quant such as Q8_0 (~99 % of the original accuracy at roughly a third of the size). Point gguf_path at the file:

models: {
  "bge-m3": { backend: "gguf", gguf_path: "D:\\models\\bge-m3\\bge-m3-q8_0.gguf" }
}

Transformer

sentence-transformers resolves an HF repo id against the local hub cache:

hf download BAAI/bge-m3     # -> ~/.cache/huggingface/hub/models--BAAI--bge-m3

models: {
  "bge-m3": { backend: "transformer", model: "BAAI/bge-m3" }
}

Config: local_embed.json5

Everything — host, port, models, active model, backend — comes from local_embed.json5. Path resolution: $LOCAL_EMBED_FILE > slife project root (dev) > ~/.local-embed/local_embed.json5.

{
  active_model: "bge-m3",          // model served by default
  models: {
    "bge-m3": { backend: "gguf", gguf_path: "…", device: "" },
    "bge-m3-transformer": { backend: "transformer", model: "BAAI/bge-m3", device: "" }
  },
  env: {                            // injected into this process before any model loads
    HF_HUB_CACHE: "C:\\…\\HuggingFace\\hub",   // where transformer repos resolve
    HF_HUB_OFFLINE: "1"                        // force offline
  },
  host: "127.0.0.1",                // standalone only
  port: 8000                        // standalone only (slife plugin spawn binds a free port)
}

• models — map of name → {backend, gguf_path | model, device, max_tokens}.
• active_model — key into models.
• env: — injected before any backend loads; an existing shell env var wins. Without HF_HUB_CACHE, transformer repos resolve against the default cache and a model downloaded elsewhere is silently re-fetched.
• Single-model convenience — top-level backend/model/gguf_path/device — still works.

Run

local-embed                 # binds 127.0.0.1:8000 by default

The server CLI takes no model/endpoint flags — the config is the only source of truth.

CLI — configure models via subcommands

local-embed set (transformer) and local-embed set-gguf (gguf) upsert a model in the config, make it active, and pin port (and, for set, the HF cache). Both are idempotent — re-running yields the same config — and leave other models untouched.

local-embed set BAAI/bge-m3 [--HF_HUB_CACHE <dir>] [--port 8000]
local-embed set-gguf bge-m3 --path D:\models\bge-m3\bge-m3-q8_0.gguf [--port 8000]

Any error — cache unset, repo not in cache, file missing — exits non-zero and writes nothing. Changes apply on the next server start.

Use

curl http://127.0.0.1:8000/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model": "bge-m3", "input": ["hello world", "another text"]}'

Returns the standard OpenAI shape:

{
  "object": "list",
  "data": [
    {"object": "embedding", "index": 0, "embedding": [0.012, ...]},
    {"object": "embedding", "index": 1, "embedding": [...]}
  ],
  "model": "bge-m3",
  "usage": {"prompt_tokens": 3, "total_tokens": 3}
}

Any OpenAI-compatible client works — point base_url at http://127.0.0.1:8000/v1 (e.g. a slife embeddings provider, or the openai Python package):

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local")
vecs = client.embeddings.create(model="bge-m3", input=["hello"])

Other endpoints (OpenAI Models API):

• GET /v1/models — every configured model, each with its real embedding dimension (dimension / dimension_known), backend, load state, and the active flag the host uses to pick the active model.
• GET /v1/models/{id} — one model's detail (the OpenAI retrieve endpoint); 404 with a standard error envelope when the id is unknown.
• GET /health — {status, backend, model, dimension, loaded}.
• POST /mcp — FastMCP streamable-HTTP endpoint (internal tool __embed_status, probed by the host's check_local_embed). Like the sharefile plugin, local-embed is a service provider — the host consumes the model over the OpenAI-compatible HTTP routes, never through MCP tools.

Dimension

The real output width is only known once the model is loaded (n_embd / get_sentence_embedding_dimension). GET /v1/models (or /v1/models/{id}) reports the real dimension, so a client can size its vector table correctly — a wrong width silently drops every mis-sized embedding.

As a slife external plugin

slife treats every embedding model as a remote OpenAI-compatible endpoint — local-embed is just one such endpoint. The model is determined by the plugin's active model: slife discovers it from GET /v1/models (the entry flagged active: true) on load.

Register local-embed as an external plugin (like mcp-plugin) so slife manages the process, and point slife's embedding config at it with the unified OpenAI format:

plugins: {
  external: [
    { name: "local-embed", module: "local_embed.server" }
  ]
},
embeddings: {
  providers: {
    "local-embed": {
      base_url: "http://127.0.0.1:8000/v1",  // stable port from local_embed.json5
      api_key: "local",
    }
  },
  active_model: "local-embed",   // provider-id only, or "local-embed/<model>"
  enabled: true
}

The plugin binds the stable port from local_embed.json5 (default 8000), so slife's base_url is fixed whether local-embed runs as the plugin or standalone. When the service is unreachable, slife degrades gracefully to keyword search.

License

MIT — see the repository root LICENSE.