eeANE

embedding engine for Apple Neural Engine

eeANE runs text embedding and reranking models on the Apple Neural Engine (ANE) of Apple Silicon Macs. Models are taken as-is in their Hugging Face distribution form and compiled locally into Core ML artifacts that load in seconds and run on the ANE — keeping your GPU and most of your unified memory free for other work.

Highlights

• ANE inference: embeddings at up to ~13,600 effective tokens/s on an M2 Mac mini, 2–3x the same model served from the MPS GPU by PyTorch — while leaving the GPU idle (see Performance below).
• No model modification, no re-distribution: eeane compile takes a Hugging Face model ID (or a local directory in HF distribution form) and converts it on your machine, then verifies the result against the FP32 original with a built-in self-check.
• Standard APIs: OpenAI-compatible /v1/embeddings and Infinity-compatible /rerank, so existing clients (Open WebUI among them) connect by changing a base URL.
• Cheap to keep running: models load on demand in well under a second and unload after an idle timeout; the always-on server itself is a small Python process with five runtime dependencies (no torch, no transformers).
• Multi-model serving with per-request routing, admission control (429/503 + Retry-After), identical-request coalescing, and graceful shutdown.
• Three architecture families supported today: ModernBERT and XLM-RoBERTa (both embedding and cross-encoder reranker models), and BERT (embedding models only). More are planned.

Requirements

• Apple Silicon Mac (M1 or later)
• macOS 13 or later
• Python 3.11 or 3.12 (3.13 and later are not yet supported). uv resolves a matching interpreter automatically; installing with pipx or pip + venv instead means providing one yourself.
• Xcode Command Line Tools (xcode-select --install) — eeane compile uses xcrun coremlcompiler
• uv — the recommended way to install eeANE (see Installation below), and required for the development workflow

eeANE requires the Apple Neural Engine: CPU-only execution is not supported (see Known limitations). Docker is not supported either — containers on macOS run inside a Linux VM, and the ANE is not passed through to it.

Installation

eeANE is on PyPI. The [compile] extra adds torch/transformers, which only eeane compile needs; the combined install below gives one environment that can both compile models and serve them.

uv (recommended)

uv tool install "eeane[compile]"

To upgrade later, run uv tool upgrade eeane.

pipx

pipx install --python python3.12 "eeane[compile]"

pipx's default Python interpreter may be 3.13 or later, which eeANE does not yet support, so pass --python naming a Python 3.11 or 3.12 executable available on your machine (for example python3.11, or a full path to one).

pip + venv

python3.12 -m venv eeane-env
eeane-env/bin/pip install "eeane[compile]"

Substitute python3.11 if that is the supported interpreter you have available instead.

Lightweight install (server only)

The [compile] extra pulls in torch and transformers, but the server itself never imports them — they are needed only when running eeane compile to convert a model into Core ML artifacts. Keeping them installed alongside the server costs disk space (a few GB) but has no effect on the server's memory use or behavior, so the combined install above is a reasonable default. If you would rather keep the always-installed environment down to eeANE's five runtime dependencies, install eeANE without the extra and run eeane compile from a disposable environment instead:

uv tool install eeane
uvx --from "eeane[compile]" eeane compile <model>

Installing from GitHub

To install a development snapshot or pin an exact repository revision, install from a git URL instead of PyPI — with any of the tools above, for example:

uv tool install "eeane[compile] @ git+https://github.com/xhighhongo41/eeANE@main"

@main tracks the latest development version; @v1.0.0 (or any other release tag) pins a released revision. To switch an existing install, run the same command with --force.

Quick start

# Compile models straight from their Hugging Face IDs (auto-downloaded)
# or from local directories in HF distribution form. One-time; the
# artifacts land under ~/.cache/eeane/ and each bucket takes ~30-100 s:
eeane compile cl-nagoya/ruri-v3-310m
eeane compile cl-nagoya/ruri-v3-reranker-310m
eeane compile intfloat/multilingual-e5-base

# Each run ends with a ready-made [[models]] TOML snippet on stdout.
# The snippet is minimal -- usually just the model id -- because the
# server resolves everything else from the compiled-model cache. Paste
# the snippets into ./eeane.toml (see eeane.example.toml), then start
# the server:
eeane serve

Then, from another shell:

curl -s http://127.0.0.1:7997/health

curl -s http://127.0.0.1:7997/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model": "intfloat/multilingual-e5-base", "input": "hello eeANE"}'

About eeane compile

eeane compile picks the model backend from the model's config.json. Three architecture families are supported: ModernBERT and XLM-RoBERTa (both embedding and cross-encoder reranker models), and BERT (embedding models only — a BERT cross-encoder reranker is rejected instead, because the compiled graph would have to pin its segment ids to zero, which changes the meaning of a query/document pair for this architecture). RobertaModel-architecture models are routed to the XLM-RoBERTa backend as well — transformers implements RoBERTa and XLM-RoBERTa as the same encoder, differing only in vocabulary. More families are planned. For embedding models, all three backends detect the mean/CLS pooling declared by the model directory's sentence-transformers 1_Pooling/config.json and compile the matching graph; an embedding model that does not declare a supported pooling mode is rejected with an error rather than compiled on a guess, because an artifact built with the wrong pooling still looks plausible while returning vectors with a different meaning. Rerankers are unaffected by this declaration — their pooling is part of the model's own classification head, not a separate sentence-transformers module. The compile log names the pooling it detected for each embedding model. See Verified models for the models that have been run end to end.

A sentence-transformers model directory may also declare a Dense module chain in its modules.json: eeane compile supports the Transformer -> Pooling -> Dense (zero or more) -> Normalize (optional, trailing) sequence, baking each declared Dense linear projection (Identity or Tanh activation) into the compiled graph and following the output width to match — a model with a Dense module ends up with a different embedding width than its raw hidden size. A module chain this sequence cannot describe is rejected with an error before any weights are read, rather than silently compiling something with a different meaning. The Normalize module itself is not compiled into the graph — the server applies L2 normalization instead, according to each model's normalize setting (on by default; see API). Dense checkpoints follow the same safetensors-first policy as the model's main weights (--allow-pickle for .bin-only Dense checkpoints; see Checkpoint formats).

The compiler detects whether a model is an embedding model or a reranker and defaults to buckets 128/512/1024 (embedding) or 512/1024 (reranker), clipped to the model's maximum sequence length — a model capped at 512 tokens compiles as 128/512 (embedding) or just 512 (reranker), and the compile log names each bucket it drops; --buckets 512,2048 compiles a custom set (S2048 is verified on M2 at ~518 ms/inference). Re-running skips up-to-date artifacts (--force reconverts). After every conversion a self-check verifies accuracy against the FP32 original, measures how many operations landed on the Neural Engine, and records warm latency — the printed summary doubles as a compatibility report: if you run eeANE on hardware we have not verified (M1/M3/M4...), please paste it into an issue. The accuracy sanity check is evaluated on three fixed input sets — English, Japanese and Chinese — and a variant passes as soon as any one set clears the threshold: an input in a language the model's tokenizer has little or no vocabulary for produces an amplified fp16-vs-fp32 difference that reflects the input, not the model, so scoring against a language the model can actually read is what the self-check needs to judge it fairly. The compile log names each set's measured result and which one was accepted, and the per-set numbers are recorded in the artifact metadata alongside the accepted set. The per-bucket measurements are aggregated into a calibration record (model_info.json) in the cache; buckets whose self-check failed are dropped from the recommended set that cache-resolved configs load. The tokenizer is frozen into the artifact directory and verified to reproduce the original tokenization exactly, so the server needs neither the original model files nor the transformers library at run time (see docs/dependency-policy.md).

Verified models

Every model below was compiled from its stock Hugging Face distribution, passed the self-check, and had its output compared against the reference sentence-transformers / CrossEncoder implementation on an M2 Mac. Buckets is what eeane compile produces with no --buckets flag, after clipping to the model's maximum sequence length. Models are grouped by the backend their config.json selects — note that a model's name does not always predict it (paraphrase-multilingual-mpnet-base-v2 is an XLM-RoBERTa model, and multilingual-e5-small is a BERT one).

Any other model built on one of these three architectures is likely to work as well; these are simply the ones that have been run end to end.

ModernBERT

XLM-RoBERTa

Models built on the plain RoBERTa architecture are also served by this backend (transformers implements RoBERTa as the same encoder, differing only in vocabulary), which is why the granite-embedding r1 English models below (RobertaModel in their config.json) sit in this table rather than BERT's.

BERT (embedding models only)

¹ Ships pytorch_model.bin only, so compiling it needs --allow-pickle (see Checkpoint formats).

Checkpoint formats

eeane compile accepts safetensors checkpoints by default. A Hugging Face repo id or local model directory that ships pytorch_model.bin weights only (no .safetensors file) is rejected with a clear error, whether the source is a Hub download or a local directory. Pass --allow-pickle to opt into pickle-based .bin weights instead: eeane compile then forces transformers to load them with torch.load(weights_only=True), and logs a WARNING. weights_only=True reduces but does not eliminate the risk of loading a pickle file — bypasses have been found before (e.g. CVE-2026-24747, fixed in torch 2.10.0), and the torch version eeane compile depends on is pinned, for compatibility with the Core ML conversion toolchain, to a release that predates that fix (see docs/dependency-policy.md). Only use --allow-pickle with checkpoints from publishers you trust. When a repository does have safetensors, --allow-pickle changes nothing: safetensors are always preferred, and the .bin files are never downloaded. BAAI/bge-m3, for example, ships pytorch_model.bin only, so compiling it needs the flag:

eeane compile BAAI/bge-m3 --allow-pickle

Configuration

The server runs with built-in defaults out of the box. To change them, copy eeane.example.toml to ./eeane.toml (or ~/.config/eeane/eeane.toml) and edit it. Config files are searched in this order: --config PATH > ./eeane.toml > ~/.config/eeane/eeane.toml > built-in defaults. The --host/--port/--log-level CLI flags and the EEANE_API_KEY environment variable override the file.

eeane serve --config /path/to/eeane.toml
eeane serve --host 192.168.1.20 --port 7997

# Validate a config file and print the resolved effective configuration
# (the API key value is never printed) without starting the server:
eeane check-config --config /path/to/eeane.toml

The config file lists the served models — any number of embedding and reranker entries. A [[models]] entry usually needs only its id = "...": the kind, frozen tokenizer, per-bucket artifacts and embedding width are then resolved from the compiled-model cache (server.cache_root, default ~/.cache/eeane/), honouring the calibration's recommended buckets. Spelling out kind, tokenizer and [models.artifacts] explicitly still works and pins the entry independently of the cache. Within each kind the first-listed entry is the default model, used when a request does not name one. Reranker entries may be omitted entirely for an embedding-only server (/rerank then answers 503). python -m eeane.server and python -m eeane <subcommand> remain available as backward-compatible aliases for eeane serve and the eeane command respectively (prefix both with uv run in the development environment).

Serving and operations

Model loading

The default load_policy for a [[models]] entry is "on_demand" ([server] default_load_policy can change the default; see eeane.example.toml for the setting): the server does not load any model at start-up, and loads one the moment a request first needs it. Once a model's artifacts have loaded once, a load is well under a second (0.3-0.8 s measured on an M2 Mac); the exception is the very first load of an artifact right after eeane compile produced it, which can take tens of seconds while macOS builds its Neural Engine cache for it — a one-time cost that later loads, even after a server restart, do not pay again. That wait is included in the response time of whichever request triggers it.

An on-demand model that has answered no request for keep_alive seconds ([server] keep_alive, default 300, overridable per model; 0 unloads it as soon as it goes idle) is unloaded automatically and reloaded on the next request that needs it. Set load_policy = "resident" on an entry to load it at start-up and keep it in memory for the server's whole run. Set load_policy = "disabled" to keep an entry in the config file without serving it: it is absent from GET /models and GET /health, and a request naming its id gets a 404.

[server] max_loaded_models caps how many models may be in memory at once (unset means no limit). When loading a model would exceed it, the longest-idle on_demand model is unloaded to make room; resident models and models currently handling a request are never evicted this way, so a configuration whose resident entries alone exceed the cap is rejected at start-up.

Batch-2 artifacts for embedding requests

Embedding models (not rerankers) may optionally be compiled with a second artifact per bucket that packs two inputs into one Neural Engine call: eeane compile <model> --buckets <S> --batch 2, run alongside the normal batch-1 compile. Serving it is opt-in through [models.batch_artifacts] on a [[models]] entry — a bucket -> artifact-path table mirroring [models.artifacts]. When a request routes two or more of its inputs to the same bucket, they are paired up and inferred through the batch-2 artifact instead of one at a time, which raised throughput for requests carrying many short inputs by about 25% in benchmarks on an M2 Mac. An id-only entry resolves batch_artifacts automatically from the compiled-model cache once a batch-2 artifact has been compiled for it; the explicit form (which spells out [models.artifacts]) must spell out [models.batch_artifacts] too — it cannot be set on its own. A configuration without any batch-2 artifacts behaves exactly as before.

Request admission, queueing and shutdown

server.max_pending_requests caps how many inference requests the server admits at once, counting both requests currently running and requests still waiting their turn (default 500; 0 means unlimited). A request that arrives once the cap is reached is rejected immediately with 429 Too Many Requests and a Retry-After header.

server.queue_timeout caps how long an admitted request may wait between being accepted and actually starting inference (default 600 seconds; 0 disables the timeout). A request that waits past this limit is abandoned with 503 Service Unavailable and a Retry-After header. Once a request has started inference it is never interrupted by this timeout, however long it runs. Either way, Retry-After tells the client how long to wait before retrying.

server.coalesce_requests (default true) merges an incoming request with an identical one (same model, same input) that is already being processed: instead of running inference twice, the second request attaches to the first and receives the same result once it completes.

server.graceful_shutdown_timeout bounds how long the server waits for in-flight requests to finish when it receives SIGTERM or Ctrl-C (default: unset, meaning it waits for all of them to finish however long that takes; no new connections are accepted while it waits). Set it to a number of seconds to cap that wait instead.

Serving beyond localhost

Binding to a non-loopback address (--host or server.host) exposes the server to your network. Set an API key — api_key in the config file (keep it chmod 600) or the EEANE_API_KEY environment variable — and every endpoint except GET /health will require an Authorization: Bearer <key> header; the server logs a warning when it serves a non-loopback address without one. /health stays open for monitoring and is rate-limited instead (server.health_rate_limit, default 60 requests/min per client IP, 0 disables). These are application-level safeguards only: for exposure beyond a trusted LAN/VPN, put the server behind a reverse proxy or firewall.

Running as a service

To start the server automatically at login and keep it running, set it up as a macOS launchd agent — see docs/launchd.md for a step-by-step guide and a ready-made plist template. Thanks to on-demand loading, an always-on eeANE agent costs almost nothing while idle.

API

• GET /health — status and one entry per served model (id, kind, buckets in service, loaded), unauthenticated, rate-limited
• GET /models (alias: GET /v1/models) — OpenAI-compatible listing of every served model
• POST /v1/embeddings (alias: POST /embeddings) — OpenAI-compatible (input as string or list, encoding_format float/base64, optional dimensions); embeddings are L2-normalized by default (per-model normalize)
• POST /rerank, POST /v1/rerank — Infinity-compatible (query/documents/top_n/return_documents/raw_scores)

The optional model field of the embeddings and rerank requests selects the served model by its configured id; omitting it selects the first-listed model of the endpoint's kind. An unknown id gets a 404 listing the servable ids, and naming a model of the other kind gets a 400. The embeddings and rerank endpoints are served both under /v1 and at the root, so a base URL with or without the /v1 suffix works. Each input is routed to the smallest fitting sequence-length bucket of its model and truncated to the largest bucket when longer, with a server-side warning.

The optional dimensions field of an embeddings request (OpenAI-compatible) truncates each returned embedding to its first dimensions components before re-normalizing (when normalization is enabled), returning a smaller vector without re-running inference. It is meaningful for models trained with a Matryoshka representation learning (MRL) objective, whose embeddings stay meaningful at any prefix length; requesting more dimensions than the model's embedding width gets a 400.

To use eeANE from Open WebUI: set the embedding engine to OpenAI with base URL http://127.0.0.1:7997/v1, and the reranking engine to External with URL http://127.0.0.1:7997/rerank. If you configured an API key, enter it as the OpenAI API key / External reranker API key — Open WebUI sends it as the Authorization header eeANE expects.

Performance

Figures below were measured on an M2 Mac mini (macOS 13+, 16 GB), with the same models served from the MPS GPU by PyTorch (sentence-transformers) as the baseline:

• Embedding throughput: up to ~13,600 effective (padding-excluded) tokens/s on the ANE — 2–3x the MPS baseline, at a similar power draw but 2.6–3.8x the energy efficiency per token, with the GPU left entirely free.
• Reranking: a 36-document rerank over HTTP completes in ~2.0–5.6 s depending on chunk length, ~2–8x faster than the same request against MPS-based serving of the same model.
• Memory: a server holding one 310M-class embedding model and one reranker resident stays around 750 MB; compiled weights live mostly outside the Python process, and on-demand entries release their memory when idle.
• Load times: ~0.2–0.8 s per model once macOS has cached a compiled artifact (the very first load after compiling takes tens of seconds, once).

Responses over HTTP are verified to match direct Core ML inference exactly (tools/verify_server.py in a repository checkout).

Troubleshooting

• 404 model not found: the model field a client sends must match a served model's configured id exactly. Check the ids eeANE actually serves with GET /models. Clients migrated from an older eeANE version should note that the model field used to be ignored entirely, so a request naming anything (or nothing) used to succeed — that leniency is gone.
• 500 ... produced a non-finite output ...: see Known limitations below. This means the model ran off the Neural Engine; verify Neural Engine availability on the machine serving the request.
• A request occasionally takes much longer than usual: the first request after a model was idle past keep_alive pays the on-demand reload (typically well under a second), and the very first request after eeane compile pays the one-time Neural Engine cache build (tens of seconds). Both are expected; use load_policy = "resident" if you need to avoid even the sub-second reload.
• eeane compile fails with no .safetensors weights are available ...: the model ships a pickle-based pytorch_model.bin checkpoint only, and eeane compile requires safetensors by default. The error message already names the fix: pass --allow-pickle (see Checkpoint formats above) — only for checkpoints from publishers you trust.
• eeane compile fails with ... must declare its pooling in the sentence-transformers '1_Pooling/config.json' ...: compiling an embedding model needs a 1_Pooling/config.json sentence-transformers pooling declaration; when it is missing, unreadable, or names an unsupported mode (anything other than mean or CLS), eeane compile stops with an error instead of guessing. Check that the model is actually distributed in sentence-transformers form — for a local model directory, check that the 1_Pooling/config.json the publisher ships alongside it is present. eeANE does not fall back to assuming mean pooling: a wrongly-pooled artifact would silently return vectors with a different meaning.

Known limitations

• ANE only: eeANE targets the Apple Neural Engine; running a compiled model on a CPU-only compute path is not supported. On a machine or configuration where the Neural Engine is not actually available to a compiled model, inference can produce non-finite (NaN/Inf) output — this has been observed across every architecture eeANE supports. Rather than silently return such a result, the server detects non-finite output at inference time and answers with 500 Internal Server Error. Seeing this error is a strong signal that the Neural Engine is not actually being used in your environment.
• Verified hardware: all published measurements and verifications were run on an M2 Mac. Other Apple Silicon generations (M1/M3/M4...) are expected to work but are unverified by the maintainer; the self-check summary that eeane compile prints doubles as a compatibility report for exactly this reason. Reports from other machines — success or failure — are very welcome as GitHub issues.
• Long documents: each input is truncated to its model's largest compiled bucket (add larger buckets with eeane compile --buckets if you need them); rerankers have no sliding-window handling for documents beyond that.
• BAAI/bge-m3 is dense-only: eeANE compiles and serves bge-m3's dense embedding output; its separate sparse and multi-vector (ColBERT-style) representations are additional weight files that eeane compile does not fetch or expose.
• BERT cross-encoder rerankers are not supported: see About eeane compile above — pinning the compiled graph's segment ids to zero would change the meaning of a query/document pair for this architecture. BERT embedding models are unaffected.
• Accuracy on out-of-vocabulary input: compiled models run in fp16. For input a model's tokenizer cannot represent well — feeding English text to a Chinese-only model, say — the rounding difference against an fp32 reference grows noticeably, because the input is already far outside what the model was trained on. eeane compile's own self-check sidesteps this by scoring three fixed language sets (English, Japanese, Chinese) and accepting the variant on whichever set the model actually has vocabulary for (see About eeane compile above), but the same caution still applies whenever you compare a model's fp16 and fp32 output on input outside its intended languages yourself. Within a model's intended languages the agreement is far tighter (cosine ≥ 0.9999 on every model listed above).

Development

To work on eeANE itself, or to use the repository-only tools below, clone the repository and run commands with uv run from the checkout instead of installing the package:

git clone https://github.com/xhighhongo41/eeANE.git
cd eeANE
uv sync --extra compile   # torch/transformers are needed only for compiling
uv run eeane compile cl-nagoya/ruri-v3-310m
uv run eeane serve

uv run eeane <subcommand> runs the same compile/serve/ check-config subcommands described above, from the checkout rather than an installed package.

To verify a running server end to end (accuracy vs. direct Core ML inference, API compatibility, latency), and to lint and test the codebase in one step — both assume a repository checkout:

uv run python tools/verify_server.py all
# Check one specific served model against direct Core ML inference:
uv run python tools/verify_server.py verify-embedding --model intfloat/multilingual-e5-base
uv run python tools/verify_server.py verify-rerank --model BAAI/bge-reranker-v2-m3
./tools/check.sh   # ruff lint + format check + pytest, in one step

Trying the PoC (historical development snapshot)

The poc/ scripts are the frozen v0.1–v0.3 research record; the supported conversion path is eeane compile above. They remain runnable for benchmarking studies:

git clone https://github.com/xhighhongo41/eeANE.git
cd eeANE
uv sync

# Place the models in HF distribution form under models/ruri-v3-310m and
# models/ruri-v3-reranker-310m (e.g. download with
# `huggingface-cli download cl-nagoya/ruri-v3-310m`), then:

# Embedding model (v0.1)
uv run python poc/convert_embedding.py --seq-len 512   # HF -> .mlmodelc
uv run python poc/verify_accuracy.py --seq-len 512     # accuracy vs FP32
uv run python poc/benchmark_latency.py --seq-len 512 --compute-units CPU_AND_NE --compute-plan

# Reranker model (v0.2)
uv run python poc/convert_reranker.py --seq-len 512
uv run python poc/verify_reranker_accuracy.py --seq-len 512
uv run python poc/benchmark_latency.py --model reranker --seq-len 512 --compute-units CPU_AND_NE --compute-plan

# Performance study (v0.3)
uv run python poc/run_sweep.py --seq-lens 128,512 --batches 1,2      # S x B latency matrix
uv run python poc/benchmark_throughput.py --model embedding --chunk-tokens 128 --batch 2
uv run python poc/benchmark_mps.py --model embedding --chunk-tokens 512 --batch 32  # GPU baseline

Acknowledgments and related projects

eeANE was inspired by Infinity, the open-source serving engine that showed how convenient a self-hosted, API-compatible embedding and reranking server can be — eeANE's /rerank API deliberately follows Infinity's schema so that clients can switch between the two by changing a URL.

eeANE exists in the first place because its author could run ModernBERT-based embedding models on a GPU with Infinity. The two projects complement rather than compete with each other: eeANE runs models exclusively on the Apple Neural Engine of Apple Silicon Macs, and supports a deliberately small set of model architectures. If you want to serve embedding or reranking models on Linux or Windows, on NVIDIA/AMD GPUs or CPUs, or need a much wider model catalogue, by all means use Infinity.

Changelog

Details for each release: GitHub Releases.

License

GPL-3.0-or-later. See LICENSE.

The test corpus under testdata/corpus/ consists of public-domain literary works from Aozora Bunko and is not covered by the GPL; see testdata/corpus/README.md.

日本語のREADMEは README_ja.md を参照してください。