fastretrieval

Fast multi-model retrieval runtime: ONNX and GGUF embeddings, reranking, and a declarative model contract

What it is

fastretrieval is a Python runtime for multi-model retrieval with text embeddings and reranking on ONNX Runtime or GGUF (llama-cpp-python) with no PyTorch dependency. It uses a declarative model contract so built-in Qwen3 reference models and custom models can share the same runtime, and supports Matryoshka (MRL) truncation, instruction-aware queries, and optional GPU acceleration. It is derived from fastembed and keeps Qwen3 model names as model identifiers rather than as the package boundary. Supported runtimes are CPython 3.10, 3.11, 3.12, 3.13, and 3.14.

What it does

Table of contents

• What it does
• Features
• Supported Models
• Installation
• Usage
• Converting your own model
• Configuration
• Migrating from qwen3-embed
• Development
• Related Projects
• Contributing
• License

Features

• Last-token pooling: Uses the final token representation (with left-padding) instead of mean pooling.
• MRL support: Matryoshka Representation Learning allows truncating embeddings to any dimension from 32 to 1024 while preserving quality.
• Instruction-aware: Query embedding supports task instructions for better retrieval performance.
• Causal LM reranking: Reranker uses yes/no logit scoring via causal language model, producing calibrated [0, 1] scores.
• Multiple backends: ONNX Runtime (INT8, Q4F16) and GGUF (Q4_K_M via llama-cpp-python).
• GPU optional, no PyTorch: Runs on ONNX Runtime or llama-cpp-python -- no heavy ML framework required. Auto-detects GPU (CUDA, DirectML) when available.
• Multilingual: Built-in Qwen3 reference models support multi-language inputs.

Supported Models

The entries below are the built-in Qwen3 reference models. Other model families can be registered through the declarative model contract.

ONNX (default)

GGUF (optional, requires llama-cpp-python)

HuggingFace Repos

Installation

pip install fastretrieval            # text only
pip install "fastretrieval[image]"   # adds image and ColPali
pip install "fastretrieval[all]"     # adds GGUF backend too

# For GGUF support
pip install fastretrieval[gguf]

Usage

Text Embedding

from fastretrieval import TextEmbedding

# INT8 (default)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX")

# Q4F16 (smaller, slightly less accurate)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX-Q4F16")

# GGUF (requires: pip install fastretrieval[gguf])
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF")

documents = [
    "Qwen3 is a multilingual embedding model.",
    "ONNX Runtime enables fast CPU inference.",
]

embeddings = list(model.embed(documents))
# Each embedding: numpy array of shape (1024,), L2-normalized

# Matryoshka Representation Learning (MRL) -- truncate to smaller dims
embeddings_256 = list(model.embed(documents, dim=256))
# Each embedding: numpy array of shape (256,), L2-normalized

# Query with instruction (for retrieval tasks)
queries = list(
    model.query_embed(
        ["What is Qwen3?"],
        task="Given a question, retrieve relevant passages",
    )
)

Reranking

from fastretrieval import TextCrossEncoder

reranker = TextCrossEncoder(model_name="n24q02m/Qwen3-Reranker-0.6B-ONNX")

# YesNo variant: ~10x less RAM (~598MB vs ~12GB at inference)
# reranker = TextCrossEncoder(model_name="n24q02m/Qwen3-Reranker-0.6B-ONNX-YesNo")

query = "What is Qwen3?"
documents = [
    "Qwen3 is a series of large language models by Alibaba.",
    "The weather today is sunny.",
    "Qwen3-Embedding supports multilingual text embedding.",
]

scores = list(reranker.rerank(query, documents))
# scores: list of float in [0, 1], higher = more relevant

# Or rerank pairs directly
pairs = [
    ("What is AI?", "Artificial intelligence is a branch of computer science."),
    ("What is ML?", "Machine learning is a subset of AI."),
]
pair_scores = list(reranker.rerank_pairs(pairs))

Reranker determinism

Reranker scores are batch-invariant: the score of a (query, document) pair does not depend on batch size or the other documents scored in the same call. ONNX reranker variants are scored one sequence at a time (no padding), which keeps RoPE positions correct regardless of batch composition.

Custom models (bring your own)

Qwen3 models are built-in reference models, but any ONNX-able embedding model can be registered and then loaded by id. Use CustomModelSpec with one of the four output shapes: CLS/MEAN (bert-bi), LAST_TOKEN (causal), or DISABLED (raw).

from fastretrieval import CustomModelSpec, TextEmbedding

# Multilingual (incl. Vietnamese) + code, CLS-pooled, 768-dim
CustomModelSpec(
    model_id="onnx-community/gte-multilingual-base",
    hf="onnx-community/gte-multilingual-base",
    model_file="onnx/model_quantized.onnx",
    dim=768,
    pooling="CLS",
    normalization=True,
).register()

model = TextEmbedding("onnx-community/gte-multilingual-base")
embeddings = list(model.embed(["xin chào", "def add(a, b): return a + b"]))

Other verified examples: bge-m3 (pooling="CLS", dim=1024), EmbeddingGemma-300m (pooling="MEAN", dim=768). MRL truncation (embed(..., dim=256)) works for custom models whose vectors are Matryoshka-trained. Custom models are scored per-row, so — like the built-in INT8 reranker — their scores are batch-invariant by construction.

A BYO reranker registers the same way with CustomRerankerSpec. Any standard ONNX cross-encoder (a single relevance logit per pair — bge-reranker, gte-reranker, ms-marco, jina-reranker) works; there is no dim/pooling to set:

from fastretrieval import CustomRerankerSpec, TextCrossEncoder

CustomRerankerSpec(
    model_id="onnx-community/gte-multilingual-reranker-base",
    hf="onnx-community/gte-multilingual-reranker-base",
    model_file="onnx/model_quantized.onnx",
).register()

encoder = TextCrossEncoder("onnx-community/gte-multilingual-reranker-base")
scores = list(encoder.rerank("xin chào", ["tài liệu A", "tài liệu B"]))

PyTorch-only models can be converted first (in a throwaway env, since the export deps don't co-resolve with the lean runtime pins):

# pip install "optimum-onnx[onnxruntime]" torch transformers onnx
from fastretrieval.export import export_to_onnx

export_to_onnx("intfloat/multilingual-e5-base", "./e5-onnx")

Converting your own model

fastretrieval does not ship a closed model zoo. Point the converter at a model family in the support matrix and it produces an artifact with a declarative contract manifest. Unsupported architectures fail closed instead of producing a misleading artifact.

The conversion dependencies (torch, transformers, optimum-onnx, and the ONNX Runtime quantizer stack including onnx-ir) are deliberately not runtime dependencies. Run conversion in a throwaway environment:

uv run --with-requirements fastretrieval/convert/requirements.txt \
  python -m fastretrieval.convert onnx intfloat/multilingual-e5-base \
  --out ./e5 --pooling mean --normalize

uv run --with-requirements fastretrieval/convert/requirements.txt \
  python -m fastretrieval.convert verify ./e5 \
  --source intfloat/multilingual-e5-base

verify validates the manifest, loads every ONNX variant through ONNX Runtime, and compares the converted outputs with the original model on the same probes. The card command writes a model card only after the manifest and artifact formats pass validation. GGUF conversion additionally requires a built llama.cpp checkout and --llama-cpp (or LLAMA_CPP_HOME).

Large models can run in the optional Modal backend. It mounts local sources and downloads the completed artifact back to the requested output directory; it does not publish to a model hub:

uv run --with-requirements fastretrieval/convert/requirements.txt \
  python -m fastretrieval.convert onnx Qwen/Qwen3-Embedding-0.6B \
  --out ./qwen3 --pooling last_token --normalize --backend modal

The first acceptance profiles are Qwen3 and BERT-family text models. Adding a model family requires its architecture, task, modality, pooling, normalization, and output-shape profile plus parity tests; the model name alone is never enough.

Configuration

Environment variables

The FASTRETRIEVAL_* names take precedence when both names are set. The deprecated QWEN3_EMBED_* names remain readable and emit a DeprecationWarning so existing configurations do not silently stop working.

GPU Acceleration

Both ONNX and GGUF backends auto-detect GPU when available (Device.AUTO is the default).

ONNX

Requires onnxruntime-gpu (CUDA) or onnxruntime-directml (Windows) instead of onnxruntime:

pip install onnxruntime-gpu  # NVIDIA CUDA
# or
pip install onnxruntime-directml  # Windows AMD/Intel/NVIDIA

from fastretrieval import TextEmbedding, Device

# Auto-detect GPU (default)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX")

# Force CPU
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX", cuda=Device.CPU)

# Force CUDA
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX", cuda=Device.CUDA)

GGUF

GPU is handled by llama-cpp-python. The default pip install fastretrieval[gguf] is CPU-only. For CUDA GPU support, build with:

CMAKE_ARGS="-DGGML_CUDA=on" pip install fastretrieval[gguf]

from fastretrieval import TextEmbedding, Device

# Auto-detect GPU (default, offloads all layers)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF")

# Force CPU only
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF", cuda=Device.CPU)

Development

uv sync --group dev                              # Install dev dependencies
uv run ruff check .                              # Lint
uv run ruff format --check .                     # Format check
uv run ty check                                  # Type check
uv run pytest                                    # All tests (integration tests download ~1.2 GB)
uv run pytest -m "not integration" --tb=short    # Unit tests only (CI default)

# Shortcuts (optional, via mise): mise run setup / lint / test / fix

Migrating from qwen3-embed

The library was previously published as qwen3-embed. New releases use the fastretrieval distribution and import package; Qwen3 model identifiers remain unchanged.

Package and import names

# Before
pip install qwen3-embed[gguf]

# After
pip install fastretrieval[gguf]

# Before
from qwen3_embed import TextEmbedding, TextCrossEncoder

# After
from fastretrieval import TextEmbedding, TextCrossEncoder

The public API is unchanged. The old environment variable names remain supported as deprecated compatibility aliases:

The old names still work and emit a DeprecationWarning; when both names are set, the FASTRETRIEVAL_* value wins. Existing qwen3-embed releases remain on PyPI and continue to receive security fixes while consumers migrate.

Related Projects

• wet-mcp -- MCP web search server with vector-based docs search, uses fastretrieval for local embedding
• mnemo-mcp -- MCP memory server with semantic search powered by fastretrieval
• better-code-review-graph -- Knowledge graph for code reviews, uses fastretrieval for local ONNX embedding
• modalcom-ai-workers -- GPU-serverless workers that convert Qwen3 models to ONNX/GGUF format

Contributing

See CONTRIBUTING.md.

License

Apache-2.0 -- See LICENSE. Original fastembed by Qdrant.