fastretrieval

Fast multi-model retrieval runtime: ONNX and GGUF embeddings, reranking, and a declarative model contract

What it is

fastretrieval is a Python runtime for multi-model retrieval with text embeddings and reranking on ONNX Runtime or GGUF (llama-cpp-python) with no PyTorch dependency. It uses a declarative model contract so built-in Qwen3 reference models and custom models can share the same runtime, and supports Matryoshka (MRL) truncation, instruction-aware queries, and optional GPU acceleration. It is derived from fastembed and keeps Qwen3 model names as model identifiers rather than as the package boundary. Supported runtimes are CPython 3.11, 3.12, 3.13, and 3.14.

What it does

Table of contents

• What it does
• Features
• Supported Models
• Installation
• Usage
• Converting your own model
• Configuration
• Migrating from qwen3-embed
• Releases and Deployment
• Development
• Related Projects
• Contributing
• License

Features

• Model-specific pooling: Built-in Qwen3 text models use last-token pooling; other models use their declared preprocessing and pooling contract.
• MRL support: Qwen3 reference embeddings support truncation from 32 to 1024 dimensions before L2 normalization.
• Instruction-aware: Query embedding supports task instructions for better retrieval performance.
• Causal LM reranking: Qwen3 rerankers use yes/no logit scoring, producing [0, 1] scores.
• Multiple backends: ONNX Runtime and optional GGUF via llama-cpp-python; available formats depend on the selected model.
• GPU optional, no PyTorch: Runs on ONNX Runtime or llama-cpp-python -- no heavy ML framework required. Auto-detects GPU (CUDA, DirectML) when available.
• Multilingual: Built-in Qwen3 reference models support multi-language inputs.

Supported Models

The public facades have independent registries. TextEmbedding() selects n24q02m/Qwen3-Embedding-0.6B-ONNX (INT8, 1024 dimensions); every other facade requires an explicit model_name. A registry's first entry is not an implicit default. Qwen3 models are reference models, not a restriction on supported model families.

Qwen3 ONNX reference models

Qwen3 GGUF reference models (requires the gguf extra)

Other built-in ONNX models

The sparse registry also reports the deprecated misspelling prithvida/Splade_PP_en_v1; use the canonical prithivida spelling above. Call each facade's list_supported_models() for its installed registry metadata without loading model weights. Register additional dense and reranker models with CustomModelSpec / CustomRerankerSpec and the declarative fastretrieval.contract API; custom registration does not change the facade defaults.

HuggingFace Repos

Installation

pip install fastretrieval            # text only
pip install "fastretrieval[image]"   # adds image and ColPali
pip install "fastretrieval[all]"     # adds GGUF backend too

# For GGUF support
pip install fastretrieval[gguf]

Usage

Text Embedding

from fastretrieval import TextEmbedding

# INT8 (default)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX")

# Q4F16 (smaller, slightly less accurate)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX-Q4F16")

# GGUF (requires: pip install fastretrieval[gguf])
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF")

documents = [
    "Qwen3 is a multilingual embedding model.",
    "ONNX Runtime enables fast CPU inference.",
]

embeddings = list(model.embed(documents))
# Each embedding: numpy array of shape (1024,), L2-normalized

# Matryoshka Representation Learning (MRL) -- truncate to smaller dims
embeddings_256 = list(model.embed(documents, dim=256))
# Each embedding: numpy array of shape (256,), L2-normalized

# Query with instruction (for retrieval tasks)
queries = list(
    model.query_embed(
        ["What is Qwen3?"],
        task="Given a question, retrieve relevant passages",
    )
)

Reranking

from fastretrieval import TextCrossEncoder

reranker = TextCrossEncoder(model_name="n24q02m/Qwen3-Reranker-0.6B-ONNX")

# YesNo variant: ~10x less RAM (~598MB vs ~12GB at inference)
# reranker = TextCrossEncoder(model_name="n24q02m/Qwen3-Reranker-0.6B-ONNX-YesNo")

query = "What is Qwen3?"
documents = [
    "Qwen3 is a series of large language models by Alibaba.",
    "The weather today is sunny.",
    "Qwen3-Embedding supports multilingual text embedding.",
]

scores = list(reranker.rerank(query, documents))
# scores: list of float in [0, 1], higher = more relevant

# Or rerank pairs directly
pairs = [
    ("What is AI?", "Artificial intelligence is a branch of computer science."),
    ("What is ML?", "Machine learning is a subset of AI."),
]
pair_scores = list(reranker.rerank_pairs(pairs))

Reranker determinism

Reranker scores are batch-invariant: the score of a (query, document) pair does not depend on batch size or the other documents scored in the same call. ONNX reranker variants are scored one sequence at a time (no padding), which keeps RoPE positions correct regardless of batch composition.

Custom models (bring your own)

Qwen3 models are built-in reference models, but any ONNX-able embedding model can be registered and then loaded by id. Use CustomModelSpec with one of the four output shapes: CLS/MEAN (bert-bi), LAST_TOKEN (causal), or DISABLED (raw).

from fastretrieval import CustomModelSpec, TextEmbedding

# Multilingual (incl. Vietnamese) + code, CLS-pooled, 768-dim
CustomModelSpec(
    model_id="onnx-community/gte-multilingual-base",
    hf="onnx-community/gte-multilingual-base",
    model_file="onnx/model_quantized.onnx",
    dim=768,
    pooling="CLS",
    normalization=True,
).register()

model = TextEmbedding("onnx-community/gte-multilingual-base")
embeddings = list(model.embed(["xin chào", "def add(a, b): return a + b"]))

Other verified examples: bge-m3 (pooling="CLS", dim=1024), EmbeddingGemma-300m (pooling="MEAN", dim=768). MRL truncation (embed(..., dim=256)) works for custom models whose vectors are Matryoshka-trained. Custom models are scored per-row, so — like the built-in INT8 reranker — their scores are batch-invariant by construction.

A BYO reranker registers the same way with CustomRerankerSpec. Any standard ONNX cross-encoder (a single relevance logit per pair — bge-reranker, gte-reranker, ms-marco, jina-reranker) works; there is no dim/pooling to set:

from fastretrieval import CustomRerankerSpec, TextCrossEncoder

CustomRerankerSpec(
    model_id="onnx-community/gte-multilingual-reranker-base",
    hf="onnx-community/gte-multilingual-reranker-base",
    model_file="onnx/model_quantized.onnx",
).register()

encoder = TextCrossEncoder("onnx-community/gte-multilingual-reranker-base")
scores = list(encoder.rerank("xin chào", ["tài liệu A", "tài liệu B"]))

PyTorch-only models can be converted first (in a throwaway env, since the export deps don't co-resolve with the lean runtime pins):

# pip install "optimum-onnx[onnxruntime]" torch transformers onnx
from fastretrieval.export import export_to_onnx

export_to_onnx("intfloat/multilingual-e5-base", "./e5-onnx")

Converting your own model

fastretrieval does not ship a closed model zoo. Point the converter at a model family in the support matrix and it produces an artifact with a declarative contract manifest. Unsupported architectures fail closed instead of producing a misleading artifact.

The conversion dependencies (torch, transformers, optimum-onnx, and the ONNX Runtime quantizer stack including onnx-ir) are deliberately not runtime dependencies. Run conversion in a throwaway environment:

uv run --with-requirements fastretrieval/convert/requirements.txt \
  python -m fastretrieval.convert onnx intfloat/multilingual-e5-base \
  --out ./e5 --pooling mean --normalize

uv run --with-requirements fastretrieval/convert/requirements.txt \
  python -m fastretrieval.convert verify ./e5 \
  --source intfloat/multilingual-e5-base

verify validates the manifest, loads every ONNX variant through ONNX Runtime, and compares the converted outputs with the original model on the same probes. The card command writes a model card only after the manifest and artifact formats pass validation. GGUF conversion additionally requires a built llama.cpp checkout and --llama-cpp (or LLAMA_CPP_HOME).

Large models can run in the optional Modal backend. It mounts local sources and downloads the completed artifact back to the requested output directory; it does not publish to a model hub:

uv run --with-requirements fastretrieval/convert/requirements.txt \
  python -m fastretrieval.convert onnx Qwen/Qwen3-Embedding-0.6B \
  --out ./qwen3 --pooling last_token --normalize --backend modal

The first acceptance profiles are Qwen3 and BERT-family text models. Adding a model family requires its architecture, task, modality, pooling, normalization, and output-shape profile plus parity tests; the model name alone is never enough.

Configuration

Environment variables

The FASTRETRIEVAL_* names take precedence when both names are set. The deprecated QWEN3_EMBED_* names remain readable and emit a DeprecationWarning so existing configurations do not silently stop working.

Programmatic cache location

Consumers that need to inspect or clear the model cache should use the public helper instead of importing an internal module:

from fastretrieval import define_cache_dir

cache_dir = define_cache_dir()

define_cache_dir() honors FASTRETRIEVAL_CACHE_PATH first and the deprecated QWEN3_EMBED_CACHE_PATH alias second. An explicit path argument overrides both.

GPU Acceleration

Both ONNX and GGUF backends auto-detect GPU when available (Device.AUTO is the default).

ONNX

Requires onnxruntime-gpu (CUDA) or onnxruntime-directml (Windows) instead of onnxruntime:

pip install onnxruntime-gpu  # NVIDIA CUDA
# or
pip install onnxruntime-directml  # Windows AMD/Intel/NVIDIA

from fastretrieval import TextEmbedding, Device

# Auto-detect GPU (default)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX")

# Force CPU
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX", cuda=Device.CPU)

# Force CUDA
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX", cuda=Device.CUDA)

GGUF

GPU is handled by llama-cpp-python. The default pip install fastretrieval[gguf] is CPU-only. For CUDA GPU support, build with:

CMAKE_ARGS="-DGGML_CUDA=on" pip install fastretrieval[gguf]

from fastretrieval import TextEmbedding, Device

# Auto-detect GPU (default, offloads all layers)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF")

# Force CPU only
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF", cuda=Device.CPU)

Development

uv sync --group dev                              # Install dev dependencies
uv run ruff check .                              # Lint
uv run ruff format --check .                     # Format check
uv run ty check                                  # Type check
uv run pytest                                    # All tests (integration tests download ~1.2 GB)
uv run pytest -m "not integration" --tb=short    # Unit tests only (CI default)

# Shortcuts (optional, via mise): mise run setup / lint / test / fix

Note: pull requests see two identical "Analyze (CodeQL)" checks — CI intentionally runs CodeQL from two configurations (ci.yml and .github/workflows/codeql.yml) until the legacy code-scanning registration is removed. The cleanup runbook lives in the header of .github/workflows/codeql.yml.

Migrating from qwen3-embed

The library was previously published as qwen3-embed. New releases use the fastretrieval distribution and import package; Qwen3 model identifiers remain unchanged.

Package and import names

# Before
pip install qwen3-embed[gguf]

# After
pip install fastretrieval[gguf]

# Before
from qwen3_embed import TextEmbedding, TextCrossEncoder

# After
from fastretrieval import TextEmbedding, TextCrossEncoder

The public API is unchanged. The old environment variable names remain supported as deprecated compatibility aliases:

The old names still work and emit a DeprecationWarning; when both names are set, the FASTRETRIEVAL_* value wins. Existing qwen3-embed releases remain on PyPI and continue to receive security fixes while consumers migrate.

Releases and Deployment

Merges to main release automatically: .github/workflows/cd.yml runs python-semantic-release (v10) and publishes the version to PyPI via Trusted Publishing (PUBLISH_PYPI=true). Beta prereleases are cut only by a manual workflow_dispatch with release_type=beta. Before any release, the readme-sync job verifies that pyproject.toml points at the README shipped to the registry, so README edits pass through the same gate. The Docker, MCP-registry, and downstream-notify jobs in the CD template are gated off for this repo (PUBLISH_DOCKER, PUBLISH_MCP_REGISTRY, and NOTIFY_DOWNSTREAM are unset).

The public reference page is mcp.n24q02m.com/reference/fastretrieval/. fastretrieval is a library: it ships no MCP endpoint and has no hosted deployment.

Related Projects

• wet-mcp -- MCP web search server with vector-based docs search, uses fastretrieval for local embedding
• mnemo-mcp -- MCP memory server with semantic search powered by fastretrieval
• better-code-review-graph -- Knowledge graph for code reviews, uses fastretrieval for local ONNX embedding
• modalcom-ai-workers -- GPU-serverless workers that convert Qwen3 models to ONNX/GGUF format

Contributing

See CONTRIBUTING.md.

License

Apache-2.0 -- See LICENSE. Original fastembed by Qdrant.