qwen3-embed 1.11.2b2

Lightweight Qwen3 text embedding & reranking via ONNX Runtime and GGUF (fork of fastembed)

Qwen3 Embed

Lightweight Qwen3 text embedding and reranking via ONNX Runtime and GGUF

Sister projects from n24q02m (click to expand)

Project	Tagline	Tag
better-code-review-graph	Knowledge graph for token-efficient code reviews -- fixed search, configurabl...	MCP
better-email-mcp	IMAP/SMTP email server for AI agents -- 6 composite tools with multi-account ...	MCP
better-godot-mcp	Composite MCP server for Godot Engine -- 17 mega-tools for AI-assisted game d...	MCP
better-notion-mcp	Markdown-first Notion API server for AI agents -- 10 composite tools replacin...	MCP
better-telegram-mcp	MCP server for Telegram with dual-mode support: Bot API (httpx) for quick bot...	MCP
claude-plugins	Full documentation: mcp.n24q02m.com — unified docs for all 8 servers + the mc...	Marketplace
imagine-mcp	Production-grade MCP server for image and video understanding + generation ac...	MCP
jules-task-archiver	Chrome Extension for bulk operations on Jules tasks via batchexecute API -- a...	Tooling
mcp-core	Unified MCP Streamable HTTP 2025-11-25 transport, OAuth 2.1 Authorization Ser...	MCP
mnemo-mcp	Persistent AI memory with hybrid search and embedded sync. Open, free, unlimi...	MCP
qwen3-embed	Lightweight Qwen3 text embedding and reranking via ONNX Runtime and GGUF	Library
skret	Secrets without the server.	CLI
web-core	Shared web infrastructure package for search, scraping, HTTP security, and st...	Library
wet-mcp	Open-source MCP Server for web search, content extraction, library docs & mul...	MCP

Table of contents

• Features
• Supported Models
• Installation
• Usage
• Configuration
• Development
• Related Projects
• Contributing
• License

Trimmed fork of fastembed, keeping only Qwen3 models.

Features

• Last-token pooling: Uses the final token representation (with left-padding) instead of mean pooling.
• MRL support: Matryoshka Representation Learning allows truncating embeddings to any dimension from 32 to 1024 while preserving quality.
• Instruction-aware: Query embedding supports task instructions for better retrieval performance.
• Causal LM reranking: Reranker uses yes/no logit scoring via causal language model, producing calibrated [0, 1] scores.
• Multiple backends: ONNX Runtime (INT8, Q4F16) and GGUF (Q4_K_M via llama-cpp-python).
• GPU optional, no PyTorch: Runs on ONNX Runtime or llama-cpp-python -- no heavy ML framework required. Auto-detects GPU (CUDA, DirectML) when available.
• Multilingual: Both models support multi-language inputs.

Supported Models

ONNX (default)

Model	Type	Dims	Max Tokens	Size
n24q02m/Qwen3-Embedding-0.6B-ONNX	Embedding	32-1024 (MRL)	32768	573 MB
n24q02m/Qwen3-Embedding-0.6B-ONNX-Q4F16	Embedding	32-1024 (MRL)	32768	517 MB
n24q02m/Qwen3-Reranker-0.6B-ONNX	Reranker	-	40960	573 MB
n24q02m/Qwen3-Reranker-0.6B-ONNX-Q4F16	Reranker	-	40960	518 MB
n24q02m/Qwen3-Reranker-0.6B-ONNX-YesNo	Reranker	-	40960	598 MB

GGUF (optional, requires llama-cpp-python)

Model	Type	Dims	Max Tokens	Size
n24q02m/Qwen3-Embedding-0.6B-GGUF	Embedding	32-1024 (MRL)	32768	378 MB
n24q02m/Qwen3-Reranker-0.6B-GGUF	Reranker	-	40960	378 MB

HuggingFace Repos

Format	Embedding	Reranker
ONNX	n24q02m/Qwen3-Embedding-0.6B-ONNX	n24q02m/Qwen3-Reranker-0.6B-ONNX
GGUF	n24q02m/Qwen3-Embedding-0.6B-GGUF	n24q02m/Qwen3-Reranker-0.6B-GGUF

Installation

pip install qwen3-embed

# For GGUF support
pip install qwen3-embed[gguf]

Usage

Text Embedding

from qwen3_embed import TextEmbedding

# INT8 (default)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX")

# Q4F16 (smaller, slightly less accurate)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX-Q4F16")

# GGUF (requires: pip install qwen3-embed[gguf])
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF")

documents = [
    "Qwen3 is a multilingual embedding model.",
    "ONNX Runtime enables fast CPU inference.",
]

embeddings = list(model.embed(documents))
# Each embedding: numpy array of shape (1024,), L2-normalized

# Matryoshka Representation Learning (MRL) -- truncate to smaller dims
embeddings_256 = list(model.embed(documents, dim=256))
# Each embedding: numpy array of shape (256,), L2-normalized

# Query with instruction (for retrieval tasks)
queries = list(model.query_embed(
    ["What is Qwen3?"],
    task="Given a question, retrieve relevant passages",
))

Reranking

from qwen3_embed import TextCrossEncoder

reranker = TextCrossEncoder(model_name="n24q02m/Qwen3-Reranker-0.6B-ONNX")

# YesNo variant: ~10x less RAM (~598MB vs ~12GB at inference)
# reranker = TextCrossEncoder(model_name="n24q02m/Qwen3-Reranker-0.6B-ONNX-YesNo")

query = "What is Qwen3?"
documents = [
    "Qwen3 is a series of large language models by Alibaba.",
    "The weather today is sunny.",
    "Qwen3-Embedding supports multilingual text embedding.",
]

scores = list(reranker.rerank(query, documents))
# scores: list of float in [0, 1], higher = more relevant

# Or rerank pairs directly
pairs = [
    ("What is AI?", "Artificial intelligence is a branch of computer science."),
    ("What is ML?", "Machine learning is a subset of AI."),
]
pair_scores = list(reranker.rerank_pairs(pairs))

Configuration

GPU Acceleration

Both ONNX and GGUF backends auto-detect GPU when available (Device.AUTO is the default).

ONNX

Requires onnxruntime-gpu (CUDA) or onnxruntime-directml (Windows) instead of onnxruntime:

pip install onnxruntime-gpu  # NVIDIA CUDA
# or
pip install onnxruntime-directml  # Windows AMD/Intel/NVIDIA

from qwen3_embed import TextEmbedding, Device

# Auto-detect GPU (default)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX")

# Force CPU
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX", cuda=Device.CPU)

# Force CUDA
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-ONNX", cuda=Device.CUDA)

GGUF

GPU is handled by llama-cpp-python. The default pip install qwen3-embed[gguf] is CPU-only. For CUDA GPU support, build with:

CMAKE_ARGS="-DGGML_CUDA=on" pip install qwen3-embed[gguf]

from qwen3_embed import TextEmbedding, Device

# Auto-detect GPU (default, offloads all layers)
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF")

# Force CPU only
model = TextEmbedding(model_name="n24q02m/Qwen3-Embedding-0.6B-GGUF", cuda=Device.CPU)

Development

mise run setup   # Install deps + pre-commit hooks
mise run lint    # ruff check + format --check
mise run test    # pytest
mise run fix     # ruff auto-fix + format

Related Projects

• wet-mcp -- MCP web search server with vector-based docs search, uses qwen3-embed for local embedding
• mnemo-mcp -- MCP memory server with semantic search powered by qwen3-embed
• better-code-review-graph -- Knowledge graph for code reviews, uses qwen3-embed for local ONNX embedding
• modalcom-ai-workers -- GPU-serverless workers that convert Qwen3 models to ONNX/GGUF format

Contributing

See CONTRIBUTING.md.

License

Apache-2.0 -- See LICENSE. Original fastembed by Qdrant.