embed-rerank 1.1.0

Single Model Embedding & Reranker API with Apple Silicon acceleration

🔥 Single Model Embedding & Reranking API

Lightning-fast local embeddings & reranking for Apple Silicon (MLX-first, OpenAI & TEI compatible)

---

⚡ Why This Matters

Transform your text processing with 10x faster embeddings and reranking on Apple Silicon. Drop-in replacement for OpenAI API and Hugging Face TEI with zero code changes required.

🏆 Performance Comparison

Operation	This API (MLX)	OpenAI API	Hugging Face TEI
Embeddings	0.78ms	200ms+	15ms
Reranking	1.04ms	N/A	25ms
Model Loading	0.36s	N/A	3.2s
Cost	$0	$0.02/1K	$0

Tested on Apple M4 Max

---

🚀 Quick Start

Option 1: Install from PyPI (Recommended)

# Install the package
pip install embed-rerank

# Start the server (default port 9000)
embed-rerank

# Or with custom port and options
embed-rerank --port 8080 --host 127.0.0.1

# See all options
embed-rerank --help

CLI Options:

• --host: Server host (default: 0.0.0.0)
• --port: Server port (default: 9000)
• --reload: Enable auto-reload for development
• --log-level: Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)

Testing Options:

• --test quick: Run quick validation tests
• --test performance: Run performance benchmark tests
• --test quality: Run quality validation tests
• --test full: Run comprehensive test suite
• --test-url: Custom server URL for testing
• --test-output: Test output directory

Performance Testing:

# Start server in background
embed-rerank --port 8080 &

# Run performance tests
embed-rerank --test performance --test-url http://localhost:8080

# Run comprehensive test suite
embed-rerank --test full --test-url http://localhost:8080

# Stop server
pkill -f embed-rerank

Environment Variables:

# Alternative: Use environment variables
export PORT=8080
export HOST=127.0.0.1
embed-rerank

Option 2: From Source

# 1. Clone and setup
git clone https://github.com/joonsoo-me/embed-rerank.git
cd embed-rerank
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

# 2. Start server (macOS/Linux)
./tools/server-run.sh

# 3. Test it works
curl http://localhost:9000/health/

🎉 Done! Visit http://localhost:9000/docs for interactive API documentation.

---

🛠 Server Management (macOS/Linux)

# Start server (background)
./tools/server-run.sh

# Start server (foreground/development)
./tools/server-run-foreground.sh

# Stop server
./tools/server-stop.sh

Windows Support: Coming soon! Currently optimized for macOS/Linux.

---

⚙️ Configuration

Create .env file (optional):

# Server
PORT=9000
HOST=0.0.0.0

# Backend
BACKEND=auto                                   # auto | mlx | torch
MODEL_NAME=mlx-community/Qwen3-Embedding-4B-4bit-DWQ

# Model Cache (first run downloads ~2.3GB model)
MODEL_PATH=                               # Custom model directory
TRANSFORMERS_CACHE=                           # HF cache override
# Default: ~/.cache/huggingface/hub/

# Performance
BATCH_SIZE=32
MAX_TEXTS_PER_REQUEST=100

---

📂 Model Cache Management

The service automatically manages model downloads and caching:

Environment Variable	Purpose	Default
MODEL_PATH	Custom model directory	(uses HF cache)
TRANSFORMERS_CACHE	Override HF cache location	~/.cache/huggingface/transformers
HF_HOME	HF home directory	~/.cache/huggingface
(auto)	Default HF cache	~/.cache/huggingface/hub/

Cache Location Check

# Find where your model is cached
python3 -c "
import os
print('MODEL_PATH:', os.getenv('MODEL_PATH', '<not set>'))
print('TRANSFORMERS_CACHE:', os.getenv('TRANSFORMERS_CACHE', '<not set>'))
print('HF_HOME:', os.getenv('HF_HOME', '<not set>'))
print('Default cache:', os.path.expanduser('~/.cache/huggingface/hub'))
"

# List cached Qwen3 models
ls ~/.cache/huggingface/hub | grep -i qwen3 || echo "No Qwen3 models found in cache"

---

🌐 Three APIs, One Service

API	Endpoint	Use Case
Native	/api/v1/embed, /api/v1/rerank	New projects
OpenAI	/v1/embeddings	Existing OpenAI code
TEI	/embed, /rerank	Hugging Face TEI replacement

OpenAI Compatible (Drop-in)

import openai

client = openai.OpenAI(
    api_key="dummy-key",
    base_url="http://localhost:9000/v1"
)

response = client.embeddings.create(
    input=["Hello world", "Apple Silicon is fast!"],
    model="text-embedding-ada-002"
)
# 🚀 10x faster than OpenAI, same code!

TEI Compatible

curl -X POST "http://localhost:9000/embed" 
  -H "Content-Type: application/json" 
  -d '{"inputs": ["Hello world"], "truncate": true}'

Native API

# Embeddings
curl -X POST "http://localhost:9000/api/v1/embed/" 
  -H "Content-Type: application/json" 
  -d '{"texts": ["Apple Silicon", "MLX acceleration"]}'

# Reranking  
curl -X POST "http://localhost:9000/api/v1/rerank/" 
  -H "Content-Type: application/json" 
  -d '{"query": "machine learning", "passages": ["AI is cool", "Dogs are pets", "MLX is fast"]}'

---

🧪 Testing

Built-in Performance Testing

# Quick validation (basic functionality)
embed-rerank --test quick

# Performance benchmark (latency, throughput, concurrency)
embed-rerank --test performance --test-url http://localhost:9000

# Quality validation (semantic similarity, multilingual)  
embed-rerank --test quality --test-url http://localhost:9000

# Full comprehensive test suite
embed-rerank --test full --test-url http://localhost:9000

Test Results Include:

• 📊 Latency metrics (mean, P95, P99)
• 🚀 Throughput analysis (texts/sec)
• 🔄 Concurrency testing
• 🧠 Semantic similarity validation
• 🌍 Multilingual support testing
• 📈 JSON reports with detailed metrics

Advanced Testing

# Comprehensive test suite (shell script)
./tools/server-tests.sh

# Quick health & model loaded info check
curl http://localhost:9000/health/

# Run pytest
pytest tests/ -v

---

🚀 What You Get

• ✅ Zero Code Changes: Drop-in replacement for OpenAI API and TEI
• ⚡ 10x Performance: Apple MLX acceleration on Apple Silicon
• 💰 Zero Costs: No API fees, runs locally
• 🔒 Privacy: Your data never leaves your machine
• 🎯 Three APIs: Native, OpenAI, and TEI compatibility
• 📊 Production Ready: Health checks, monitoring, structured logging

---

📄 License

MIT License - build amazing things with this code!