local-llm-embed

Get sentence embeddings out of any local causal LLM you already have running. No fine-tuning, no separate encoder model.

If you're running Llama / Qwen / Phi / Mistral / TinyLlama via transformers, Ollama, vLLM, or llama.cpp and want embeddings for RAG / retrieval / classification — this library extracts them from the model you already have, in a few lines of code.

Why

The trade-off is honest: dedicated encoders still beat raw LLM probes on pure semantic similarity (STS) by ~6 points. But on classification-style tasks like Banking77, this library matches or slightly beats the baseline — using a model you already have in memory.

Install

pip install local-llm-embed

Or with the Hugging Face Hub helpers (for downloading pre-fit whiteners):

pip install "local-llm-embed[hub]"

Quick start

from local_llm_embed import LocalLLMEmbedder

embedder = LocalLLMEmbedder("Qwen/Qwen2.5-0.5B-Instruct")
emb = embedder.encode(["The cat sat on the mat.", "A feline rested."])
print(emb.shape)  # (2, 896)
print(emb @ emb.T)  # cosine similarity matrix

By default this uses prefix+whiten (the most universally strong variation in our benchmarks). The whitener is fit lazily on the first batch you encode.

Use a calibration set for better whitening

calibration = [...]  # ~1000 representative texts from your domain
embedder.fit_whitener(calibration)
embedder.save_whitener("./domain_whiten.npz")

# later, in another session:
embedder = LocalLLMEmbedder("Qwen/Qwen2.5-0.5B-Instruct",
                             whitener_path="./domain_whiten.npz")

Pick a different variation

embedder = LocalLLMEmbedder(
    "microsoft/Phi-3.5-mini-instruct",
    variation="echo+whiten",   # best for STS
    layer="final",
    pooling="weighted_mean",
)

Variations

Three train-free recipes are bundled. They're combinations of well-known techniques, ranked by how well they performed in our internal benchmark (STSB validation, Banking77; see BENCHMARKS.md):

• prefix+whiten (default) — feed the text in, take the chosen layer + pooling, then center & ZCA-whiten the resulting matrix. Whitening removes the anisotropy ("everything-looks-similar") problem that causal LMs suffer from. Universal +0.12 to +0.22 STS Spearman over no whitening.
• echo+whiten — duplicate the text and pool only over the second copy, so each pooled token has seen the full sentence (works around the causal mask). Best STS combination in our tests.
• prefix — no transformation. For comparison / debugging.

Hardware notes

Causal LMs at fp32 are heavy on RAM. We default to bf16 if your CPU has the avx512_bf16 flag (most modern AMD / Intel desktop CPUs do); on GPU just pass device="cuda" and the same bf16 default applies.

Limitations (be honest)

• For pure semantic textual similarity, dedicated contrastive encoders (BGE-M3, all-MiniLM-L6-v2) still win by ~6 STS points. This library is for the case where you already have a causal LM loaded.
• Whitening requires a calibration set of at least a few hundred texts. Without it, self-whitening is used at encode time (fits on the batch you're encoding). That's worse than a good calibration set but still better than raw probes.
• bidirectional inference (LLM2Vec-style attention-mask removal) is not bundled. We benchmarked it; it consistently hurt without fine-tuning and we don't want to ship a footgun.

Acknowledgements

The technique is a combination of BERT-whitening (Su et al. 2021), Echo Embeddings (Springer et al. 2024), and PromptEOL (Jiang et al. 2023). This library packages the train-free subset.

License

MIT.