Skip to main content

Accelerating LLM Inference via Vector Index Based Output Embeddings

Paper

Arxiv Link (online soon)

Abstract

Large output embedding matrices create a significant memory bandwidth bottleneck during autoregressive decoding, especially for compact LLMs with large multilingual vocabularies. We reformulate the output projection followed by top-k token selection as a maximum inner product search over token embeddings and replace the dense vocabulary projection with an HNSW-based vector index. The resulting output head retrieves only a small candidate set of high-scoring tokens and can be integrated into existing decoding pipelines by scattering retrieved logits into a sparse full-vocabulary tensor. On CPU inference with Gemma 3, Llama 3.2, and Qwen 3 models, our method substantially accelerates the output projection and improves end-to-end batch-size-one decoding throughput by up to 82% for Gemma 3 270M, while preserving generation quality under AlpacaEval evaluation. These results suggest approximate retrieval is a practical alternative to dense output projections in latency-sensitive small-batch decoding.

Getting Started

For a full walkthrough, check out the quickstart notebook.

Using the vector index embedding is straightforward and integrates seamlessly with Hugging Face transformers by simply replacing the model's standard lm_head.

For immediate use, we have provided prebuilt vector indices for the following models:

  • google/gemma-3-270m-it / google/gemma-3-1b-it
  • Qwen/Qwen3-0.6B / Qwen/Qwen3-1.7B
  • meta-llama/Llama-3.2-1B-Instruct /meta-llama/Llama-3.2-3B-Instruct

Install

pip install vector-index-embedding transformers

Run

from transformers import pipeline
from vectorindex import VectorIndexEmbedding

model_id = "Qwen/Qwen3-0.6B"
pipe = pipeline("text-generation", model=model_id, device="cpu")
pipe.model.lm_head = VectorIndexEmbedding.from_pretrained(model_id, ef=200)

prompt = [{"role": "user", "content": "Who was Alan Turing?"}]
out = pipe(prompt)
print(out[0]["generated_text"][-1]["content"])

Faster hnswlib

Our faster hnswlib implementation can be found here: github.com/martinloretzzz/hnswlib

Warning: This implementation might not work on all systems as it was only tested on the one where we're running the benchmarks and the SIMD implementation was only adapted for that architecture.

This fork has 2 improvements for fast inner products on high dimensional data:

  • We calculate all the inner products in parallel, that way reducing memory accesses in half (we load one element of the query and compare it to N other vectors at the same time)
  • We removed a heuristic that restricted multi-threading, as our data is extremely high dimensional and always benefits from using all cores.

Citation

If you use this work, please cite:

TBD

Metadata

Release files for vector-index-embedding 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for vector-index-embedding 0.1.1
File Size Uploaded Provenance Actions
vector_index_embedding-0.1.1.tar.gz 4.6 kB Without provenance Details for vector_index_embedding-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for vector-index-embedding 0.1.1
File Interpreter ABI Platform Actions
vector_index_embedding-0.1.1-py3-none-any.whl Python 3 none any Details for vector_index_embedding-0.1.1-py3-none-any.whl

Total release size: 9.8 kB

Release files / vector_index_embedding-0.1.1.tar.gz

Download URL vector_index_embedding-0.1.1.tar.gz
Size 4.6 kB
Tags Source
SHA-256 checksum
How to use checksums
5a1c3be51d892cc7c7c96332fbe86476350d7ee95ba2db7976d4b151cb033b58
BLAKE2b-256 checksum
How to use checksums
2ed1709fa19de6b5fbcdc83d617a2a337576558be8d8c6cf3e649017d4eec299
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.5

Release files / vector_index_embedding-0.1.1-py3-none-any.whl

Download URL vector_index_embedding-0.1.1-py3-none-any.whl
Size 5.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
9aabf397d87ae484871622310a7c107616600af12fd3d03ffbd4cfa1719b2402
BLAKE2b-256 checksum
How to use checksums
2c814b93695857697faaa95229b5f0eeceddbb5ee96a721cb49e8ba025b4d3e9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.5

Security for vector-index-embedding 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page