LibVQ

A Library For Dense Retrieval Oriented Vector Quantization

Introduction

Vector quantization (VQ) is widely applied to many ANN libraries, like FAISS, ScaNN, SPTAG, DiskANN to facilitate real-time and memory-efficient dense retrieval. However, conventional vector quantization methods, like IVF, PQ, OPQ, are not optimized for the retrieval quality. In this place, We present LibVQ, the first library developed for dense retrieval oriented vector quantization. LibVQ is highlighted for the following features:

• Knowledge Distillation. The knowledge distillation based learning process can be directly applied to the off-the-shelf embeddings. It gives rise to the strongest retrieval performance in comparison with any existing VQ based ANN indexes.

• Flexible usage and input conditions. LibVQ may flexibly support different usages, e.g., training VQ parameters only, or joint adaptation of query encoder. LibVQ is designed to handle a wide range of input conditions: it may work only with off-the-shelf embeddings; it may also leverage extra data, e.g., relevance labels, and source queries, for further enhancement.

• Learning and Deployment. The learning is backended by PyTorch, which can be easily configured for the efficient training based on different computation resources. The well-trained VQ parameters are wrapped up with FAISS backend ANN indexes, e.g., IndexPQ, IndexIVFPQ, etc., which are directly deployable for large-scale dense retrieval applications.

Install

• From PyPI

pip install LibVQ

• From source

git clone https://github.com/staoxiao/LibVQ.git
cd LibVQ
pip install .

Workflow

In LibVQ, users can construct a index and train it by a simple way. Please refer to our docs for more details. Besides, we provide some examples below to illustrate the usage of LibVQ.

Examples

MSMARCO

• IVFPQ (Compression Ratio = 96)

• PQ (Compression Ratio = 96)

NQ

• IVFPQ (Compression Ratio = 384)

• PQ (Compression Ratio = 384)

Related Work

• Distii-VQ: Unifies the learning of IVF and PQ within a knowledge distillation framework. Accpted as a full paper by SIGIR 2022.

• BiDR: Applies the learnable PQ in large-scale index and proposes the progressively optimized docs' embeddings for the better retrieval performance. Accpted as a full paper by WWW 2022.

• MoPQ: This work identifies the limitation of using reconstruction loss minimization as the training objective of learnable PQ and proposes the Multinoulli Contrastive Loss. Accpted as a full paper by EMNLP 2021.