Skip to main content

# AutoFaiss

Project description

AutoFaiss

pypi ci

Automatically create Faiss knn indices with the most optimal similarity search parameters.

It selects the best indexing parameters to achieve the highest recalls given memory and query speed constraints.

How to use autofaiss?

To install run pip install autofaiss

It's probably best to create a virtual env:

python -m venv .venv/autofaiss_env
source .venv/autofaiss_env/bin/activate
pip install -U pip
pip install autofaiss

Create embeddings

import os
import numpy as np
embeddings = np.random.rand(1000, 100)
os.mkdir("embeddings")
np.save("embeddings/part1.npy", embeddings)
os.mkdir("my_index_folder")

Generate a Knn index

autofaiss quantize --embeddings_path="embeddings" --output_path="my_index_folder" --metric_type="ip"

Try the index

import faiss
import glob
import numpy as np

my_index = faiss.read_index(glob.glob("my_index_folder/*.index")[0])

query_vector = np.float32(np.random.rand(1, 100))
k = 5
distances, indices = my_index.search(query_vector, k)

print(list(zip(distances[0], indices[0])))

autofaiss quantize

embeddings_path -> local path of the embeddings in numpy. output_path -> destination path on the hdfs for the created index. metric_type -> Similarity distance for the queries.

index_key -> (optional) describe the index to build.
index_param -> (optional) describe the hyperparameters of the index.
memory_available -> (optional) describe the amount of memory available on the machine.
use_gpu -> (optional) wether to use GPU or not (not tested).

Install from source

First, create a virtual env and install dependencies:

python -m venv .venv/autofaiss_env
source .venv/autofaiss_env/bin/activate
make install

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

autofaiss-1.0.1.tar.gz (29.2 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page