Fast Sentence Embeddings

Status

fse 1.x is in maintenance mode. 1.1.0 is the last feature release of the 1.x line. From here on, 1.x receives bug and compatibility fixes only.

fse 2.0 is in development as a rewrite by the maintainer. It will change the API and the file format of saved models; a migration guide and a way to export 1.x models will come with it. Until then, feature requests are closed.

• Version 1.1.0 is tested on Python 3.11 to 3.13 with current numpy, scipy, gensim and huggingface-hub releases.
• Python 3.14 and newer are not supported yet, because gensim does not build there (see gensim#3640).
• Bug reports are welcome.

If you start a new project, also look at actively maintained libraries such as sentence-transformers or model2vec.

Upgrading from 1.0

• SIF and uSIF give different vectors than 1.0. Their principal components are now computed exactly over all sentence vectors instead of from a random sample, and uSIF now weights the removed components like the reference implementation. Retrain SIF and uSIF models to get vectors that are comparable with new ones; uSIF models saved by 1.0 keep their old weighting when loaded. The STS Benchmark table below is remeasured.
• Arguments are validated strictly: wrong types and out-of-range values raise a pydantic.ValidationError instead of being coerced or ignored.
• fse 1.1 requires Python 3.11 or newer and adds the dependencies pydantic and threadpoolctl.

All changes are listed in CHANGELOG.md.

About

Fast Sentence Embeddings is a Python library that serves as an addition to Gensim. This library is intended to compute sentence vectors for large collections of sentences or documents with as little hassle as possible:

from fse import Vectors, Average, IndexedList

vecs = Vectors.from_pretrained("glove-wiki-gigaword-50")
model = Average(vecs)

sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]]

model.train(IndexedList(sentences))

model.sv.similarity(0,1)

Audience

This package builds upon Gensim and is intended to compute sentence/paragraph vectors for large databases. Use this package if:

• (Sentence) Transformers are too slow
• Your dataset is too large for existing solutions (spacy)
• Using GPUs is not an option.

The average (online) inference time for a well optimized (and batched) sentence-transformer is around 1ms-10ms per sentence. If that is not enough and you are willing to sacrifice a bit in terms of quality, this is your package.

Features

Find the corresponding blog post(s) here (code may be outdated):

• Visualizing 100,000 Amazon Products
• Sentence Embeddings. Fast, please!

fse implements three algorithms for sentence embeddings. You can choose between unweighted sentence averages, smooth inverse frequency averages, and unsupervised smooth inverse frequency averages.

Key features of fse are:

[X] Millions of sentences per second with several workers (1)

[X] Provides HUB access to various pre-trained models for convenience

[X] Supports Average, SIF, and uSIF Embeddings

[X] Full support for Gensims Word2Vec and all other compatible classes

[X] Full support for Gensims FastText with out-of-vocabulary words

[X] Induction of word frequencies for pre-trained embeddings

[X] Incredibly fast Cython core routines

[X] Dedicated input file formats for easy usage (including disk streaming)

[X] Ram-to-disk training for large corpora

[X] Disk-to-disk training for even larger corpora

[X] Many fail-safe checks for easy usage

[X] Simple interface for developing your own models

[X] Extensive documentation of all functions

[X] Optimized Input Classes

(1) Measured on 16 vCPUs of an AMD EPYC with 16 workers: up to 2.2 million sentences per second for Average and 1.7 million for SIF, reading a CorpusFile of 2 million sentences with 20 tokens and 300 dimensions; one worker reaches 180k to 440k. The numbers vary with the machine and its load; see Choosing an input class and measure your own setup with benchmarks/bench_train.py.

Installation

Pre-built wheels are available for Linux (x86_64, aarch64), macOS (arm64, x86_64) and Windows (x86_64) on Python 3.11 to 3.13:

pip install -U fse

or, with uv:

uv add fse

On other platforms pip builds the Cython extension from the source distribution. This needs a C compiler; Cython and numpy are installed automatically during the build. To build from a clone of this repository:

uv pip install .

If the C extension cannot be loaded, fse falls back to a much slower numpy implementation and emits a RuntimeWarning.

Usage

Using pre-trained models with fse is easy. You can just use them from the hub and download them accordingly. They will be stored locally so you can re-use them later.

from fse import Vectors, Average, IndexedList
vecs = Vectors.from_pretrained("glove-wiki-gigaword-50")
model = Average(vecs)

sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]]

model.train(IndexedList(sentences))

model.sv.similarity(0,1)

If your vectors are large and you don't have a lot of RAM, you can supply the mmap argument as follows to read the vectors from disk instead of loading them into RAM:

Vectors.from_pretrained("glove-wiki-gigaword-50", mmap="r")

To check which vectors are on the hub, please check: https://huggingface.co/fse. For example, you will find:

• glove-twitter-25
• glove-twitter-50
• glove-twitter-100
• glove-twitter-200
• glove-wiki-gigaword-100
• glove-wiki-gigaword-300
• word2vec-google-news-300
• paragram-25
• paranmt-300
• paragram-300-sl999
• paragram-300-ws353
• fasttext-wiki-news-subwords-300
• fasttext-crawl-subwords-300 (Use with FTVectors)

Only load models you trust. Vectors.from_pretrained, FTVectors.from_pretrained and the load methods of all models unpickle the downloaded or stored files, and unpickling untrusted data can execute arbitrary code.

In order to use fse with a custom model you must first estimate a Gensim model which contains a gensim.models.keyedvectors.KeyedVectors class, for example Word2Vec or Fasttext. Then you can proceed to compute sentence embeddings for a corpus as follows:

from gensim.models import FastText
sentences = [["cat", "say", "meow"], ["dog", "say", "woof"]]
ft = FastText(sentences, min_count=1, vector_size=10)

from fse import Average, IndexedList
model = Average(ft)
model.train(IndexedList(sentences))

model.sv.similarity(0,1)

Choosing an input class

Every model trains on (tokens, index) tuples; the input classes produce them. Training uses one thread unless you pass workers; on a machine with several cores, pass workers=os.cpu_count(). With two or more workers, IndexedList, CIndexedList, SplitIndexedList, TextList and CorpusFile are read by every worker for its own part of the input, so they scale with workers; IndexedLineDocument, SplitCIndexedList and the classes with your own splitting are read in Python by one producer thread, which limits them from about two to four workers on. While a model with several workers trains, fse limits the threads of BLAS (OpenBLAS, MKL, Accelerate) to share the cores, for the whole process.

Measured on 16 vCPUs of an AMD EPYC (2 million sentences of 20 tokens, 300 dimensions; the vectors are bit-identical to those of the single producer thread), in sentences per second:

With the single producer thread before #88, 8 workers reached 325k (IndexedList, Average) and 191k (IndexedList, SIF) sentences per second on the same machine. The numbers vary with the load of the machine; measure your own setup with benchmarks/bench_train.py.

import os

from fse import Average, CorpusFile, TextList

model = Average(ft, workers=os.cpu_count())
model.train(CorpusFile("sentences.txt"))  # line i of the file is sentence i
vectors = model.infer(TextList(["cat say meow", "dog say woof"]))

CorpusFile and TextList give exactly the sentences of IndexedLineDocument and SplitIndexedList: a line ends at \n, text is decoded strictly as UTF-8, and words are split at every character for which str.isspace() is true. Invalid UTF-8 raises a UnicodeDecodeError that names the sentence. The file must not change from the scan to the end of training: changes that alter its size, modification time or inode raise an error, but truncating a memory-mapped file while it is read can end the process on Linux.

To keep only close matches, pass min_similarity to most_similar or the similar_by_* methods:

model.sv.most_similar(0, topn=10, min_similarity=0.8)

Smaller sentence vectors

sv_dtype="float16" halves the memory of the sentence vectors and sv_dtype="int8" quarters it. The vectors are still computed in float32; they are only stored smaller, and every read (sv[i], get_vector, get_vectors, iter_blocks) returns float32. Training, updates, infer, the similarity searches, save/load and memory-mapped files (sv_mapfile_path) work as with float32.

from fse import SIF, CorpusFile

model = SIF(ft, sv_dtype="int8", sv_mapfile_path="vectors", workers=8)
model.train(CorpusFile("sentences.txt"))
model.sv.most_similar(0, topn=10)

model.sv.convert("float16")  # change the format of trained or loaded vectors

int8 stores one float32 scale per vector, so the cosine of two stored int8 vectors equals the cosine of the vectors they represent. On the STS Benchmark (all 15 vector sets of the table below, with CBOW, SIF-10 and uSIF), float16 changes the scores by at most 0.01 points and int8 by at most 0.02 points. Both round the vectors, so close neighbours can swap places: float16 changes at most 0.4 % and int8 up to 7 % of the 10 nearest neighbours of a sentence. Keep float32 where neighbour lists must match it exactly.

With index i for the i-th sentence (every input class except the C* lists with custom indices) and the compiled extension, training holds a ring of float32 windows of about 256 MB each instead of a float32 matrix of all sentences, and stores every finished window right away. Jobs hold at most 65,536 sentences, so the ring has two windows for up to 1,024 dimensions and a few more beyond. SIF and uSIF read the input twice for that, once to learn the principal components and once to store the vectors. Other inputs, and subclasses that replace the training hooks, train into a float32 matrix of all new rows first, and the log says why.

To apply a model to all vectors, for example a logistic regression, use score. It reads float16 and int8 vectors as stored, in parallel, and can write into a memmap:

import numpy as np

logits = model.sv.score(clf.coef_.T, clf.intercept_)  # X @ W + b, in blocks of rows
probabilities = 1 / (1 + np.exp(-logits))

for first_row, block in model.sv.iter_blocks():  # float32 blocks, for anything else
    ...

Smaller word vectors

wv_dtype="float16" or "int8" makes the model keep the word vectors it averages (and the ngram vectors of FastText models) in half or a quarter of the memory. The model quantizes a copy once when it is created; the word vectors you pass in stay unchanged. To keep their float32 values out of RAM as well, load them memory-mapped, for example Vectors.from_pretrained("word2vec-google-news-300", mmap="r"). With wv_mapfile_path the codes are written to a file and memory-mapped.

from fse import SIF, CorpusFile, Vectors

vectors = Vectors.from_pretrained("word2vec-google-news-300", mmap="r")
model = SIF(vectors, wv_dtype="int8", sv_dtype="int8", workers=8)
model.train(CorpusFile("sentences.txt"))

On the STS Benchmark (all 15 vector sets, with CBOW, SIF-10 and uSIF), float16 word vectors change the scores by at most 0.01 points and int8 by at most 0.12 points (glove-twitter-25 with SIF-10; every 300-dimensional set stays within 0.02 points).

The main benefit is memory. Averaging also gets somewhat faster where fse reads the codes with vector instructions: on x86-64 Linux and macOS with AVX2, FMA and F16C (Intel since Haswell, AMD since Zen), and on Apple Silicon and other arm64 CPUs with NEON; fse picks the routines when it is imported. Measured end to end (training from a CorpusFile of 2 million sentences, 300 dimensions, Average and SIF), int8 word vectors were, compared with float32 word vectors in fse 1.0:

On x86-64 CPUs without AVX2 and on Windows, fse uses portable routines instead: float16 and int8 then save memory but train 20 to 35 % slower (measured on the AMD machine with the portable routines forced). float32 itself keeps its results unchanged; prefetching the rows of the next words made it up to about 1.2 times as fast.

Additional Information

Within the folder notebooks you can find the following guides, run with fse 1.1.0:

Tutorial.ipynb offers a walk-through of the most important functions: the input classes, training with several workers, inference, similarity queries, smaller vectors and score.

STS-Benchmarks.ipynb contains an example of how to use the library with pre-trained models to replicate the STS Benchmark results [4] reported in the papers; it produces the table under Results.

Speed Comparison.ipynb times training for the input classes, numbers of workers and storage formats. It replaces Speed Comparision.ipynb, which compared the numpy and Cython routines through internals that no longer exist.

In order to use the fse model, you first need some pre-trained gensim word embedding model, which is then used by fse to compute the sentence embeddings.

After computing sentence embeddings, you can use them in supervised or unsupervised NLP applications, as they serve as a formidable baseline.

The models presented are based on

• Deep-averaging embeddings [1]
• Smooth inverse frequency embeddings [2]
• Unsupervised smooth inverse frequency embeddings [3]

Credits to Radim Řehůřek and all contributors for the awesome library and code that Gensim provides. A whole lot of the code found in this lib is based on Gensim.

Results

Pearson correlation (x100) on the test set of the STS Benchmark, computed with the procedure of notebooks/STS-Benchmarks.ipynb (lang_freq="en") with fse 1.1.0 on Python 3.13, numpy 2.5, scipy 1.18, gensim 4.4, wordfreq 3.1 and nltk 3.10. SIF and uSIF compute their principal components exactly since 1.1.0; against the randomized SVD of scikit-learn 1.9 used before, in the same environment, SIF-10 scores move by at most 0.04 points and uSIF scores not at all. Compared with the table published for fse 1.0, uSIF scores rise by 0.2 to 1.4 points. Most of that comes from a fix in 1.1.0 to how uSIF weights the removed principal components, worth 0.2 to 1.3 points when both formulas run in this environment (see CHANGELOG.md). Newer dependency versions shift SIF scores by up to 0.3 points and CBOW scores by up to 0.04 points.

Literature

1. Iyyer M, Manjunatha V, Boyd-Graber J, Daumé III H (2015) Deep Unordered Composition Rivals Syntactic Methods for Text Classification. Proc. 53rd Annu. Meet. Assoc. Comput. Linguist. 7th Int. Jt. Conf. Nat. Lang. Process., 1681-1691.

2. Arora S, Liang Y, Ma T (2017) A Simple but Tough-to-Beat Baseline for Sentence Embeddings. Int. Conf. Learn. Represent. (Toulon, France), 1-16.

3. Ethayarajh K (2018) Unsupervised Random Walk Sentence Embeddings: A Strong but Simple Baseline. Proceedings of the 3rd Workshop on Representation Learning for NLP. (Toulon, France), 91-100.

4. Eneko Agirre, Daniel Cer, Mona Diab, Iñigo Lopez-Gazpio, Lucia Specia. Semeval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation. Proceedings of SemEval 2017.

Changelog

See CHANGELOG.md.

Development

The project uses uv and a Makefile:

make install   # create the environment and build the extension
make check     # format check, lint, type check and tests

See CONTRIBUTING.md for details and SECURITY.md for reporting vulnerabilities.

Author

Built by Dr. Oliver Borchers. For questions that do not belong in a GitHub issue, use the contact page.

Copyright (C) 2019 to 2026 Oliver Borchers, licensed under the GNU General Public License v3.0.

Citation

If you found this software useful, please cite it in your publication.

@misc{Borchers2019,
	author = {Borchers, Oliver},
	title = {Fast sentence embeddings},
	year = {2019},
	publisher = {GitHub},
	journal = {GitHub Repository},
	howpublished = {\url{https://github.com/oborchers/Fast_Sentence_Embeddings}},
}