A set of tools to compress gensim fasttext models
This Python 3 package allows to compress fastText word embedding models
gensim package) by orders of magnitude,
without seriously affecting their quality. It can be installed with
pip install compress-fasttext
This blogpost (in Russian) gives more details about the motivation and methods for compressing fastText models.
You can use this package to compress your own fastText model (or one downloaded e.g. from RusVectores):
import gensim import compress_fasttext big_model = gensim.models.fasttext.FastTextKeyedVectors.load('path-to-original-model') small_model = compress_fasttext.prune_ft_freq(big_model, pq=True) small_model.save('path-to-new-model')
Different compression methods include:
- matrix decomposition (
- product quantization (
- optimization of feature hashing (
- feature selection (
The recommended approach is combination of feature selection and quantization (
compress-fasttext is already installed, you can download and use this tiny model
import compress_fasttext small_model = compress_fasttext.models.CompressedFastTextKeyedVectors.load( 'https://github.com/avidale/compress-fasttext/releases/download/v0.0.1/ft_freqprune_100K_20K_pq_100.bin' ) print(small_model['спасибо'])
CompressedFastTextKeyedVectors inherits from
but makes a few additional optimizations.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size compress-fasttext-0.0.5.tar.gz (8.2 kB)||File type Source||Python version None||Upload date||Hashes View|