Skip to main content

Finalfusion in Python

Project description

finalfusion-python

Documentation Status

Introduction

finalfusion is a Python package for reading, writing and using finalfusion embeddings, but also supports other commonly used embeddings like fastText, GloVe and word2vec.

The Python package supports the same types of embeddings as the finalfusion-rust crate:

  • Vocabulary:
    • No subwords
    • Subwords
  • Embedding matrix:
    • Array
    • Memory-mapped
    • Quantized
  • Norms
  • Metadata

Installation

The finalfusion module is available on PyPi for Linux, Mac and Windows. You can use pip to install the module:

$ pip install --upgrade finalfusion

Installing from source

Building from source depends on Cython. If you install the package using pip, you don't need to explicitly install the dependency since it is specified in pyproject.toml.

$ git clone https://github.com/finalfusion/finalfusion-python
$ cd finalfusion-python
$ pip install .

If you want to build wheels from source, wheel needs to be installed. It's then possible to build wheels through:

$ python setup.py bdist_wheel

The wheels can be found in dist.

Package Usage

Basic usage

import finalfusion
# loading from different formats
w2v_embeds = finalfusion.load_word2vec("/path/to/w2v.bin")
text_embeds = finalfusion.load_text("/path/to/embeds.txt")
text_dims_embeds = finalfusion.load_text_dims("/path/to/embeds.dims.txt")
fasttext_embeds = finalfusion.load_fasttext("/path/to/fasttext.bin")
fifu_embeds = finalfusion.load_finalfusion("/path/to/embeddings.fifu")

# serialization to formats works similarly
finalfusion.compat.write_word2vec("to_word2vec.bin", fifu_embeds)

# embedding lookup
embedding = fifu_embeds["Test"]

# reading an embedding into a buffer
import numpy as np
buffer = np.zeros(fifu_embeds.storage.shape[1], dtype=np.float32)
fifu_embeds.embedding("Test", out=buffer)

# similarity and analogy query
sim_query = fifu_embeds.word_similarity("Test")
analogy_query = fifu_embeds.analogy("A", "B", "C")

# accessing the vocab and printing the first 10 words
vocab = fifu_embeds.vocab
print(vocab.words[:10])

# SubwordVocabs give access to the subword indexer:
subword_indexer = vocab.subword_indexer
print(subword_indexer.subword_indices("Test", with_ngrams=True))

# accessing the storage and calculate its dot product with an embedding
res = embedding.dot(fifu_embeds.storage)

# printing metadata
print(fifu_embeds.metadata) 

Beyond Embeddings

# load only a vocab from a finalfusion file
from finalfusion import load_vocab
vocab = load_vocab("/path/to/finalfusion_file.fifu")

# serialize vocab to single file
vocab.write("/path/to/vocab_file.fifu.voc")

# more specific loading functions exist
from finalfusion.vocab import load_finalfusion_bucket_vocab
fifu_bucket_vocab = load_finalfusion_bucket_vocab("/path/to/vocab_file.fifu.voc")

The package supports loading and writing all finalfusion chunks this way. This is only supported by the Python package, reading will fail with e.g. the finalfusion-rust.

Scripts

finalfusion also includes a conversion script ffp-convert to convert between the supported formats.

# convert from fastText format to finalfusion
$ ffp-convert -f fasttext fasttext.bin -t finalfusion embeddings.fifu

ffp-bucket-to-explicit can be used to convert bucket embeddings to embeddings with an explicit ngram lookup.

# convert finalfusion bucket embeddings to explicit
$ ffp-bucket-to-explicit -f finalfusion embeddings.fifu explicit.fifu

Finally, the package comes with ffp-similar and ffp-analogy to do analogy and similarity queries.

# get the 5 nearest neighbours of "Tübingen"
$ echo Tübingen | ffp-similar embeddings.fifu
# get the 5 top answers for "Tübingen" is to "Stuttgart" like "Heidelberg" to...
$ echo Tübingen Stuttgart Heidelberg | ffp-analogy embeddings.fifu

Where to go from here

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for finalfusion, version 0.7.1
Filename, size File type Python version Upload date Hashes
Filename, size finalfusion-0.7.1-cp36-cp36m-macosx_10_14_x86_64.whl (297.6 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp36-cp36m-manylinux2014_x86_64.whl (688.6 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp36-cp36m-win_amd64.whl (302.5 kB) File type Wheel Python version cp36 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp37-cp37m-macosx_10_14_x86_64.whl (296.7 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp37-cp37m-manylinux2014_x86_64.whl (689.1 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp37-cp37m-win_amd64.whl (302.3 kB) File type Wheel Python version cp37 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp38-cp38-macosx_10_14_x86_64.whl (298.0 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp38-cp38-manylinux2014_x86_64.whl (780.2 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size finalfusion-0.7.1-cp38-cp38m-win_amd64.whl (303.7 kB) File type Wheel Python version cp38 Upload date Hashes View
Filename, size finalfusion-0.7.1.tar.gz (227.8 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page