Compute Word Mover's Distance using any type of Word Embedding model
Project description
Word Mover's Distance
In this package you will find the implementation of Word Mover's Distance for a generic Word Embeddings model.
I largely reused code available in the gensim library, in particular the wmdistance function, making it more general so that it can be used with other Word Embeddings models, such as GloVe.
You can find a real-world usage of this package in my news summariser repository, where I use Word Mover's distance for finding the most similar sentences in a given news article.
How to install
The preferred way to install this package is through pip:
pip install word-mover-distance
On Mac and Linux it works like a charm. On Windows, however, it is highly likely you will experience some issues: this is due to pyemd, which needs some C++ dependencies during build time. A quick way to solve this issue is to install "Build Tools for Visual Studio 2019" following this procedure:
- Go to the following page and download "Build Tools for Visual Studio 2019" https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019
- Double click on the .exe file once finished and select to install C++ build tools
- Check that among the suggested packages to install it is also selected "Windows 10 SDK" (the newest version is fine) as this is the critical dependency
- Once the installation has finished reopen your PowerShell/Command Prompt and retry to install the library with the original pip instruction
If storage/connectivity speed is critical for your usecase and/or you would like to know more about the issue have a look at this Stack Overflow discussion.
Basic usage
Import the library:
import word_embedding.model as model
Initialise a Word Embedding object
You can pass the path where the model is stored:
model = model.WordEmbedding(model_fn="/path/where/my/model/is/stored.txt")
or you can pass the model itself, previously loaded (assuming your model is a dictionary, whose keys are the various words and its values the vector representation of the various words):
model = model.WordEmbedding(model=my_word_embedding_model)
Compute Word Mover's distance
s1 = 'Obama speaks to the media in Chicago'.lower().split()
s2 = 'The president spoke to the press in Chicago'.lower().split()
wmdistance = model.wmdistance(s1, s2)
1.8119693993679309
Remember that the wmdistance(s1, s2)
method expects two List[str]
as input!
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for word_mover_distance-0.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 34b2000686408a5671c3e45706259bf6653de41befac91029213d4a6e58fb96d |
|
MD5 | d3736f6ee79b42fa98f52fac9b7ca371 |
|
BLAKE2b-256 | 78a1866364df51adbc01e8e0a7b1e75cd8ee471e5c72e1a6bec0a417869a9335 |
Hashes for word_mover_distance-0.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 34aa21a2bdf5cecb5f0e56c0b3608865894bc4128c9138a4b4cc0dfb814adec5 |
|
MD5 | 412022a69e0b88dcc471844c02a95cb9 |
|
BLAKE2b-256 | 9130f603e785b568f13304d4bcdb804a8c95227ea00305d055f09a61fe603f99 |