vec2best 1.1.0

A Unified Framework for Intrinsic Evaluation of Word-Embedding Algorithms

vec2best: A Unified Framework for Intrinsic Evaluation of Word-Embedding Algorithms

Description • Requirements • Installation • Usage

---

Description

vec2best is a library for Python which represents a framework for evaluating word embeddings trained using various methods and hyper-parameters on a range of tasks from the literature. The tool yields a holistic evaluation metric for each model called the $PCE$ (Principal Component Evaluation).

vec2best implements the state-of-the-art intrinsic evaluations tasks of word similarity, word analogy, concept categorisation, and outlier detection over the benchmarks in the following table.

Task	Evaluation	Metric	Benchmark
Similarity	Spearman correlation	Cosine similarity	WS353, RG65, RW, MEN, MTurk287, SimLex999, MC30, MTurk771, YP130, Verb143, SimVerb3500, SemEval17, WS353REL, WS353SIM
Analogy	Accuracy	3CosAdd, 3CosMul	Google, MSR
	Spearman correlation	3CosAdd	SemEval2012
Categorization	Purity	Clustering	AP, BLESS, BM (battig), ESSLI 1a, ESSLI 2b, ESSLI 2c
Outlier detection	Accuracy	Compactness score	8-8-8, WordSim500

Requirements

• Python 3.6
• scikit-learn
• six
• word-embeddings-benchmarks

The package also relies on a modified version on the following repositories for outlier detection:

• https://github.com/peblair/wiki-sem-500
• http://lcl.uniroma1.it/outlier-detection/

Installation

vec2best can be installed through pip (the Python package manager) in the following way:

pip install vec2best

Usage

To compute the $PCE$ you need to apply the function compute_pce(path_to_model) and the only parameter that you need to set is the path in which you saved the embedding models (in a .vec or .txt format) you want to evaluate.

The function compute_pce(path_to_model) has other six parameters (categorization=True, similarity=True}, analogy=True, outlier_detection=True, pce_min=True, pce_max=True, pce_mean=True) set by default as True, and so the output consists in the evaluation of the models over the three tasks and over the $PCE^{MIN}$, $PCE^{MAX}$, $PCE^{MEAN}$. By setting some of those parameters as False, the $PCE$ can be computed over a subset of those tasks or the evaluation could be computed only for one or two of the three types of $PCE$.

The output is saved in the folder results/pce, and the output on the screen shows the percentage of explained variance of the first principal component, and the top 3 models according to the chosen $PCE$.

See the following example:

from vec2best import compute_pce
path_to_model = 'data/example_models' 
compute_pce(path_to_model, analogy=False,outlier_detection=False, 
pce_max=False, pce_mean=False)

The output will look like:

PCE min - percentage of explained variance: 0.95
                                 categorization    similarity    PCE_min
example_models/ft_0_5_50_5.vec   0.38              0.29          1.00
example_models/glove_5_50_5.vec  0.41              0.25          0.94
example_models/wv2_model_11.vec  0.24              0.17          0.34