Embedding Evaluator

EmbeddingEvaluator is a tool to provide metrics for evaluating different embedding models.

The current version only supports evaluation of evaluate only embeddings of the type:

• FastText

It evaluates the embeddings based on the two following metrics:

• Analogy
• Outlier Detection

Installation

The EmbeddingEvaluator can be installed from PyPi:

pip install embeddingevaluator

Usage

Analogy Metrics

To use the EmbeddingEvaluator to measure different embeddings basead on analogy metrics the user needs a file with the following configuration:

Ouliter Detection

To use the EmbeddingEvaluator to measure different embeddings basead on outlier detection metrics the user needs a file with the following configuration:

• Eight words which are semantically very similar and are all connected with each other by a clear well-known relation. (Cluster)
• Two words which are very similar to the ones in the cluster.
• Two words which are similar and related to the ones in the cluster.
• Two words which are related, but not similar to the ones in the cluster.
• Two words which are unrelated and not similar to the ones in the cluster.

Initialize the EmbeddingEvaluator

The EmbeddingEvaluator has three parameters as input:

• Input Metrics:
  A dictionary with a list of the paths for the input evaluation files.

Example:

input_metric = {'analogy': ['file_1', 'file_2'],
                'outlier': ['file_1']}

• Input Models: A dictionary with the model names and the paths to the models.

Example:

input_model = {'model_1': 'path_1', 
               'model_2': 'path_2'}

Initialize the class:

emb_evaluator = EmbeddingMetrics(input_metric, input_model)

Summary a model's metrics

To summarize the metrics of a model.

emb_evaluator.summary_metrics('model_1') 

Compare models metrics

To compare the metrics of two or more models.

emb_evaluator.compare_models(['model_1', 'model_2']) 

References

Levy, O. and Goldberg, Y.: Linguistic Regularities in Sparse and Explicit Word Representations (2014) Collados, J.C. and Navigli, R.: Find the word that does not belong: A Framework for an Intrinsic Evaluation of Word Vector Representations (2016)