Skip to main content

A package for training and evaluating multimodal knowledge graph embeddings

Project description


GitHub Actions License DOI Optuna integrated

PyKEEN (Python KnowlEdge EmbeddiNgs) is a Python package designed to train and evaluate knowledge graph embedding models (incorporating multi-modal information).


Installation PyPI - Python Version PyPI

The latest stable version of PyKEEN can be downloaded and installed from PyPI with:

$ pip install pykeen

The latest version of PyKEEN can be installed directly from the source on GitHub with:

$ pip install git+

More information about installation (e.g., development mode, Windows installation, Colab, Kaggle, extras) can be found in the installation documentation.

Quickstart Documentation Status

This example shows how to train a model on a dataset and test on another dataset.

The fastest way to get up and running is to use the pipeline function. It provides a high-level entry into the extensible functionality of this package. The following example shows how to train and evaluate the TransE model on the Nations dataset. By default, the training loop uses the stochastic local closed world assumption (sLCWA) training approach and evaluates with rank-based evaluation.

from pykeen.pipeline import pipeline

result = pipeline(

The results are returned in an instance of the PipelineResult dataclass that has attributes for the trained model, the training loop, the evaluation, and more. See the tutorials on understanding the evaluation and making novel link predictions.

PyKEEN is extensible such that:

  • Each model has the same API, so anything from pykeen.models can be dropped in
  • Each training loop has the same API, so can be dropped in
  • Triples factories can be generated by the user with from pykeen.triples.TriplesFactory

The full documentation can be found at


Below are the models, datasets, training modes, evaluators, and metrics implemented in pykeen.

Datasets (26)

The citation for each dataset corresponds to either the paper describing the dataset, the first paper published using the dataset with knowledge graph embedding models, or the URL for the dataset if neither of the first two are available.

Name Documentation Citation Entities Relations Triples
Clinical Knowledge Graph pykeen.datasets.CKG Santos et al., 2020 7617419 11 26691525
CN3l Family pykeen.datasets.CN3l Chen et al., 2017 3206 42 21777
CoDEx (large) pykeen.datasets.CoDExLarge Safavi et al., 2020 77951 69 612437
CoDEx (medium) pykeen.datasets.CoDExMedium Safavi et al., 2020 17050 51 206205
CoDEx (small) pykeen.datasets.CoDExSmall Safavi et al., 2020 2034 42 36543
ConceptNet pykeen.datasets.ConceptNet Speer et al., 2017 28370083 50 34074917
Countries pykeen.datasets.Countries Bouchard et al., 2015 271 2 1158
Commonsense Knowledge Graph pykeen.datasets.CSKG Ilievski et al., 2020 2087833 58 4598728
DB100K pykeen.datasets.DB100K Ding et al., 2018 99604 470 697479
DBpedia50 pykeen.datasets.DBpedia50 Shi et al., 2017 24624 351 34421
Drug Repositioning Knowledge Graph pykeen.datasets.DRKG gnn4dr/DRKG 97238 107 5874257
FB15k pykeen.datasets.FB15k Bordes et al., 2013 14951 1345 592213
FB15k-237 pykeen.datasets.FB15k237 Toutanova et al., 2015 14505 237 310079
Hetionet pykeen.datasets.Hetionet Himmelstein et al., 2017 45158 24 2250197
Kinships pykeen.datasets.Kinships Kemp et al., 2006 104 25 10686
Nations pykeen.datasets.Nations ZhenfengLei/KGDatasets 14 55 1992
OGB BioKG pykeen.datasets.OGBBioKG Hu et al., 2020 45085 51 5088433
OGB WikiKG pykeen.datasets.OGBWikiKG Hu et al., 2020 2500604 535 17137181
OpenBioLink pykeen.datasets.OpenBioLink Breit et al., 2020 180992 28 4563407
OpenBioLink pykeen.datasets.OpenBioLinkLQ Breit et al., 2020 480876 32 27320889
Unified Medical Language System pykeen.datasets.UMLS ZhenfengLei/KGDatasets 135 46 6529
WK3l-120k Family pykeen.datasets.WK3l120k Chen et al., 2017 119748 3109 1375406
WK3l-15k Family pykeen.datasets.WK3l15k Chen et al., 2017 15126 1841 209041
WordNet-18 pykeen.datasets.WN18 Bordes et al., 2014 40943 18 151442
WordNet-18 (RR) pykeen.datasets.WN18RR Toutanova et al., 2015 40559 11 92583
YAGO3-10 pykeen.datasets.YAGO310 Mahdisoltani et al., 2015 123143 37 1089000

Models (28)

Name Reference Citation
CompGCN pykeen.models.CompGCN Vashishth et al., 2020
ComplEx pykeen.models.ComplEx Trouillon et al., 2016
ComplExLiteral pykeen.models.ComplExLiteral Kristiadi et al., 2018
ConvE pykeen.models.ConvE Dettmers et al., 2018
ConvKB pykeen.models.ConvKB Nguyen et al., 2018
CrossE pykeen.models.CrossE Zhang et al., 2019
DistMult pykeen.models.DistMult Yang et al., 2014
DistMultLiteral pykeen.models.DistMultLiteral Kristiadi et al., 2018
ERMLP pykeen.models.ERMLP Dong et al., 2014
ERMLPE pykeen.models.ERMLPE Sharifzadeh et al., 2019
HolE pykeen.models.HolE Nickel et al., 2016
KG2E pykeen.models.KG2E He et al., 2015
MuRE pykeen.models.MuRE Balažević et al., 2019
NTN pykeen.models.NTN Socher et al., 2013
PairRE pykeen.models.PairRE Chao et al., 2020
ProjE pykeen.models.ProjE Shi et al., 2017
QuatE pykeen.models.QuatE Zhang et al., 2019
RESCAL pykeen.models.RESCAL Nickel et al., 2011
RGCN pykeen.models.RGCN Schlichtkrull et al., 2018
RotatE pykeen.models.RotatE Sun et al., 2019
SimplE pykeen.models.SimplE Kazemi et al., 2018
StructuredEmbedding pykeen.models.StructuredEmbedding Bordes et al., 2011
TransD pykeen.models.TransD Ji et al., 2015
TransE pykeen.models.TransE Bordes et al., 2013
TransH pykeen.models.TransH Wang et al., 2014
TransR pykeen.models.TransR Lin et al., 2015
TuckER pykeen.models.TuckER Balažević et al., 2019
UnstructuredModel pykeen.models.UnstructuredModel Bordes et al., 2014

Losses (7)

Name Reference Description
bceaftersigmoid pykeen.losses.BCEAfterSigmoidLoss A module for the numerically unstable version of explicit Sigmoid + BCE loss.
bcewithlogits pykeen.losses.BCEWithLogitsLoss A module for the binary cross entropy loss.
crossentropy pykeen.losses.CrossEntropyLoss A module for the cross entropy loss that evaluates the cross entropy after softmax output.
marginranking pykeen.losses.MarginRankingLoss A module for the margin ranking loss.
mse pykeen.losses.MSELoss A module for the mean square error loss.
nssa pykeen.losses.NSSALoss An implementation of the self-adversarial negative sampling loss function proposed by [sun2019]_.
softplus pykeen.losses.SoftplusLoss A module for the softplus loss.

Regularizers (5)

Name Reference Description
combined pykeen.regularizers.CombinedRegularizer A convex combination of regularizers.
lp pykeen.regularizers.LpRegularizer A simple L_p norm based regularizer.
no pykeen.regularizers.NoRegularizer A regularizer which does not perform any regularization.
powersum pykeen.regularizers.PowerSumRegularizer A simple x^p based regularizer.
transh pykeen.regularizers.TransHRegularizer A regularizer for the soft constraints in TransH.

Optimizers (6)

Name Reference Description
adadelta torch.optim.Adadelta Implements Adadelta algorithm.
adagrad torch.optim.Adagrad Implements Adagrad algorithm.
adam torch.optim.Adam Implements Adam algorithm.
adamax torch.optim.Adamax Implements Adamax algorithm (a variant of Adam based on infinity norm).
adamw torch.optim.AdamW Implements AdamW algorithm.
sgd torch.optim.SGD Implements stochastic gradient descent (optionally with momentum).

Training Loops (2)

Name Reference Description
lcwa A training loop that uses the local closed world assumption training approach.
slcwa A training loop that uses the stochastic local closed world assumption training approach.

Negative Samplers (3)

Name Reference Description
basic pykeen.sampling.BasicNegativeSampler A basic negative sampler.
bernoulli pykeen.sampling.BernoulliNegativeSampler An implementation of the Bernoulli negative sampling approach proposed by [wang2014]_.
pseudotyped pykeen.sampling.PseudoTypedNegativeSampler A sampler that accounts for which entities co-occur with a relation.

Stoppers (2)

Name Reference Description
early pykeen.stoppers.EarlyStopper A harness for early stopping.
nop pykeen.stoppers.NopStopper A stopper that does nothing.

Evaluators (2)

Name Reference Description
rankbased pykeen.evaluation.RankBasedEvaluator A rank-based evaluator for KGE models.
sklearn pykeen.evaluation.SklearnEvaluator An evaluator that uses a Scikit-learn metric.

Metrics (16)

Name Description
AUC-ROC The area under the ROC curve, on [0, 1]. Higher is better.
Adjusted Arithmetic Mean Rank (AAMR) The mean over all chance-adjusted ranks, on (0, 2). Lower is better.
Adjusted Arithmetic Mean Rank Index (AAMRI) The re-indexed adjusted mean rank (AAMR), on [-1, 1]. Higher is better.
Average Precision The area under the precision-recall curve, on [0, 1]. Higher is better.
Geometric Mean Rank (GMR) The geometric mean over all ranks, on [1, inf). Lower is better.
Harmonic Mean Rank (HMR) The harmonic mean over all ranks, on [1, inf). Lower is better.
Hits @ K The relative frequency of ranks not larger than a given k, on [0, 1]. Higher is better
Inverse Arithmetic Mean Rank (IAMR) The inverse of the arithmetic mean over all ranks, on (0, 1]. Higher is better.
Inverse Geometric Mean Rank (IGMR) The inverse of the geometric mean over all ranks, on (0, 1]. Higher is better.
Inverse Median Rank The inverse of the median over all ranks, on (0, 1]. Higher is better.
Mean Rank (MR) The arithmetic mean over all ranks on, [1, inf). Lower is better.
Mean Reciprocal Rank (MRR) The inverse of the harmonic mean over all ranks, on (0, 1]. Higher is better.
Median Rank The median over all ranks, on [1, inf). Lower is better.

Trackers (7)

Name Reference Description
console pykeen.trackers.ConsoleResultTracker A class that directly prints to console.
csv pykeen.trackers.CSVResultTracker Tracking results to a CSV file.
json pykeen.trackers.JSONResultTracker Tracking results to a JSON lines file.
mlflow pykeen.trackers.MLFlowResultTracker A tracker for MLflow.
neptune pykeen.trackers.NeptuneResultTracker A tracker for
tensorboard pykeen.trackers.TensorBoardResultTracker A tracker for TensorBoard.
wandb pykeen.trackers.WANDBResultTracker A tracker for Weights and Biases.

Hyper-parameter Optimization

Samplers (3)

Name Reference Description
grid optuna.samplers.GridSampler Sampler using grid search.
random optuna.samplers.RandomSampler Sampler using random sampling.
tpe optuna.samplers.TPESampler Sampler using TPE (Tree-structured Parzen Estimator) algorithm.

Any sampler class extending the optuna.samplers.BaseSampler, such as their sampler implementing the CMA-ES algorithm, can also be used.



PyKEEN includes a set of curated experimental settings for reproducing past landmark experiments. They can be accessed and run like:

$ pykeen experiments reproduce tucker balazevic2019 fb15k

Where the three arguments are the model name, the reference, and the dataset. The output directory can be optionally set with -d.


PyKEEN includes the ability to specify ablation studies using the hyper-parameter optimization module. They can be run like:

$ pykeen experiments ablation ~/path/to/config.json

Large-scale Reproducibility and Benchmarking Study

We used PyKEEN to perform a large-scale reproducibility and benchmarking study which are described in our article:

  title={Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework},
  author={Ali, Mehdi and Berrendorf, Max and Hoyt, Charles Tapley and Vermue, Laurent and Galkin, Mikhail and Sharifzadeh, Sahand and Fischer, Asja and Tresp, Volker and Lehmann, Jens},
  journal={arXiv preprint arXiv:2006.13365},

We have made all code, experimental configurations, results, and analyses that lead to our interpretations available at


Contributions, whether filing an issue, making a pull request, or forking, are appreciated. See for more information on getting involved.



This project has been supported by several organizations (in alphabetical order):


The development of PyKEEN has been funded by the following grants:

Funding Body Program Grant
DARPA Automating Scientific Knowledge Extraction (ASKE) HR00111990009
German Federal Ministry of Education and Research (BMBF) Maschinelles Lernen mit Wissensgraphen (MLWin) 01IS18050D
German Federal Ministry of Education and Research (BMBF) Munich Center for Machine Learning (MCML) 01IS18036A
Innovation Fund Denmark (Innovationsfonden) Danish Center for Big Data Analytics driven Innovation (DABAI) Grand Solutions


The PyKEEN logo was designed by Carina Steinborn


If you have found PyKEEN useful in your work, please consider citing our article:

    author = {Ali, Mehdi and Berrendorf, Max and Hoyt, Charles Tapley and Vermue, Laurent and Sharifzadeh, Sahand and Tresp, Volker and Lehmann, Jens},
    journal = {Journal of Machine Learning Research},
    number = {82},
    pages = {1--6},
    title = {{PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings}},
    url = {},
    volume = {22},
    year = {2021}

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for pykeen, version 1.5.0
Filename, size File type Python version Upload date Hashes
Filename, size pykeen-1.5.0-py3-none-any.whl (465.4 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size pykeen-1.5.0.tar.gz (1.5 MB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page