pykeen 1.8.1

A package for training and evaluating multimodal knowledge graph embeddings

PyKEEN

PyKEEN (Python KnowlEdge EmbeddiNgs) is a Python package designed to train and evaluate knowledge graph embedding models (incorporating multi-modal information).

Installation • Quickstart • Datasets • Inductive Datasets (5) • Models • Support • Citation

Installation

The latest stable version of PyKEEN can be downloaded and installed from PyPI with:

$ pip install pykeen

The latest version of PyKEEN can be installed directly from the source on GitHub with:

$ pip install git+https://github.com/pykeen/pykeen.git

More information about installation (e.g., development mode, Windows installation, Colab, Kaggle, extras) can be found in the installation documentation.

Quickstart

This example shows how to train a model on a dataset and test on another dataset.

The fastest way to get up and running is to use the pipeline function. It provides a high-level entry into the extensible functionality of this package. The following example shows how to train and evaluate the TransE model on the Nations dataset. By default, the training loop uses the stochastic local closed world assumption (sLCWA) training approach and evaluates with rank-based evaluation.

from pykeen.pipeline import pipeline

result = pipeline(
    model='TransE',
    dataset='nations',
)

The results are returned in an instance of the PipelineResult dataclass that has attributes for the trained model, the training loop, the evaluation, and more. See the tutorials on using your own dataset, understanding the evaluation, and making novel link predictions.

PyKEEN is extensible such that:

• Each model has the same API, so anything from pykeen.models can be dropped in
• Each training loop has the same API, so pykeen.training.LCWATrainingLoop can be dropped in
• Triples factories can be generated by the user with from pykeen.triples.TriplesFactory

The full documentation can be found at https://pykeen.readthedocs.io.

Implementation

Below are the models, datasets, training modes, evaluators, and metrics implemented in pykeen.

Datasets (34)

The following datasets are built in to PyKEEN. The citation for each dataset corresponds to either the paper describing the dataset, the first paper published using the dataset with knowledge graph embedding models, or the URL for the dataset if neither of the first two are available. If you want to use a custom dataset, see the Bring Your Own Dataset tutorial. If you have a suggestion for another dataset to include in PyKEEN, please let us know here.

Name	Documentation	Citation	Entities	Relations	Triples
Aristo-v4	pykeen.datasets.AristoV4	Chen et al., 2021	42016	1593	279425
BioKG	pykeen.datasets.BioKG	Walsh et al., 2019	105524	17	2067997
Clinical Knowledge Graph	pykeen.datasets.CKG	Santos et al., 2020	7617419	11	26691525
CN3l Family	pykeen.datasets.CN3l	Chen et al., 2017	3206	42	21777
CoDEx (large)	pykeen.datasets.CoDExLarge	Safavi et al., 2020	77951	69	612437
CoDEx (medium)	pykeen.datasets.CoDExMedium	Safavi et al., 2020	17050	51	206205
CoDEx (small)	pykeen.datasets.CoDExSmall	Safavi et al., 2020	2034	42	36543
ConceptNet	pykeen.datasets.ConceptNet	Speer et al., 2017	28370083	50	34074917
Countries	pykeen.datasets.Countries	Bouchard et al., 2015	271	2	1158
Commonsense Knowledge Graph	pykeen.datasets.CSKG	Ilievski et al., 2020	2087833	58	4598728
DB100K	pykeen.datasets.DB100K	Ding et al., 2018	99604	470	697479
DBpedia50	pykeen.datasets.DBpedia50	Shi et al., 2017	24624	351	34421
Drug Repositioning Knowledge Graph	pykeen.datasets.DRKG	gnn4dr/DRKG	97238	107	5874257
FB15k	pykeen.datasets.FB15k	Bordes et al., 2013	14951	1345	592213
FB15k-237	pykeen.datasets.FB15k237	Toutanova et al., 2015	14505	237	310079
Hetionet	pykeen.datasets.Hetionet	Himmelstein et al., 2017	45158	24	2250197
Kinships	pykeen.datasets.Kinships	Kemp et al., 2006	104	25	10686
Nations	pykeen.datasets.Nations	ZhenfengLei/KGDatasets	14	55	1992
NationsL	pykeen.datasets.NationsLiteral	pykeen/pykeen	14	55	1992
OGB BioKG	pykeen.datasets.OGBBioKG	Hu et al., 2020	45085	51	5088433
OGB WikiKG2	pykeen.datasets.OGBWikiKG2	Hu et al., 2020	2500604	535	17137181
OpenBioLink	pykeen.datasets.OpenBioLink	Breit et al., 2020	180992	28	4563407
OpenBioLink LQ	pykeen.datasets.OpenBioLinkLQ	Breit et al., 2020	480876	32	27320889
OpenEA Family	pykeen.datasets.OpenEA	Sun et al., 2020	15000	248	38265
PharmKG	pykeen.datasets.PharmKG	Zheng et al., 2020	188296	39	1093236
PharmKG8k	pykeen.datasets.PharmKG8k	Zheng et al., 2020	7247	28	485787
Unified Medical Language System	pykeen.datasets.UMLS	ZhenfengLei/KGDatasets	135	46	6529
WD50K (triples)	pykeen.datasets.WD50KT	Galkin et al., 2020	40107	473	232344
Wikidata5M	pykeen.datasets.Wikidata5M	Wang et al., 2019	4594149	822	20624239
WK3l-120k Family	pykeen.datasets.WK3l120k	Chen et al., 2017	119748	3109	1375406
WK3l-15k Family	pykeen.datasets.WK3l15k	Chen et al., 2017	15126	1841	209041
WordNet-18	pykeen.datasets.WN18	Bordes et al., 2014	40943	18	151442
WordNet-18 (RR)	pykeen.datasets.WN18RR	Toutanova et al., 2015	40559	11	92583
YAGO3-10	pykeen.datasets.YAGO310	Mahdisoltani et al., 2015	123143	37	1089000

Inductive Datasets

The following inductive datasets are built in to PyKEEN.

Name	Documentation	Citation
ILPC2022 Large	pykeen.datasets.ILPC2022Large	Galkin et al., 2022
ILPC2022 Small	pykeen.datasets.ILPC2022Small	Galkin et al., 2022
FB15k-237	pykeen.datasets.InductiveFB15k237	Teru et al., 2020
NELL	pykeen.datasets.InductiveNELL	Teru et al., 2020
WordNet-18 (RR)	pykeen.datasets.InductiveWN18RR	Teru et al., 2020

Models (42)

Name	Model	Interaction	Citation
AutoSF	pykeen.models.AutoSF	pykeen.nn.AutoSFInteraction	Zhang et al., 2020
BoxE	pykeen.models.BoxE	pykeen.nn.BoxEInteraction	Abboud et al., 2020
Canonical Tensor Decomposition	pykeen.models.CP	pykeen.nn.CPInteraction	Lacroix et al., 2018
CompGCN	pykeen.models.CompGCN		Vashishth et al., 2020
ComplEx	pykeen.models.ComplEx	pykeen.nn.ComplExInteraction	Trouillon et al., 2016
ComplEx Literal	pykeen.models.ComplExLiteral	pykeen.nn.LiteralInteraction	Kristiadi et al., 2018
ConvE	pykeen.models.ConvE	pykeen.nn.ConvEInteraction	Dettmers et al., 2018
ConvKB	pykeen.models.ConvKB	pykeen.nn.ConvKBInteraction	Nguyen et al., 2018
CrossE	pykeen.models.CrossE	pykeen.nn.CrossEInteraction	Zhang et al., 2019
DistMA	pykeen.models.DistMA	pykeen.nn.DistMAInteraction	Shi et al., 2019
DistMult	pykeen.models.DistMult	pykeen.nn.DistMultInteraction	Yang et al., 2014
DistMult Literal	pykeen.models.DistMultLiteral	pykeen.nn.LiteralInteraction	Kristiadi et al., 2018
DistMult Literal (Gated)	pykeen.models.DistMultLiteralGated	pykeen.nn.LiteralInteraction	Kristiadi et al., 2018
ER-MLP	pykeen.models.ERMLP	pykeen.nn.ERMLPInteraction	Dong et al., 2014
ER-MLP (E)	pykeen.models.ERMLPE	pykeen.nn.ERMLPEInteraction	Sharifzadeh et al., 2019
Fixed Model	pykeen.models.FixedModel		Berrendorf et al., 2021
HolE	pykeen.models.HolE	pykeen.nn.HolEInteraction	Nickel et al., 2016
InductiveNodePiece	pykeen.models.InductiveNodePiece		Galkin et al., 2021
InductiveNodePieceGNN	pykeen.models.InductiveNodePieceGNN		Galkin et al., 2021
KG2E	pykeen.models.KG2E	pykeen.nn.KG2EInteraction	He et al., 2015
MuRE	pykeen.models.MuRE	pykeen.nn.MuREInteraction	Balažević et al., 2019
MultiLinearTucker		pykeen.nn.MultiLinearTuckerInteraction	Tucker et al., 1966
NTN	pykeen.models.NTN	pykeen.nn.NTNInteraction	Socher et al., 2013
NodePiece	pykeen.models.NodePiece		Galkin et al., 2021
PairRE	pykeen.models.PairRE	pykeen.nn.PairREInteraction	Chao et al., 2020
ProjE	pykeen.models.ProjE	pykeen.nn.ProjEInteraction	Shi et al., 2017
QuatE	pykeen.models.QuatE	pykeen.nn.QuatEInteraction	Zhang et al., 2019
R-GCN	pykeen.models.RGCN		Schlichtkrull et al., 2018
RESCAL	pykeen.models.RESCAL	pykeen.nn.RESCALInteraction	Nickel et al., 2011
RotatE	pykeen.models.RotatE	pykeen.nn.RotatEInteraction	Sun et al., 2019
SimplE	pykeen.models.SimplE	pykeen.nn.SimplEInteraction	Kazemi et al., 2018
Structured Embedding	pykeen.models.SE	pykeen.nn.SEInteraction	Bordes et al., 2011
TorusE	pykeen.models.TorusE	pykeen.nn.TorusEInteraction	Ebisu et al., 2018
TransD	pykeen.models.TransD	pykeen.nn.TransDInteraction	Ji et al., 2015
TransE	pykeen.models.TransE	pykeen.nn.TransEInteraction	Bordes et al., 2013
TransF	pykeen.models.TransF	pykeen.nn.TransFInteraction	Feng et al., 2016
TransH	pykeen.models.TransH	pykeen.nn.TransHInteraction	Wang et al., 2014
TransR	pykeen.models.TransR	pykeen.nn.TransRInteraction	Lin et al., 2015
Transformer		pykeen.nn.TransformerInteraction	Galkin et al., 2020
TripleRE		pykeen.nn.TripleREInteraction	Yu et al., 2021
TuckER	pykeen.models.TuckER	pykeen.nn.TuckerInteraction	Balažević et al., 2019
Unstructured Model	pykeen.models.UM	pykeen.nn.UMInteraction	Bordes et al., 2014

Losses (13)

Name	Reference	Description
Binary cross entropy (after sigmoid)	pykeen.losses.BCEAfterSigmoidLoss	A module for the numerically unstable version of explicit Sigmoid + BCE loss.
Binary cross entropy (with logits)	pykeen.losses.BCEWithLogitsLoss	A module for the binary cross entropy loss.
Cross entropy	pykeen.losses.CrossEntropyLoss	A module for the cross entropy loss that evaluates the cross entropy after softmax output.
Double Margin	pykeen.losses.DoubleMarginLoss	A limit-based scoring loss, with separate margins for positive and negative elements from [sun2018]_.
Focal	pykeen.losses.FocalLoss	A module for the focal loss proposed by [lin2018]_.
Margin ranking	pykeen.losses.MarginRankingLoss	A module for the pairwise hinge loss (i.e., margin ranking loss).
Mean square error	pykeen.losses.MSELoss	A module for the mean square error loss.
Self-adversarial negative sampling	pykeen.losses.NSSALoss	An implementation of the self-adversarial negative sampling loss function proposed by [sun2019]_.
Pairwise logistic	pykeen.losses.PairwiseLogisticLoss	The pairwise logistic loss.
Pointwise Hinge	pykeen.losses.PointwiseHingeLoss	A module for the pointwise hinge loss.
Soft margin ranking	pykeen.losses.SoftMarginRankingLoss	A module for the soft pairwise hinge loss (i.e., soft margin ranking loss).
Softplus	pykeen.losses.SoftplusLoss	A module for the pointwise logistic loss (i.e., softplus loss).
Soft Pointwise Hinge	pykeen.losses.SoftPointwiseHingeLoss	A module for the soft pointwise hinge loss .

Regularizers (5)

Name	Reference	Description
combined	pykeen.regularizers.CombinedRegularizer	A convex combination of regularizers.
lp	pykeen.regularizers.LpRegularizer	A simple L_p norm based regularizer.
no	pykeen.regularizers.NoRegularizer	A regularizer which does not perform any regularization.
powersum	pykeen.regularizers.PowerSumRegularizer	A simple x^p based regularizer.
transh	pykeen.regularizers.TransHRegularizer	A regularizer for the soft constraints in TransH.

Training Loops (2)

Name	Reference	Description
lcwa	pykeen.training.LCWATrainingLoop	A training loop that is based upon the local closed world assumption (LCWA).
slcwa	pykeen.training.SLCWATrainingLoop	A training loop that uses the stochastic local closed world assumption training approach.

Negative Samplers (3)

Name	Reference	Description
basic	pykeen.sampling.BasicNegativeSampler	A basic negative sampler.
bernoulli	pykeen.sampling.BernoulliNegativeSampler	An implementation of the Bernoulli negative sampling approach proposed by [wang2014]_.
pseudotyped	pykeen.sampling.PseudoTypedNegativeSampler	A sampler that accounts for which entities co-occur with a relation.

Stoppers (2)

Name	Reference	Description
early	pykeen.stoppers.EarlyStopper	A harness for early stopping.
nop	pykeen.stoppers.NopStopper	A stopper that does nothing.

Evaluators (4)

Name	Reference	Description
classification	pykeen.evaluation.ClassificationEvaluator	An evaluator that uses a classification metrics.
macrorankbased	pykeen.evaluation.MacroRankBasedEvaluator	Macro-average rank-based evaluation.
rankbased	pykeen.evaluation.RankBasedEvaluator	A rank-based evaluator for KGE models.
sampledrankbased	pykeen.evaluation.SampledRankBasedEvaluator	A rank-based evaluator using sampled negatives instead of all negatives.

Metrics (44)

Name	Interval	Direction	Description	Type
AUC-ROC	[0, 1]	📈	Area Under the ROC Curve	Classification
Accuracy	[0, 1]	📈	(TP + TN) / (TP + TN + FP + FN)	Classification
Average Precision	[0, 1]	📈	A summary statistic over the precision-recall curve	Classification
Balanced Accuracy	[0, 1]	📈	An adjusted version of the accuracy for imbalanced datasets	Classification
Diagnostic Odds Ratio	[0, ∞)	📈	LR+/LR-	Classification
F1 Score	[0, 1]	📈	2TP / (2TP + FP + FN)	Classification
False Discovery Rate	[0, 1]	📉	FP / (FP + TP)	Classification
False Negative Rate	[0, 1]	📉	FN / (FN + TP)	Classification
False Omission Rate	[0, 1]	📉	FN / (FN + TN)	Classification
False Positive Rate	[0, 1]	📉	FP / (FP + TN)	Classification
Fowlkes Mallows Index	[0, 1]	📈	√PPV x √TPR	Classification
Informedness	[0, 1]	📈	TPR + TNR - 1	Classification
Markedness	[0, 1]	📈	PPV + NPV - 1	Classification
Matthews Correlation Coefficient	[-1, 1]	📈	A balanced measure applicable even with class imbalance	Classification
Negative Likelihood Ratio	[0, ∞)	📉	FNR / TNR	Classification
Negative Predictive Value	[0, 1]	📈	TN / (TN + FN)	Classification
Positive Likelihood Ratio	[0, ∞)	📈	TPR / FPR	Classification
Positive Predictive Value	[0, 1]	📈	TP / (TP + FP)	Classification
Prevalence Threshold	[0, 1]	📉	√FPR / (√TPR + √FPR)	Classification
Threat Score	[0, 1]	📈	TP / (TP + FN + FP)	Classification
True Negative Rate	[0, 1]	📈	TN / (TN + FP)	Classification
True Positive Rate	[0, 1]	📈	TP / (TP + FN)	Classification
Adjusted Arithmetic Mean Rank (AAMR)	[0, 2)	📉	The mean over all ranks divided by its expected value.	Ranking
Adjusted Arithmetic Mean Rank Index (AAMRI)	[-1, 1]	📈	The re-indexed adjusted mean rank (AAMR)	Ranking
Adjusted Geometric Mean Rank Index (AGMRI)	(-E[f]/(1-E[f]), 1]	📈	The re-indexed adjusted geometric mean rank (AGMRI)	Ranking
Adjusted Hits at K	(-E[f]/(1-E[f]), 1]	📈	The re-indexed adjusted hits at K	Ranking
Adjusted Inverse Harmonic Mean Rank	(-E[f]/(1-E[f]), 1]	📈	The re-indexed adjusted MRR	Ranking
Geometric Mean Rank (GMR)	[1, ∞)	📉	The geometric mean over all ranks.	Ranking
Harmonic Mean Rank (HMR)	[1, ∞)	📉	The harmonic mean over all ranks.	Ranking
Hits @ K	[0, 1]	📈	The relative frequency of ranks not larger than a given k.	Ranking
Inverse Arithmetic Mean Rank (IAMR)	(0, 1]	📈	The inverse of the arithmetic mean over all ranks.	Ranking
Inverse Geometric Mean Rank (IGMR)	(0, 1]	📈	The inverse of the geometric mean over all ranks.	Ranking
Inverse Median Rank	(0, 1]	📈	The inverse of the median over all ranks.	Ranking
Mean Rank (MR)	[1, ∞)	📉	The arithmetic mean over all ranks.	Ranking
Mean Reciprocal Rank (MRR)	(0, 1]	📈	The inverse of the harmonic mean over all ranks.	Ranking
Median Rank	[1, ∞)	📉	The median over all ranks.	Ranking
z-Geometric Mean Rank (zGMR)	(-∞, ∞)	📈	The z-scored geometric mean rank	Ranking
z-Hits at K	(-∞, ∞)	📈	The z-scored hits at K	Ranking
z-Mean Rank (zMR)	(-∞, ∞)	📈	The z-scored mean rank	Ranking
z-Mean Reciprocal Rank (zMRR)	(-∞, ∞)	📈	The z-scored mean reciprocal rank	Ranking

Trackers (8)

Name	Reference	Description
console	pykeen.trackers.ConsoleResultTracker	A class that directly prints to console.
csv	pykeen.trackers.CSVResultTracker	Tracking results to a CSV file.
json	pykeen.trackers.JSONResultTracker	Tracking results to a JSON lines file.
mlflow	pykeen.trackers.MLFlowResultTracker	A tracker for MLflow.
neptune	pykeen.trackers.NeptuneResultTracker	A tracker for Neptune.ai.
python	pykeen.trackers.PythonResultTracker	A tracker which stores everything in Python dictionaries.
tensorboard	pykeen.trackers.TensorBoardResultTracker	A tracker for TensorBoard.
wandb	pykeen.trackers.WANDBResultTracker	A tracker for Weights and Biases.

Experimentation

Reproduction

PyKEEN includes a set of curated experimental settings for reproducing past landmark experiments. They can be accessed and run like:

$ pykeen experiments reproduce tucker balazevic2019 fb15k

Where the three arguments are the model name, the reference, and the dataset. The output directory can be optionally set with -d.

Ablation

PyKEEN includes the ability to specify ablation studies using the hyper-parameter optimization module. They can be run like:

$ pykeen experiments ablation ~/path/to/config.json

Large-scale Reproducibility and Benchmarking Study

We used PyKEEN to perform a large-scale reproducibility and benchmarking study which are described in our article:

@article{ali2020benchmarking,
  author={Ali, Mehdi and Berrendorf, Max and Hoyt, Charles Tapley and Vermue, Laurent and Galkin, Mikhail and Sharifzadeh, Sahand and Fischer, Asja and Tresp, Volker and Lehmann, Jens},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  title={Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models under a Unified Framework},
  year={2021},
  pages={1-1},
  doi={10.1109/TPAMI.2021.3124805}}
}

We have made all code, experimental configurations, results, and analyses that lead to our interpretations available at https://github.com/pykeen/benchmarking.

Contributing

Contributions, whether filing an issue, making a pull request, or forking, are appreciated. See CONTRIBUTING.md for more information on getting involved.

Acknowledgements

Supporters

This project has been supported by several organizations (in alphabetical order):

• Bayer
• CoronaWhy
• Enveda Biosciences
• Fraunhofer Institute for Algorithms and Scientific Computing
• Fraunhofer Institute for Intelligent Analysis and Information Systems
• Fraunhofer Center for Machine Learning
• Harvard Program in Therapeutic Science - Laboratory of Systems Pharmacology
• Ludwig-Maximilians-Universität München
• Munich Center for Machine Learning (MCML)
• Siemens
• Smart Data Analytics Research Group (University of Bonn & Fraunhofer IAIS)
• Technical University of Denmark - DTU Compute - Section for Cognitive Systems
• Technical University of Denmark - DTU Compute - Section for Statistics and Data Analysis
• University of Bonn

Funding

The development of PyKEEN has been funded by the following grants:

Funding Body	Program	Grant
DARPA	Young Faculty Award (PI: Benjamin Gyori)	W911NF2010255
DARPA	Automating Scientific Knowledge Extraction (ASKE)	HR00111990009
German Federal Ministry of Education and Research (BMBF)	Maschinelles Lernen mit Wissensgraphen (MLWin)	01IS18050D
German Federal Ministry of Education and Research (BMBF)	Munich Center for Machine Learning (MCML)	01IS18036A
Innovation Fund Denmark (Innovationsfonden)	Danish Center for Big Data Analytics driven Innovation (DABAI)	Grand Solutions

Logo

The PyKEEN logo was designed by Carina Steinborn

Citation

If you have found PyKEEN useful in your work, please consider citing our article:

@article{ali2021pykeen,
    author = {Ali, Mehdi and Berrendorf, Max and Hoyt, Charles Tapley and Vermue, Laurent and Sharifzadeh, Sahand and Tresp, Volker and Lehmann, Jens},
    journal = {Journal of Machine Learning Research},
    number = {82},
    pages = {1--6},
    title = {{PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings}},
    url = {http://jmlr.org/papers/v22/20-825.html},
    volume = {22},
    year = {2021}
}