latte-metrics 0.0.1.dev5

Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Documentation

https://latte.readthedocs.io/en/latest

Method Chart for Modular API

TorchMetrics: https://torchmetrics.readthedocs.io/en/latest/pages/implement.html

Keras Metric: https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Metric

Torch/Keras wrapper will

1. convert torch/tf types to numpy types (and move everything to CPU)
2. call native class methods
3. if there are return values, convert numpy types back to torch/tf types

	Native	TorchMetrics	Keras Metric
base class	latte.metrics.LatteMetric	torchmetrics.Metric	tf.keras.metrics.Metric
super class	object	torch.nn.Module	tf.keras.layers.Layer
adding buffer	self.add_state	self.add_state	self.add_weight
updating buffer	self.update_state	self.update	self.update_state
resetting buffer	self.reset_state	self.reset	self.reset_state
computing metric values	self.compute	self.compute	self.result

Supported metrics

🧪 Beta support | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric	Latte Functional	Latte Modular	TorchMetrics	Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG)	🧪	🧪	🧪	🧪
📝 Dependency-blind Mutual Information Gap (DMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Mutual Information Gap (XMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Latent Information Gap (DLIG)	🧪	🧪	🧪	🧪
📝 Separate Attribute Predictability (SAP)	🧪	🧪	🧪	🧪
📝 Modularity	🧪	🧪	🧪	🧪
📝 Disentanglement metric score (β-VAE paper)	🕣	🕣	🕣	🕣
Interpolatability Metrics
📝 Smoothness	🔨	🕣	🕣	🕣
📝 Monotonicity	🔨	🕣	🕣	🕣

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle	Functional	TorchMetrics	Keras Metric	Included
Classic Disentanglement	🕣	🕣	🕣	MIG, SAP, Modularity
Dependency-aware Disentanglement	🕣	🕣	🕣	MIG, DMIG, XMIG, DLIG
Interpolatability	🕣	🕣	🕣	Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.