latte-metrics 0.0.1a2

Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models

Cross-framework Python Package for Evaluation of Latent-based Generative Models

Latte

Latte (for LATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.

Installation

For developers working on local clone, cd to the repo and replace latte with .. For example, pip install .[tests]

pip install latte-metrics           # core (numpy only)
pip install latte-metrics[pytorch]  # with torchmetrics wrapper
pip install latte-metrics[keras]    # with tensorflow wrapper
pip install latte-metrics[tests]    # for testing

Running tests locally

pip install .[tests]
pytest tests/ --cov=latte

Example

Functional API

import latte
from latte.functional.disentanglement.mutual_info import mig
import numpy as np

latte.seed(42)

z = np.random.randn(16, 8)
a = np.random.randn(16, 2)

mutual_info_gap = mig(z, a, discrete=False, reg_dim=[4, 3])

Modular API

import latte
from latte.metrics.core.disentanglement import MutualInformationGap
import numpy as np

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.compute()

TorchMetrics API

import latte
from latte.metrics.torch.disentanglement import MutualInformationGap
import torch

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update(z, attributes)

mig_val = mig.compute()

Keras Metric API

import latte
from latte.metrics.keras.disentanglement import MutualInformationGap
from tensorflow import keras as tfk

latte.seed(42)

mig = MutualInformationGap()

# ... 
# initialize data and model
# ...

for data, attributes in range(batches):
  recon, z = model(data)

  mig.update_state(z, attributes)

mig_val = mig.result()

Documentation

https://latte.readthedocs.io/en/latest

Method Chart for Modular API

TorchMetrics: https://torchmetrics.readthedocs.io/en/latest/pages/implement.html

Keras Metric: https://www.tensorflow.org/api_docs/python/tf/keras/metrics/Metric

Torch/Keras wrapper will

1. convert torch/tf types to numpy types (and move everything to CPU)
2. call native class methods
3. if there are return values, convert numpy types back to torch/tf types

	Native	TorchMetrics	Keras Metric
base class	latte.metrics.LatteMetric	torchmetrics.Metric	tf.keras.metrics.Metric
super class	object	torch.nn.Module	tf.keras.layers.Layer
adding buffer	self.add_state	self.add_state	self.add_weight
updating buffer	self.update_state	self.update	self.update_state
resetting buffer	self.reset_state	self.reset	self.reset_state
computing metric values	self.compute	self.compute	self.result

Supported metrics

🧪 Beta support | ✔️ Stable | 🔨 In Progress | 🕣 In Queue | 👀 KIV |

Metric	Latte Functional	Latte Modular	TorchMetrics	Keras Metric
Disentanglement Metrics
📝 Mutual Information Gap (MIG)	🧪	🧪	🧪	🧪
📝 Dependency-blind Mutual Information Gap (DMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Mutual Information Gap (XMIG)	🧪	🧪	🧪	🧪
📝 Dependency-aware Latent Information Gap (DLIG)	🧪	🧪	🧪	🧪
📝 Separate Attribute Predictability (SAP)	🧪	🧪	🧪	🧪
📝 Modularity	🧪	🧪	🧪	🧪
📝 β-VAE Score	👀	👀	👀	👀
📝 FactorVAE Score	👀	👀	👀	👀
📝 DCI Score	👀	👀	👀	👀
📝 Interventional Robustness Score (IRS)	👀	👀	👀	👀
📝 Consistency	👀	👀	👀	👀
📝 Restrictiveness	👀	👀	👀	👀
Interpolatability Metrics
📝 Smoothness	🧪	🧪	🧪	🧪
📝 Monotonicity	🧪	🧪	🧪	🧪
📝 Latent Density Ratio	🕣	🕣	🕣	🕣
📝 Linearity	👀	👀	👀	👀

Bundled metric modules

🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In Queue

Metric Bundle	Latte Functional	Latte Modular	TorchMetrics	Keras Metric	Included
Dependency-aware Disentanglement	🧪	🔨	🕣	🕣	MIG, DMIG, XMIG, DLIG
LIAD-based Interpolatability	🧪	🔨	🕣	🕣	Smoothness, Monotonicity

Cite

For individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.

If you find our package useful please cite us as

@software{
  watcharasupat2021latte,
  author = {Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},
  title = {{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},
  url = {https://github.com/karnwatcharasupat/latte},
  version = {0.0.1-alpha1}
}