mcr2 0.0.1

A performant implementation of the principle of Maximum Coding Rate Reduction (MCR2).

Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction

This repository is an unofficial implementation of the following paper,

ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction (2021)

by Kwan Ho Ryan Chan* (UC Berkeley), Yaodong Yu* (UC Berkeley), Chong You* (UC Berkeley), Haozhi Qi (UC Berkeley), John Wright (Columbia), and Yi Ma (UC Berkeley),

which includes the implementations of the Maximal Coding Rate Reduction (MCR²) objective function part (MCR² paper link). This also serves as the host repository for the Pip package.

What is Maximal Coding Rate Reduction?

Our goal is to learn a mapping that maps the high-dimensional data that lies in a low-dimensional manifold to low-dimensional subspaces with the following three properties:

1. Between-Class Discriminative: Features of samples from different classes/clusters should be highly uncorrelatedand belong to different low-dimensional linear subspaces
2. Within-Class Compressible: Features of samples from the same class/cluster should be relatively correlated in a sense that they belong to a low-dimensional linear subspace
3. Maximally Diverse Representation: Dimension (or variance) of features for each class/cluster should beas large as possibleas long as they stay uncorrelated from the other classes

To achieve this, we propose an objective function called Maximal Coding Rate Reduction (MCR²). In our paper, we provide not only theoretical guarantees to the desired properties upon convergence, but also practical properties such as robustness to label corruption and empirical results such as state-of-the-art unsupervised clustering performance. For more details on algorithm design, please refer to our paper.