torch-householder 1.0.1

This package implements the Householder transformation algorithm for calculating orthogonal matrices and orthonormal frames with differentiable bindings to PyTorch. In particular, the package provides an enhanced drop-in replacement for the torch.orgqr function, which was renamed into torch.linalg.householder_product as of PyTorch 1.9.

APIs for orthogonal transformations have been around since LAPACK; however, their support in the deep learning frameworks is lacking. Recently, orthogonal constraints have become popular in deep learning as a way to regularize models and improve training dynamics, and hence the need to backpropagate through orthogonal transformations arised.

PyTorch 1.7 implements matrix exponential function torch.matrix_exp, which can be repurposed to performing the orthogonal transformation when the input matrix is skew-symmetric. This is the baseline we use in Speed and Precision evaluation.

PyTorch 1.9 renamed torch.orgqr into torch.linalg.householder_product, and added support of autograd, batching, and GPU execution.

Compared to torch.matrix_exp, the Householder transformation implemented in this package has the following advantages:

• Orders of magnitude lower memory footprint
• Ability to transform non-square matrices (orthonormal frames)
• A significant speed-up for non-square matrices
• Better numerical precision for all matrix and batch sizes

Find more details and the most up-to-date information on the project webpage: https://www.github.com/toshas/torch-householder