Skip to main content

A library for approximating loss landscapes in low-dimensional parameter subspaces

Project description

loss-landscapes

loss-landscapes is a library for approximating the value of neural network loss functions in low-dimensional subspaces suitable for visualization. The intended purpose of this library is to simplify the production of loss landscape visualizations such as those seen in Visualizing the Loss Landscape of Neural Nets.

What is a Loss Landscape?

Let L : Parameters -> Real Numbers be a loss function, which maps the vector containing all parameters of a neural network model to a real number. This is true of loss functions in general, of course, but we focus on neural networks. Since the network has n parameters, the loss function L takes an n-dimensional input. We can define the loss landscape as the set of all n+1-dimensional points (param, L(param)), for all parameter vectors param. For example, the image below, reproduced from the paper mentioned above by Li et al (2018), provides a visual representation of what a loss function over a two-dimensional parameter space might look like:

Loss Landscape

Of course, in reality machine learning models have many more parameters than just two, so the input space of the loss function is virtually never two-dimensional. In fact, the model whose loss landscape is shown in the image above actually has far more than two parameters, too. Because we can't produce visualizations in more than three dimensions, we cannot hope to visualize the "true" shape of the loss landscape. Instead, a number of techniques exist for reducing the parameter space to one or two dimensions, ranging from dimensionality reduction techniques like PCA, to restricting ourselves to a particular subspace of the overall parameter space. For more details, read Li et al's paper above.

What does loss-landscapes do?

This library facilitates the computation of a neural network model's loss landscape. Its core design principles are that the library should not mandate any visualization technique or any particular definition of loss, and it should be agnostic to the numerical computation library in use. As an example, it allows a PyTorch user to examine the topology of a model's loss function around its current parameters with filter-normalized directions by calling

landscape = loss_landscape.random_plane(model, evaluation_function, normalize='filter')

This line would return a 2-dimensional array of loss values, which could in turn be plotted with any software. Below is a simple contour plot made in matplotlib that demonstrates what a planar loss landscape could look like. Check the examples directory for more in-depth examples of what is possible.

Loss Contour

Current and Planned Features

Currently, the library has the following features:

  1. Computing loss functions along a line defined by two points
  2. Computing loss functions along a random direction, starting from a specified point
  3. Computing loss functions along a plane defined by three points
  4. Computing loss functions along a plane defined by a specified point and two random direction vectors

Furthermore, filter normalization can be applied to the random direction vectors. Currently, the only supported numerical computation library is PyTorch.

Future versions will increase the number of loss landscapes that are computable, as well as expand support to TensorFlow, scikit-learn, and more.

Installation

The package will be uploaded on PyPI once it is sufficiently developed. Up to that point, it can be installed by cloning this repository, and running pip install -e ./loss-landscapes/, where loss-landscapes is the directory this repository was cloned into.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

loss_landscapes-0.1.0.tar.gz (6.3 kB view details)

Uploaded Source

Built Distribution

loss_landscapes-0.1.0-py3-none-any.whl (6.5 kB view details)

Uploaded Python 3

File details

Details for the file loss_landscapes-0.1.0.tar.gz.

File metadata

  • Download URL: loss_landscapes-0.1.0.tar.gz
  • Upload date:
  • Size: 6.3 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7

File hashes

Hashes for loss_landscapes-0.1.0.tar.gz
Algorithm Hash digest
SHA256 7b6c4d95ed99fb99f9809743a3208643541487f0e780d70ba875d20da86824e6
MD5 ce52d6db91422d6c64ddad44e46e16af
BLAKE2b-256 0a245f20daa9cab95cee63070d785f0e538a96a5df0d237cd184af9acbd66899

See more details on using hashes here.

File details

Details for the file loss_landscapes-0.1.0-py3-none-any.whl.

File metadata

  • Download URL: loss_landscapes-0.1.0-py3-none-any.whl
  • Upload date:
  • Size: 6.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.7

File hashes

Hashes for loss_landscapes-0.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e1ea2a295398e668a4a65f2346b01987e8a22df81556eb415d097768e8d8a241
MD5 ceec177b63c46c2cda08966bb55ab64f
BLAKE2b-256 0bbae9022b6f1503e41ccfd6e5aabaeba159ffddc8ba8e7d5aa6693e9ae036ab

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page