varz 0.3.0

Painless variables in PyTorch and TensorFlow

Varz

Painless optimisation of constrained variables in PyTorch, TensorFlow, and AutoGrad

Note: Varz requires Python 3.5+ TensorFlow 2 if TensorFlow is used.

• Installation
• Manual
  • Naming
  • Constrained Variables
  • Getting and Setting Variables as a Vector
  • AutoGrad
  • TensorFlow
  • PyTorch

Installation

Before installing the package, please ensure that gcc and gfortran are available. On OS X, these are both installed with brew install gcc; users of Anaconda may want to instead consider conda install gcc. On Linux, gcc is most likely already available, and gfortran can be installed with apt-get install gfortran. Then simply

pip install varz

Manual

from varz import Vars

To begin with, create a variable container of the right data type. For use with NumPy and AutoGrad, use a np.* data type; for use with PyTorch, use a torch.* data type; and for use with TensorFlow, use a tf.* data type. In this example we'll use NumPy and AutoGrad.

>>> vs = Vars(np.float64)

Now a variable can be created by requesting it, giving it an initial value and a name.

>>> vs.get(np.random.randn(2, 2), name='x')
array([[ 1.04404354, -1.98478763],
       [ 1.14176728, -3.2915562 ]])

If the same variable is created again, because a variable with the name x already exists, the existing variable will be returned.

>>> vs.get(name='x')
array([[ 1.04404354, -1.98478763],
       [ 1.14176728, -3.2915562 ]])

Alternatively, indexing syntax may be used to get the existing variable x.

>>> vs['x']
array([[ 1.04404354, -1.98478763],
       [ 1.14176728, -3.2915562 ]])

The value of x may be changed by assigning it a different value.

>>> vs.assign('x', np.random.randn(2, 2))
array([[ 1.43477728,  0.51006941],
       [-0.74686452, -1.05285767]])

By default, assignment is non-differentiable and overwrites data. For differentiable assignment, which replaces data, set the keyword argument differentiable=True.

>>> vs.assign('x', np.random.randn(2, 2), differentiable=True)
array([[ 0.12500578, -0.21510423],
       [-0.61336039,  1.23074066]])

The variable container can be copied with vs.copy(). Note that the copy shares its variables with the original. This means that assignment will also mutate the original; differentiable assignment, however, will not.

Naming

Variables may be organised by naming them hierarchically using /s. For example, group1/bar, group1/foo, and group2/bar. This is helpful for extracting collections of variables, where wildcards may be used to match names. For example, */bar would match group1/bar and group2/bar, and group1/* would match group1/bar and group1/foo.

Constrained Variables

A variable that is constrained to be positive can be created using Vars.positive or Vars.pos.

>>> vs.pos(name='positive_variable')
0.016925610008314832

A variable that is constrained to be bounded can be created using Vars.bounded or Vars.bnd.

>>> vs.bnd(name='bounded_variable', lower=1, upper=2)
1.646772663807718

These constrained variables are created by transforming some latent unconstrained representation to the desired constrained space. The latent variables can be obtained using Vars.get_vars.

>>> vs.get_vars('positive_variable', 'bounded_variable')
[array(-4.07892742), array(-0.604883)]

To illustrate the use of wildcards, the following is equivalent:

>>> vs.get_vars('*_variable')
[array(-4.07892742), array(-0.604883)]

Getting and Setting Variables as a Vector

It may be desirable to get the latent representations of a collection of variables as a single vector, e.g. when feeding them to an optimiser. This can be achieved with Vars.get_vector.

>>> vs.get_vector('x', '*_variable')
array([ 0.12500578, -0.21510423, -0.61336039,  1.23074066, -4.07892742,
       -0.604883  ])

Similarly, to update the latent representation of a collection of variables, Vars.set_vector can be used.

>>> vs.set_vector(np.ones(6), 'x', '*_variable')
[array([[1., 1.],
        [1., 1.]]), array(1.), array(1.)]

>>> vs.get_vector('x', '*_variable')
array([1., 1., 1., 1., 1., 1.])

AutoGrad

The function varz.autograd.minimise_l_bfgs_b can be used to perform minimisation using the L-BFGS-B algorithm. The function varz.autograd.minimise_adam can be used to perform minimisation of stochastic objectives using Adam.

Example of optimising variables:

import autograd.numpy as np
from varz.autograd import Vars, minimise_l_bfgs_b

target = 5. 


def objective(x):  # `x` must be positive!
    return (x ** .5 - target) ** 2  

>>> vs = Vars(np.float64)

>>> vs.pos(10., name='x')
10.000000000000002

>>> minimise_l_bfgs_b(lambda v: objective(v['x']), vs, names=['x'])
3.17785950743424e-19  # Final objective function value.

>>> vs['x'] - target ** 2
-5.637250666268301e-09

TensorFlow

The function varz.tensorflow.minimise_l_bfgs_b can be used to perform minimisation using the L-BFGS-B algorithm. The function varz.tensorflow.minimise_adam can be used to perform minimisation of stochastic objectives using Adam.

Example of optimising variables:

import tensorflow as tf
from varz.tensorflow import Vars, minimise_l_bfgs_b

target = 5.


def objective(x):  # `x` must be positive!
    return (x ** .5 - target) ** 2  

>>> vs = Vars(tf.float64)

>>> vs.pos(10., name='x')
<tf.Tensor: id=11, shape=(), dtype=float64, numpy=10.000000000000002>

>>> minimise_l_bfgs_b(lambda v: objective(v['x']), vs, names=['x'])
3.17785950743424e-19  # Final objective function value.

>>> vs['x'] - target ** 2
<tf.Tensor: id=562, shape=(), dtype=float64, numpy=-5.637250666268301e-09>

PyTorch

All the variables held by a container can be detached from the current computation graph with Vars.detach . To make a copy of the container with detached versions of the variables, use Vars.copy with detach=True instead. Whether variables require gradients can be configured with Vars.requires_grad. By default, no variable requires a gradient.

The function varz.torch.minimise_l_bfgs_b can be used to perform minimisation using the L-BFGS-B algorithm. The function varz.torch.minimise_adam can be used to perform minimisation of stochastic objectives using Adam.

Example of optimising variables:

import torch
from varz.torch import Vars, minimise_l_bfgs_b


target = torch.tensor(5., dtype=torch.float64)


def objective(x):  # `x` must be positive!
    return (x ** .5 - target) ** 2

>>> vs = Vars(torch.float64)

>>> vs.pos(10., name='x')
tensor(10.0000, dtype=torch.float64)

>>> minimise_l_bfgs_b(lambda v: objective(v['x']), vs, names=['x'])
array(3.17785951e-19)  # Final objective function value.

>>> vs['x'] - target ** 2
tensor(-5.6373e-09, dtype=torch.float64)

Get Variables from a Source

The keyword argument source can set to a tensor from which the latent variables will be obtained.

Example:

>>> vs = Vars(np.float32, source=np.array([1, 2, 3, 4, 5]))

>>> vs.get()
array(1., dtype=float32)

>>> vs.get(shape=(3,))
array([2., 3., 4.], dtype=float32)

>>> vs.pos()
148.41316

>>> np.exp(5).astype(np.float32)
148.41316