Painless variables in PyTorch and TensorFlow
Project description
Varz
Painless optimisation of constrained variables in PyTorch, TensorFlow, and AutoGrad
Installation
Before installing the package, please ensure that gcc
and gfortran
are
available.
On OS X, these are both installed with brew install gcc
;
users of Anaconda may want to instead consider conda install gcc
.
The installation then proceeds as follows:
pip install numpy cython # These must be installed first.
pip install varz
Manual
from varz import Vars
To begin with, create a variable container of the right data type.
For use with NumPy and AutoGrad, use a np.*
data type;
for use with PyTorch, use a torch.*
data type;
and for use with TensorFlow, use a tf.*
data type.
In this example we'll use NumPy and AutoGrad.
>>> vs = Vars(np.float64)
Now a variable can be created by requesting it, giving it an initial value and a name.
>>> vs.get(np.random.randn(2, 2), name='x')
array([[ 1.04404354, -1.98478763],
[ 1.14176728, -3.2915562 ]])
If the same variable is created again, because a variable with the name x
already exists, the existing variable will be returned.
>>> vs.get(name='x')
array([[ 1.04404354, -1.98478763],
[ 1.14176728, -3.2915562 ]])
Alternatively, indexing syntax may be used to get the existing variable x
.
>>> vs['x']
array([[ 1.04404354, -1.98478763],
[ 1.14176728, -3.2915562 ]])
The value of x
may be changed by assigning it a different value.
>>> vs.assign('x', np.random.randn(2, 2))
array([[ 1.43477728, 0.51006941],
[-0.74686452, -1.05285767]])
By default, assignment is non-differentiable and overwrites data.
For differentiable assignment, set the keyword argument differentiable=True
.
>>> vs.assign('x', np.random.randn(2, 2), differentiable=True)
array([[ 0.12500578, -0.21510423],
[-0.61336039, 1.23074066]])
Note: In TensorFlow, non-differentiable assignment operations return tensors that must be run to perform the assignments.
Naming
Variables may be organised by naming them hierarchically using /
s.
For example, group1/bar
, group1/foo
, and group2/bar
.
This is helpful for extracting collections of variables, where wildcards may
be used to match names.
For example, */bar
would match group1/bar
and group2/bar
, and
group1/*
would match group1/bar
and group1/foo
.
Constrained Variables
A variable that is constrained to be positive can be created using
Vars.positive
or Vars.pos
.
>>> vs.pos(name='positive_variable')
0.016925610008314832
A variable that is constrained to be bounded can be created using
Vars.bounded
or Vars.bnd
.
>>> vs.bnd(name='bounded_variable', lower=1, upper=2)
1.646772663807718
These constrained variables are created by transforming some latent
unconstrained representation to the desired constrained space.
The latent variables can be obtained using Vars.get_vars
.
>>> vs.get_vars('positive_variable', 'bounded_variable')
[array(-4.07892742), array(-0.604883)]
To illustrate the use of wildcards, the following is equivalent:
>>> vs.get_vars('*_variable')
[array(-4.07892742), array(-0.604883)]
Getting and Setting Variables as a Vector
It may be desirable to get the latent representations of a collection of
variables as a single vector, e.g. when feeding them to an optimiser.
This can be achieved with Vars.get_vector
.
>>> vs.get_vector('x', '*_variable')
array([ 0.12500578, -0.21510423, -0.61336039, 1.23074066, -4.07892742,
-0.604883 ])
Similarly, to update the latent representation of a collection of variables,
Vars.set_vector
can be used.
>>> vs.set_vector(np.ones(6), 'x', '*_variable')
[array([[1., 1.],
[1., 1.]]), array(1.), array(1.)]
>>> vs.get_vector('x', '*_variable')
array([1., 1., 1., 1., 1., 1.])
AutoGrad
The function varz.numpy.minimise_l_bfgs_b
can be used to perform minimisation
using the L-BFGS-B algorithm.
Example of optimising variables:
import autograd.numpy as np
from varz.autograd import Vars, minimise_l_bfgs_b
target = 5.
def objective(x): # `x` must be positive!
return (x ** .5 - target) ** 2
>>> vs = Vars(np.float64)
>>> vs.pos(10., name='x')
10.000000000000002
>>> minimise_l_bfgs_b(lambda v: objective(v['x']), vs, names=['x'])
3.17785950743424e-19 # Final objective function value.
>>> vs['x'] - target ** 2
-5.637250666268301e-09
TensorFlow
All the variables held by a container can be initialised at once with
Vars.init
.
Example of optimising variables:
import tensorflow as tf
from tensorflow.contrib.opt import ScipyOptimizerInterface as SOI
from varz.tensorflow import Vars
target = tf.constant(5., dtype=tf.float64)
vs = Vars(tf.float64)
x = vs.pos(10., name='x')
objective = (x ** .5 - target) ** 2 # `x` must be positive!
>>> opt = SOI(objective, var_list=vs.get_vars('x'))
>>> sess = tf.Session()
>>> vs.init(sess)
>>> opt.minimize(sess)
>>> sess.run(vs['x'] - target ** 2)
-5.637250666268301e-09
PyTorch
All the variables held by a container can be detached from the current
computation graph with Vars.detach_vars
.
To make a copy of the container with detached versions of the variables, use
Vars.detach
instead.
Whether variables require gradients can be configured with Vars.requires_grad
.
By default, no variable requires a gradient.
The function varz.torch.minimise_l_bfgs_b
can be used to perform minimisation
using the L-BFGS-B algorithm.
Example of optimising variables:
import torch
from varz.torch import Vars, minimise_l_bfgs_b
target = torch.tensor(5., dtype=torch.float64)
def objective(x): # `x` must be positive!
return (x ** .5 - target) ** 2
>>> vs = Vars(torch.float64)
>>> vs.pos(10., name='x')
tensor(10.0000, dtype=torch.float64)
>>> minimise_l_bfgs_b(lambda v: objective(v['x']), vs, names=['x'])
array(1.36449515e-13) # Final objective function value.
>>> vs['x'] - target ** 2
tensor(1.6378e-07, dtype=torch.float64)
Get Variables from a Source
The keyword argument source
can set to a tensor from which the latent
variables will be obtained.
Example:
>>> vs = Vars(np.float32, source=np.array([1, 2, 3, 4, 5]))
>>> vs.get()
array(1., dtype=float32)
>>> vs.get(shape=(3,))
array([2., 3., 4.], dtype=float32)
>>> vs.pos()
148.41316
>>> np.exp(5).astype(np.float32)
148.41316
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.