Python package for wrapping gradient optimizers for models in Theano
Project description
# Gradient Optimizers
Optimize you Theano Models with [Adagrad](http://www.magicbroom.info/Papers/DuchiHaSi10.pdf), Hessian Free optimization, or linear updates.
pip3 install gradient-optimizers
See example notebook (TBD) for tutorial.
Two classes **GradientModel**, and **GradientHFModel**, for optimizing gradient
based models (specifically built with indexed parameters in mind (e.g.
for language models))
## GradientModel
A gradient model for updating your model with
hessian free, adagrad, or linear decay updates.
You will need to define the following attributes,
and fill them as appropriate:
self.params = []
self.indexed_params = set()
self._l2_regularization = True / False
# if L2 is true store this parameter:
self._l2_regularization_parameter = theano.shared(np.float64(l2_regularization).astype(REAL), name='l2_regularization_parameter')
Upon initialization you must run:
self._select_update_mechanism(update_method_name)
# then to compile this mechanism:
self.create_update_fun()
The update methods expect the input to be of the form:
ivector <indices/>, ivector <labels/>
If this is not the case you can modify them as appropriate.
## GradientHFModel
Implements an symbolic one step of hessian-free [1]
optimization that approximates the curvature,
requires a _compute_cost method that takes an example
as input or a _compute_cost_gradients that returns
gradients for each example provided.
Model should have a params property containing symbolic
theano variables.
[[1] James Martens, ``Deep learning via Hessian-free optimization", ICML 2010](http://www.icml2010.org/papers/458.pdf)
Make sure the following parameters are not tampered with:
self._additional_params
self._num_updates
Optimize you Theano Models with [Adagrad](http://www.magicbroom.info/Papers/DuchiHaSi10.pdf), Hessian Free optimization, or linear updates.
pip3 install gradient-optimizers
See example notebook (TBD) for tutorial.
Two classes **GradientModel**, and **GradientHFModel**, for optimizing gradient
based models (specifically built with indexed parameters in mind (e.g.
for language models))
## GradientModel
A gradient model for updating your model with
hessian free, adagrad, or linear decay updates.
You will need to define the following attributes,
and fill them as appropriate:
self.params = []
self.indexed_params = set()
self._l2_regularization = True / False
# if L2 is true store this parameter:
self._l2_regularization_parameter = theano.shared(np.float64(l2_regularization).astype(REAL), name='l2_regularization_parameter')
Upon initialization you must run:
self._select_update_mechanism(update_method_name)
# then to compile this mechanism:
self.create_update_fun()
The update methods expect the input to be of the form:
ivector <indices/>, ivector <labels/>
If this is not the case you can modify them as appropriate.
## GradientHFModel
Implements an symbolic one step of hessian-free [1]
optimization that approximates the curvature,
requires a _compute_cost method that takes an example
as input or a _compute_cost_gradients that returns
gradients for each example provided.
Model should have a params property containing symbolic
theano variables.
[[1] James Martens, ``Deep learning via Hessian-free optimization", ICML 2010](http://www.icml2010.org/papers/458.pdf)
Make sure the following parameters are not tampered with:
self._additional_params
self._num_updates
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for gradient-optimizers-0.0.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0c0d7b53f61dff6168817535f967ec602237176ec09d12d2d18bf9389ad6854a |
|
MD5 | bf4e69456da104c77f5e8e420e81237f |
|
BLAKE2b-256 | b315739c19a273c96b9d0b3773083533c4e6bc7304b82a676fbe58154cef072d |