MADGRAD Optimization Algorithm For Tensorflow
This package implements the MadGrad Algorithm proposed in Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization (Aaron Defazio and Samy Jelassi, 2021).
Table of Contents
About The Project
The MadGrad algorithm of optimization uses Dual averaging of gradients along with momentum based adaptivity to attain results that match or outperform Adam or SGD + momentum based algorithms. This project offers a Tensorflow implementation of the algorithm along with a few usage examples and tests.
Prerequisites
Prerequisites can be installed separately through the requirements.txt file as below
pip install -r requirements.txt
Installation
This project is built with Python 3 and can be pip installed directly
pip install tf-madgrad
Usage
To use the optimizer in any tf.keras model, you just need to import and instantiate the MadGrad optimizer from the tf_madgrad package.
from madgrad import MadGrad
# Create the architecture
inp = tf.keras.layers.Input(shape=shape)
...
op = tf.keras.layers.Dense(classes, activation=activation)
# Instantiate the model
model = tf.keras.models.Model(inp, op)
# Pass the MadGrad optimizer to the compile function
model.compile(optimizer=MadGrad(lr=0.01), loss=loss)
# Fit the keras model as normal
model.fit(...)
This implementation is also supported for distributed training using tf.strategy
See a MNIST example here
Contributing
Any and all contributions are welcome. Please raise an issue if the optimizer gives incorrect results or crashes unexpectedly during training.
License
Distributed under the MIT License. See LICENSE for more information.
Contact
Feel free to reach out for any issues or requests related to this implementation
Darshan Deshpande - Email | LinkedIn
Citations
@misc{defazio2021adaptivity,
title={Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization},
author={Aaron Defazio and Samy Jelassi},
year={2021},
eprint={2101.11075},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
Metadata
Release files for tf-madgrad 1.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tf-madgrad-1.0.2.tar.gz | 8.9 kB | Without provenance | Details for tf-madgrad-1.0.2.tar.gz |
Built distributions (wheels)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| tf_madgrad-1.0.2-py3.7.egg | Legacy Egg format | - | - | Details for tf_madgrad-1.0.2-py3.7.egg |
| tf_madgrad-1.0.2-py3-none-any.whl | Python 3 | none | any | Details for tf_madgrad-1.0.2-py3-none-any.whl |
Total release size: 23.0 kB
Release files / tf-madgrad-1.0.2.tar.gz
| Download URL | tf-madgrad-1.0.2.tar.gz |
|---|---|
| Size | 8.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ece312faa7ec6154b6c78d3783e6b09dec6c96d84c708808c6b9050491376aea
|
|
BLAKE2b-256 checksum How to use checksums |
ccc504a2c26f9770003fd84d9abc72542465f3252d6c93f7503d498896b386d0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.7.13
|
Release files / tf_madgrad-1.0.2-py3.7.egg
| Download URL | tf_madgrad-1.0.2-py3.7.egg |
|---|---|
| Size | 8.1 kB |
| Tags | Egg |
|
SHA-256 checksum How to use checksums |
88746a64d05b9ee36b6aaa58700d18b74e6f7a857c943ba9c584f397b08cc4a3
|
|
BLAKE2b-256 checksum How to use checksums |
70241f15ef8f9f3f8f0a4c5ab736880a7734453dc1fb437070cda1750e6cdd50
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.7.13
|
Release files / tf_madgrad-1.0.2-py3-none-any.whl
| Download URL | tf_madgrad-1.0.2-py3-none-any.whl |
|---|---|
| Size | 5.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1eec9f64ac4fe7fe47baa89cc99694a1a6b662ef732fd31c45194487bf657570
|
|
BLAKE2b-256 checksum How to use checksums |
70396923007b65809314c0a153d4b4b882e7d04f0b1d8694ef1192e7b4228a81
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.7.13
|
Security for tf-madgrad 1.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.