Skip to main content

MADGRAD Optimization Algorithm For Tensorflow

This package implements the MadGrad Algorithm proposed in Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization (Aaron Defazio and Samy Jelassi, 2021).

MIT License version-shield release-shield python-shield code-style

Table of Contents
  1. About The Project
  2. Getting Started
  3. Usage
  4. Contributing
  5. License
  6. Contact
  7. Citations

About The Project

The MadGrad algorithm of optimization uses Dual averaging of gradients along with momentum based adaptivity to attain results that match or outperform Adam or SGD + momentum based algorithms. This project offers a Tensorflow implementation of the algorithm along with a few usage examples and tests.



Prerequisites

Prerequisites can be installed separately through the requirements.txt file as below

pip install -r requirements.txt

Installation

This project is built with Python 3 and can be pip installed directly

pip install tf-madgrad

Usage

Open In Colab

To use the optimizer in any tf.keras model, you just need to import and instantiate the MadGrad optimizer from the tf_madgrad package.

from madgrad import MadGrad

# Create the architecture
inp = tf.keras.layers.Input(shape=shape)
...
op = tf.keras.layers.Dense(classes, activation=activation)

# Instantiate the model
model = tf.keras.models.Model(inp, op)

# Pass the MadGrad optimizer to the compile function
model.compile(optimizer=MadGrad(lr=0.01), loss=loss)

# Fit the keras model as normal
model.fit(...)

This implementation is also supported for distributed training using tf.strategy

See a MNIST example here

Contributing

Any and all contributions are welcome. Please raise an issue if the optimizer gives incorrect results or crashes unexpectedly during training.

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Feel free to reach out for any issues or requests related to this implementation

Darshan Deshpande - Email | LinkedIn

Citations

@misc{defazio2021adaptivity,
      title={Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization}, 
      author={Aaron Defazio and Samy Jelassi},
      year={2021},
      eprint={2101.11075},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

Metadata

Release files for tf-madgrad 1.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tf-madgrad 1.0.2
File Size Uploaded Provenance Actions
tf-madgrad-1.0.2.tar.gz 8.9 kB Without provenance Details for tf-madgrad-1.0.2.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for tf-madgrad 1.0.2
File Interpreter ABI Platform Actions
tf_madgrad-1.0.2-py3.7.egg Legacy Egg format - - Details for tf_madgrad-1.0.2-py3.7.egg
tf_madgrad-1.0.2-py3-none-any.whl Python 3 none any Details for tf_madgrad-1.0.2-py3-none-any.whl

Total release size: 23.0 kB

Release files / tf-madgrad-1.0.2.tar.gz

Download URL tf-madgrad-1.0.2.tar.gz
Size 8.9 kB
Tags Source
SHA-256 checksum
How to use checksums
ece312faa7ec6154b6c78d3783e6b09dec6c96d84c708808c6b9050491376aea
BLAKE2b-256 checksum
How to use checksums
ccc504a2c26f9770003fd84d9abc72542465f3252d6c93f7503d498896b386d0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.7.13

Release files / tf_madgrad-1.0.2-py3.7.egg

Download URL tf_madgrad-1.0.2-py3.7.egg
Size 8.1 kB
Tags Egg
SHA-256 checksum
How to use checksums
88746a64d05b9ee36b6aaa58700d18b74e6f7a857c943ba9c584f397b08cc4a3
BLAKE2b-256 checksum
How to use checksums
70241f15ef8f9f3f8f0a4c5ab736880a7734453dc1fb437070cda1750e6cdd50
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.7.13

Release files / tf_madgrad-1.0.2-py3-none-any.whl

Download URL tf_madgrad-1.0.2-py3-none-any.whl
Size 5.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
1eec9f64ac4fe7fe47baa89cc99694a1a6b662ef732fd31c45194487bf657570
BLAKE2b-256 checksum
How to use checksums
70396923007b65809314c0a153d4b4b882e7d04f0b1d8694ef1192e7b4228a81
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/4.0.1 CPython/3.7.13

Security for tf-madgrad 1.0.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page