Skip to main content

Build PyPi Downloads License

Heuristically optimize learning rates in neural networks through subsampling loss differentials.


Auto LR Tuner

The learning rate is often one of the most important hyperparameters when training neural networks. Adaptive gradient-based methods (e.g. ADAM), decreasing learning rates based on a validation set, and cosine annealing are common tricks done in practice to improve convergence.

This library provides a simple algorithm for automatically tuning learning rates for TensorFlow Keras models. Importantly, these methods are largely based on heuristics and my own experience training neural networks and there are no formal results.

Algorithm Details

The main idea behind this implementation is to estimate the optimal learning rate by trying to determine the steepness of the loss surface. Intuitively, a very small learning rate leads to almost no change in losses, while an excessively large learning rate can overshoot local minima and even increase the loss.

We start with a user-specified grid $\Theta = [lr_{min}, lr_{max}]$ of potential learning rates to search over. Next, we subsample $m$ observations from the data, and evaluate $\Delta L := L_{post} - L_{pre}$ where $L_{pre}$ is the baseline loss value, and $L_{post}$ is the loss after a single backpropagation step for a given learning rate $\theta \in \Theta$. The subsampling process is repeated $M$ times to approximate $E_{M}[\Delta L]$ and construct confidence intervals based on the subsampled distribution $\hat{P}_{\Delta L, M}$.

In this interpretation of the problem, the "optimal" learning rate is the one that consistently decreases the loss across the dataset without excessively high variance. For example in the plot below, a small learning rate near-zero may be very slow to converge while a learning rate of one actually increases the loss on average. Here the optimal rate may be something close to 0.125.

Plot

Getting Started

Install from PyPi:

$ pip install autolrtuner

Dependencies

  • python >= 3.8
  • tensorflow >= 2.5
  • numpy
  • pandas
  • scipy
  • tqdm
  • matplotlib (Optional)

Example

from autolrtuner import AutoLRTuner

# Compiled TensorFlow Keras model
model 

# Run the tuning algorithm
lrTuner = AutoLRTuner(model)
lrTuner.Tune(X, Y, num_subsamples=100, num_evals=100, batch_size=32)

# Get the optimal learning rate
bestLR = lrTuner.GetBestLR(method='mean')

# Plot the loss differentials
lrTuner.Plot()

Metadata

Release files for autolrtuner 1.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for autolrtuner 1.0.2
File Size Uploaded Provenance Actions
autolrtuner-1.0.2.tar.gz 7.5 kB Without provenance Details for autolrtuner-1.0.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for autolrtuner 1.0.2
File Interpreter ABI Platform Actions
autolrtuner-1.0.2-py3-none-any.whl Python 3 none any Details for autolrtuner-1.0.2-py3-none-any.whl

Total release size: 15.0 kB

Release files / autolrtuner-1.0.2.tar.gz

Download URL autolrtuner-1.0.2.tar.gz
Size 7.5 kB
Tags Source
SHA-256 checksum
How to use checksums
dad8a89e612565cb9d1629899a0636d42e4b68e42739dcc2ee5207b5886d5928
BLAKE2b-256 checksum
How to use checksums
633f3b8c22846f5b20423ad4dc8cf1ab6e93b68a09c5be5720bd0d9dc1a447af
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.12.5

Release files / autolrtuner-1.0.2-py3-none-any.whl

Download URL autolrtuner-1.0.2-py3-none-any.whl
Size 7.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ab50b3be740128a7cd226f5f4845d4ad77715623b24166edb9c550ab5c5dc1bb
BLAKE2b-256 checksum
How to use checksums
64af901b3d0906ff4dcceaaffcad023833b5d0fa495da0341f5f73c8e75f5ee4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/5.1.1 CPython/3.12.5

Security for autolrtuner 1.0.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page