Heuristically optimize learning rates in neural networks through subsampling loss differentials.
Auto LR Tuner
The learning rate is often one of the most important hyperparameters when training neural networks. Adaptive gradient-based methods (e.g. ADAM), decreasing learning rates based on a validation set, and cosine annealing are common tricks done in practice to improve convergence.
This library provides a simple algorithm for automatically tuning learning rates for TensorFlow Keras models. Importantly, these methods are largely based on heuristics and my own experience training neural networks and there are no formal results.
Algorithm Details
The main idea behind this implementation is to estimate the optimal learning rate by trying to determine the steepness of the loss surface. Intuitively, a very small learning rate leads to almost no change in losses, while an excessively large learning rate can overshoot local minima and even increase the loss.
We start with a user-specified grid $\Theta = [lr_{min}, lr_{max}]$ of potential learning rates to search over. Next, we subsample $m$ observations from the data, and evaluate $\Delta L := L_{post} - L_{pre}$ where $L_{pre}$ is the baseline loss value, and $L_{post}$ is the loss after a single backpropagation step for a given learning rate $\theta \in \Theta$. The subsampling process is repeated $M$ times to approximate $E_{M}[\Delta L]$ and construct confidence intervals based on the subsampled distribution $\hat{P}_{\Delta L, M}$.
In this interpretation of the problem, the "optimal" learning rate is the one that consistently decreases the loss across the dataset without excessively high variance. For example in the plot below, a small learning rate near-zero may be very slow to converge while a learning rate of one actually increases the loss on average. Here the optimal rate may be something close to 0.125.
Getting Started
Install from PyPi:
$ pip install autolrtuner
Dependencies
python >= 3.8tensorflow >= 2.5numpypandasscipytqdmmatplotlib(Optional)
Example
from autolrtuner import AutoLRTuner
# Compiled TensorFlow Keras model
model
# Run the tuning algorithm
lrTuner = AutoLRTuner(model)
lrTuner.Tune(X, Y, num_subsamples=100, num_evals=100, batch_size=32)
# Get the optimal learning rate
bestLR = lrTuner.GetBestLR(method='mean')
# Plot the loss differentials
lrTuner.Plot()
Metadata
Release files for autolrtuner 1.0.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| autolrtuner-1.0.2.tar.gz | 7.5 kB | Without provenance | Details for autolrtuner-1.0.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| autolrtuner-1.0.2-py3-none-any.whl | Python 3 | none | any | Details for autolrtuner-1.0.2-py3-none-any.whl |
Total release size: 15.0 kB
Release files / autolrtuner-1.0.2.tar.gz
| Download URL | autolrtuner-1.0.2.tar.gz |
|---|---|
| Size | 7.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
dad8a89e612565cb9d1629899a0636d42e4b68e42739dcc2ee5207b5886d5928
|
|
BLAKE2b-256 checksum How to use checksums |
633f3b8c22846f5b20423ad4dc8cf1ab6e93b68a09c5be5720bd0d9dc1a447af
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Release files / autolrtuner-1.0.2-py3-none-any.whl
| Download URL | autolrtuner-1.0.2-py3-none-any.whl |
|---|---|
| Size | 7.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ab50b3be740128a7cd226f5f4845d4ad77715623b24166edb9c550ab5c5dc1bb
|
|
BLAKE2b-256 checksum How to use checksums |
64af901b3d0906ff4dcceaaffcad023833b5d0fa495da0341f5f73c8e75f5ee4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Security for autolrtuner 1.0.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.