Skip to main content

Stabilize and achieve excellent performance with transformers

Project description


Stabilize and achieve excellent performance with transformers.

The stabilizer library offer solutions to tackle one of the biggest challenges that comes along with training State of the art Transformer models, Unstable training

Unstable training

Unstable training is the phenomenon in which training large transformer models with trivial changes such as changing the random seed drastically changes the performance of the model. Here is a screenshot of finetuning the CoLA dataset from GLUE tasks with two different random seeds applied only to the dropout of the transformer model.



pip install stabilizer

Techniques currently implemented in this library

  1. Reinitialization
  2. Layerwise Learning Rate Decay


Reinitialize the last n layers of the transformer encoder. This technique works well because we reinitialize the task specific parameters that the pretrained models have learnt specific to the pretraining task.

from stabilizer.reinitialize import reinit_autoencoder_model
from transformers import AutoModel

transformer = AutoModel.from_pretrained(
transformer.encoder = reinit_autoencoder_model(
    transformer.encoder, reinit_num_layers=1

Here is the result of the same model but reinitialized last 4 layers applied on the CoLA dataset. You can see that the model has converged to almost the same performance with reinitialization. reinit_random_seed

Layerwise Learning Rate Decay

Apply layerwise learning rate to the transformer layers. Starting from the task specific layer every layer before it gets an exponentially decreasing learning rate.

from stabilizer.llrd import get_optimizer_parameters_with_llrd
from stabilizer.model import PoolerClassifier

from transformers import AdamW, AutoModel

    transformer = AutoModel.from_pretrained(

    model = PoolerClassifier(

    model_parameters = get_optimizer_parameters_with_llrd(
    optimizer = AdamW(params=model_parameters, lr=config["lr"])

Here is the result of the same model but with LLRD applied on the CoLA dataset. Here you can see that the model has diverged quite a lot by applying LLRD. Therefore as we discussed earlier their is no universal remedy yet but some techniques work well on some datasets llrd_random_seed

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

stabilizer-1.0.2.tar.gz (6.0 kB view hashes)

Uploaded Source

Built Distribution

stabilizer-1.0.2-py3-none-any.whl (9.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page