adapters 0.0.0.dev20231111

A Unified Library for Parameter-Efficient and Modular Transfer Learning

adapters Library

This branch contains the development version of adapters, the next-generation library for parameter-efficient and modular transfer learning.

Note: For the stable version of adapter-transformers, please switch to the master branch of the repo.

Changes compared to adapter-transformers

• adapters is a standalone package, using transformers as an external dependency but not patching it directly
• All adapter-related classes now are imported via adapters namespace, e.g.:

  from adapters import BertAdapterModel
  # ...
  

• Built-in HF model classes can be adapted for usage with adapters via a wrapper method, e.g.:

  import adapters
  from transformers import BertModel
  
  model = BertModel.from_pretrained("bert-base-uncased")
  adapters.init(model)
  

Features not (yet) working:

• Loading model + adapter checkpoints using HF classes
• Using Transformers pipelines with adapters
• Using HF language modeling classes with invertible adapters

Documentation

To read the documentation of Adapters, follow the steps in docs/README.md. Currently, the documentation is not yet available from https://docs.adapterhub.ml/.

---

Adapters

A Unified Library for Parameter-Efficient and Modular Transfer Learning

adapters is an add-on to HuggingFace's Transformers library, integrating adapters into state-of-the-art language models by incorporating AdapterHub, a central repository for pre-trained adapter modules.

Installation

adapters currently supports Python 3.8+ and PyTorch 1.10+. After installing PyTorch, you can install adapters from PyPI ...

pip install -U adapters

... or from source by cloning the repository:

git clone https://github.com/adapter-hub/adapter-transformers.git
git checkout adapters
cd adapters
pip install .

Getting Started

HuggingFace's great documentation on getting started with Transformers can be found here. adapters is fully compatible with Transformers.

To get started with adapters, refer to these locations:

• Colab notebook tutorials, a series notebooks providing an introduction to all the main concepts of (adapter-)transformers and AdapterHub
• https://docs.adapterhub.ml, our documentation on training and using adapters with adapters
• https://adapterhub.ml to explore available pre-trained adapter modules and share your own adapters
• Examples folder of this repository containing HuggingFace's example training scripts, many adapted for training adapters

Implemented Methods

Currently, adapters integrates all architectures and methods listed below:

Method	Paper(s)	Quick Links
Bottleneck adapters	Houlsby et al. (2019) Bapna and Firat (2019)	Quickstart, Notebook
AdapterFusion	Pfeiffer et al. (2021)	Docs: Training, Notebook
MAD-X, Invertible adapters	Pfeiffer et al. (2020)	Notebook
AdapterDrop	Rücklé et al. (2021)	Notebook
MAD-X 2.0, Embedding training	Pfeiffer et al. (2021)	Docs: Embeddings, Notebook
Prefix Tuning	Li and Liang (2021)	Docs
Parallel adapters, Mix-and-Match adapters	He et al. (2021)	Docs
Compacter	Mahabadi et al. (2021)	Docs
LoRA	Hu et al. (2021)	Docs
(IA)^3	Liu et al. (2022)	Docs
UniPELT	Mao et al. (2022)	Docs

Supported Models

We currently support the PyTorch versions of all models listed on the Model Overview page in our documentation.

Citation

If you use this library for your work, please consider citing our paper AdapterHub: A Framework for Adapting Transformers:

@inproceedings{pfeiffer2020AdapterHub,
    title={AdapterHub: A Framework for Adapting Transformers},
    author={Pfeiffer, Jonas and
            R{\"u}ckl{\'e}, Andreas and
            Poth, Clifton and
            Kamath, Aishwarya and
            Vuli{\'c}, Ivan and
            Ruder, Sebastian and
            Cho, Kyunghyun and
            Gurevych, Iryna},
    booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
    pages={46--54},
    year={2020}
}