Adapt Transformer-based language models to new text domains
Project description
This toolkit improves the performance of HuggingFace transformer models on downstream NLP tasks, by domain-adapting models to the target domain of said NLP tasks (e.g. BERT -> LawBERT).
The overall Domain Adaptation framework can be broken down into three phases:
- Data Selection
Select a relevant subset of documents from the in-domain corpus that is likely to be beneficial for domain pre-training (see below)
- Vocabulary Augmentation
Extending the vocabulary of the transformer model with domain specific-terminology
- Domain Pre-Training
Continued pre-training of transformer model on the in-domain corpus to learn linguistic nuances of the target domain
After a model is domain-adapted, it can be fine-tuned on the downstream NLP task of choice, like any pre-trained transformer model.
Components
This toolkit provides two classes, DataSelector
and VocabAugmentor
, to simplify the Data Selection and Vocabulary Augmentation steps respectively.
Installation
This package was developed on Python 3.6+ and can be downloaded using pip
:
pip install transformers-domain-adaptation
Features
- Compatible with the HuggingFace ecosystem:
transformers 4.x
tokenizers
datasets
Usage
Please refer to our Colab guide!
Results
TODO
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for transformers-domain-adaptation-0.3.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 465e8bae317404b7c8d3ec01057ddf59affd4893ac408b92c7e873580f266f5a |
|
MD5 | 408f0ff0f26cb796c9457e0aa1e42165 |
|
BLAKE2b-256 | 157dd99b55933e3cd714ff97190c1533480fbde7fb297d026b39aaa2fea910cf |
Hashes for transformers_domain_adaptation-0.3.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 28ca22275511c5166e286eb90bcbb3077892a1ffcb7cb1fd214d8b91af0d44a2 |
|
MD5 | 4a6a5a70df29aea93917eb940e1431d4 |
|
BLAKE2b-256 | 507fd38972776c8b3fe67341e19b55219b44d5c542098868c3420fad8d430fb7 |