Adapt Transformer-based language models to new text domains
Project description
This toolkit improves the performance of HuggingFace transformer models on downstream NLP tasks, by domain-adapting models to the target domain of said NLP tasks (e.g. BERT -> LawBERT).
The overall Domain Adaptation framework can be broken down into three phases:
- Data Selection
Select a relevant subset of documents from the in-domain corpus that is likely to be beneficial for domain pre-training (see below)
- Vocabulary Augmentation
Extending the vocabulary of the transformer model with domain specific-terminology
- Domain Pre-Training
Continued pre-training of transformer model on the in-domain corpus to learn linguistic nuances of the target domain
After a model is domain-adapted, it can be fine-tuned on the downstream NLP task of choice, like any pre-trained transformer model.
Components
This toolkit provides two classes, DataSelector
and VocabAugmentor
, to simplify the Data Selection and Vocabulary Augmentation steps respectively.
Installation
This package was developed on Python 3.6+ and can be downloaded using pip
:
pip install transformers-domain-adaptation
Features
- Compatible with the HuggingFace ecosystem:
transformers 4.x
tokenizers
datasets
Usage
Please refer to our Colab guide!
Results
TODO
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for transformers-domain-adaptation-0.3.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 10a1e32d6586d9204a1e768dfa2937a123260bfc56bf15f49dc56a4be548c01c |
|
MD5 | 9c7edb6e445f9d51c6e9e08eb1271cdf |
|
BLAKE2b-256 | dcaf296e3fd7d68448de27996613819686d99a8d1bf7f396269b54065342b56b |
Hashes for transformers_domain_adaptation-0.3.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 923d29a46cc4a94b5a6f6cbbfe593ee80b73de7cd782097858384863bca2daa1 |
|
MD5 | 353c9855c554c65f52475589cb4e1e55 |
|
BLAKE2b-256 | 42579aad30bea5bdd398861151d4b42f5287a517f0d175cdcfe236ab9743d496 |