BERT for Multi-task Learning
Project description
Bert for Multi-task Learning
Note: Since 0.4.0, tf version >= 2.1 is required.
Install
pip install bert-multitask-learning
What is it
This a project that uses BERT to do multi-task learning with multiple GPU support.
Why do I need this
In the original BERT code, neither multi-task learning or multiple GPU training is possible. Plus, the original purpose of this project is NER which dose not have a working script in the original BERT code.
To sum up, compared to the original bert repo, this repo has the following features:
- Multi-task learning(major reason of re-writing the majority of code).
- Multiple GPU training
- Support sequence labeling (for example, NER) and Encoder-Decoder Seq2Seq(with transformer decoder).
What type of problems are supported?
- Masked LM and next sentence prediction Pre-train(pretrain)
- Classification(cls)
- Sequence Labeling(seq_tag)
- Seq2seq Labeling(seq2seq_tag)
- Seq2seq Text Generation(seq2seq_text)
- Multi-Label Classification(multi_cls)
How to run pre-defined problems
There are two types of chaining operations can be used to chain problems.
&
. If two problems have the same inputs, they can be chained using&
. Problems chained by&
will be trained at the same time.|
. If two problems don't have the same inputs, they need to be chained using|
. Problems chained by|
will be sampled to train at every instance.
For example, cws|NER|weibo_ner&weibo_cws
, one problem will be sampled at each turn, say weibo_ner&weibo_cws
, then weibo_ner
and weibo_cws
will trained for this turn together. Therefore, in a particular batch, some tasks might not be sampled, and their loss could be 0 in this batch.
Please see the examples in notebooks for more details about training, evaluation and export models.
Bert多任务学习
注意:版本0.4.0后要求tf>=2.1
安装
pip install bert-multitask-learning
这是什么
这是利用BERT进行多任务学习并且支持多GPU训练的项目.
我为什么需要这个项目
在原始的BERT代码中, 是没有办法直接用多GPU进行多任务学习的. 另外, BERT并没有给出序列标注和Seq2seq的训练代码.
因此, 和原来的BERT相比, 这个项目具有以下特点:
- 多任务学习
- 多GPU训练
- 序列标注以及Encoder-decoder seq2seq的支持(用transformer decoder)
目前支持的任务类型
- Masked LM和next sentence prediction预训练(pretrain)
- 单标签分类(cls)
- 序列标注(seq_tag)
- 序列到序列标签标注(seq2seq_tag)
- 序列到序列文本生成(seq2seq_text)
- 多标签分类(multi_cls)
如何运行预定义任务
目前支持的任务
- 中文命名实体识别
- 中文分词
- 中文词性标注
可以用两种方法来将多个任务连接起来.
&
. 如果两个任务有相同的输入, 不同标签的话, 那么他们可以用&
来连接. 被&
连接起来的任务会被同时训练.|
. 如果两个任务为不同的输入, 那么他们必须用|
来连接. 被|
连接起来的任务会被随机抽取来训练.
例如, 我们定义任务cws|NER|weibo_ner&weibo_cws
, 那么在生成每一条数据时, 一个任务块会被随机抽取出来, 例如在这一次抽样中, weibo_ner&weibo_cws
被选中. 那么这次weibo_ner
和weibo_cws
会被同时训练. 因此, 在一个batch中, 有可能某些任务没有被抽中, loss为0.
训练, eval和导出模型请见notebooks
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for bert_multitask_learning-0.4.3.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | e39e522337805d705a3e025860b79e082a2cd45feab2219a3ee41a8a8b7c42bb |
|
MD5 | 091cde516e9df34efaca29beffb44040 |
|
BLAKE2b-256 | 40f7a2ca4c15c680a066efc382eb4792cc04335f35f0a1f5796fe70315eb3549 |
Hashes for bert_multitask_learning-0.4.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ab38c2fcee27db038ce014072e31e96e818fa197c1c15768ed08cfc2e9b871f0 |
|
MD5 | 19c4c6e613a22d557ad520f8274af8b1 |
|
BLAKE2b-256 | 8020a195a7dbc09c212716c0238774081fd6efea39858afd687a938447e80b69 |