BERT for Multi-task Learning
Project description
Bert for Multi-task Learning
Update: Download trained checkpoint for Chinese NLP problems. (CWS|ctb_pos|msra_ner|boson_ner|weibo_ner|emotion_analysis|ontonotes_cws&ontonotes_ner, you can drop some problems if you don't need them all while starting the server. download)
Install
pip install bert-multitask-learning
What is it
This a project that uses BERT to do multi-task learning with multiple GPU support.
Why do I need this
In the original BERT code, neither multi-task learning or multiple GPU training is possible. Plus, the original purpose of this project is NER which dose not have a working script in the original BERT code.
To sum up, compared to the original bert repo, this repo has the following features:
- Multi-task learning(major reason of re-writing the majority of code).
- Multiple GPU training
- Support sequence labeling (for example, NER) and Encoder-Decoder Seq2Seq(with transformer decoder).
What type of problems are supported?
- Classification(cls)
- Sequence Labeling(seq_tag)
- Seq2seq Labeling(seq2seq_tag)
- Seq2seq Text Generation(seq2seq_text)
- (TODO)Multi-Label Classification(multi_cls)
How to run pre-defined problems
There are two types of chaining operations can be used to chain problems.
&
. If two problems have the same inputs, they can be chained using&
. Problems chained by&
will be trained at the same time.|
. If two problems don't have the same inputs, they need to be chained using|
. Problems chained by|
will be sampled to train at every instance.
For example, CWS|NER|weibo_ner&weibo_cws
, one problem will be sampled at each turn, say weibo_ner&weibo_cws
, then weibo_ner
and weibo_cws
will trained for this turn together. Therefore, in a particular batch, some tasks might not be sampled, and their loss could be 0 in this batch.
Please see the examples in notebooks for more details about training, evaluation and export models.
Bert多任务学习
更新: 在多个中文NLP任务上训练好的模型 (CWS|ctb_pos|msra_ner|boson_ner|weibo_ner|emotion_analysis|ontonotes_cws&ontonotes_ner, 如果不需要全部结果, 可以在启动服务时自行减去. 下载)
安装
pip install bert-multitask-learning
这是什么
这是利用BERT进行多任务学习并且支持多GPU训练的项目.
我为什么需要这个项目
在原始的BERT代码中, 是没有办法直接用多GPU进行多任务学习的. 另外, BERT并没有给出序列标注和Seq2seq的训练代码.
因此, 和原来的BERT相比, 这个项目具有以下特点:
- 多任务学习
- 多GPU训练
- 序列标注以及Encoder-decoder seq2seq的支持(用transformer decoder)
目前支持的任务类型
- 单标签分类(cls)
- 序列标注(seq_tag)
- 序列到序列标签标注(seq2seq_tag)
- 序列到序列文本生成(seq2seq_text)
- (TODO)多标签分类(multi_cls)
如何运行预定义任务
目前支持的任务
- 中文命名实体识别
- 中文分词
- 中文词性标注
可以用两种方法来将多个任务连接起来.
&
. 如果两个任务有相同的输入, 不同标签的话, 那么他们可以用&
来连接. 被&
连接起来的任务会被同时训练.|
. 如果两个任务为不同的输入, 那么他们必须用|
来连接. 被|
连接起来的任务会被随机抽取来训练.
例如, 我们定义任务CWS|NER|weibo_ner&weibo_cws
, 那么在生成每一条数据时, 一个任务块会被随机抽取出来, 例如在这一次抽样中, weibo_ner&weibo_cws
被选中. 那么这次weibo_ner
和weibo_cws
会被同时训练. 因此, 在一个batch中, 有可能某些任务没有被抽中, loss为0.
训练, eval和导出模型请见notebooks
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for bert_multitask_learning-0.2.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7d48f3797e640e0a3d91668d3f736fa32ae1787e29bbe5890b2391fdce9bf7b0 |
|
MD5 | 7e8b3a473c8a74bf23f2e889bff7e89d |
|
BLAKE2b-256 | 71876a34ac77209b6981a1e68ac468dc3c9f6f63fee939409e385437b11825ad |