Bert for Multi-task Learning
Note: Since 0.4.0, tf version >= 2.1 is required.
Install
pip install bert-multitask-learning
What is it
This a project that uses transformers(based on huggingface transformers) to do multi-modal multi-task learning.
Why do I need this
In the original BERT code, neither multi-task learning or multiple GPU training is possible. Plus, the original purpose of this project is NER which dose not have a working script in the original BERT code.
To sum up, compared to the original bert repo, this repo has the following features:
- Multimodal multi-task learning(major reason of re-writing the majority of code).
- Multiple GPU training
- Support sequence labeling (for example, NER) and Encoder-Decoder Seq2Seq(with transformer decoder).
What type of problems are supported?
- Masked LM and next sentence prediction Pre-train(pretrain)
- Classification(cls)
- Sequence Labeling(seq_tag)
- Multi-Label Classification(multi_cls)
- Multi-modal Mask LM(mask_lm)
How to run pre-defined problems
There are two types of chaining operations can be used to chain problems.
&. If two problems have the same inputs, they can be chained using&. Problems chained by&will be trained at the same time.|. If two problems don't have the same inputs, they need to be chained using|. Problems chained by|will be sampled to train at every instance.
For example, cws|NER|weibo_ner&weibo_cws, one problem will be sampled at each turn, say weibo_ner&weibo_cws, then weibo_ner and weibo_cws will trained for this turn together. Therefore, in a particular batch, some tasks might not be sampled, and their loss could be 0 in this batch.
Please see the examples in notebooks for more details about training, evaluation and export models.
Bert多任务学习
注意：版本0.4.0后要求tf>=2.1
安装
pip install bert-multitask-learning
这是什么
这是利用transformer(基于huggingface transformers)进行多模态多任务学习的项目.
我为什么需要这个项目
在原始的BERT代码中, 是没有办法直接用多GPU进行多任务学习的. 另外, BERT并没有给出序列标注和Seq2seq的训练代码.
因此, 和原来的BERT相比, 这个项目具有以下特点:
- 多任务学习
- 多GPU训练
- 序列标注以及Encoder-decoder seq2seq的支持(用transformer decoder)
目前支持的任务类型
- Masked LM和next sentence prediction预训练(pretrain)
- 单标签分类(cls)
- 序列标注(seq_tag)
- 多标签分类(multi_cls)
- 多模态Mask LM(mask_lm)
如何运行预定义任务
可以用两种方法来将多个任务连接起来.
&. 如果两个任务有相同的输入, 不同标签的话, 那么他们可以用&来连接. 被&连接起来的任务会被同时训练.|. 如果两个任务为不同的输入, 那么他们必须用|来连接. 被|连接起来的任务会被随机抽取来训练.
例如, 我们定义任务cws|NER|weibo_ner&weibo_cws, 那么在生成每一条数据时, 一个任务块会被随机抽取出来, 例如在这一次抽样中, weibo_ner&weibo_cws被选中. 那么这次weibo_ner和weibo_cws会被同时训练. 因此, 在一个batch中, 有可能某些任务没有被抽中, loss为0.
训练, eval和导出模型请见notebooks
1 maintainer
Release files for bert-multitask-learning 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| bert_multitask_learning-0.7.0.tar.gz | 46.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| bert_multitask_learning-0.7.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 148.1 kB
Release files / bert_multitask_learning-0.7.0.tar.gz
| Download URL | bert_multitask_learning-0.7.0.tar.gz |
|---|---|
| Size | 46.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
546b8d68308290e36d2ffd645b49deae4032e3bc888a423c1ff0e30b3364baa0
|
|
BLAKE2b-256 checksum How to use checksums |
559d12581fd57c88e19308746a67f1d76f6356c91cbcbd1d123ec346c4e35620
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3
|
Release files / bert_multitask_learning-0.7.0-py3-none-any.whl
| Download URL | bert_multitask_learning-0.7.0-py3-none-any.whl |
|---|---|
| Size | 101.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e6bff09077ffcf3a93abecad348941fadb60cc6a81471fa85afd91e90335ed42
|
|
BLAKE2b-256 checksum How to use checksums |
c2f8cb28e3483ac46f940a033c27965ebe0fc82f770690f502c79e6e825b3805
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.3.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.2.0.post20200714 requests-toolbelt/0.9.1 tqdm/4.47.0 CPython/3.8.3
|
