Skip to main content

A pretrained NLP model for every NLP tasks

Project description

ERNIE 2.0 is a continual pre-training framework for language understanding in which pre-training tasks can be incrementally built and learned through multi-task learning. ERNIE 2.0 builds a strong basic for nearly every NLP tasks: Text Classification, Ranking, NER, Reading Comprehension, Genration and so on.


Table of contents

Quick Tour

import numpy as np
import paddle.fluid.dygraph as D
from ernie.tokenizing_ernie import ErnieTokenizer
from ernie.modeling_ernie import ErnieModel

D.guard().__enter__() # activate paddle `dygrpah` mode

model = ErnieModel.from_pretrained('ernie-1.0')    # Try to get pretrained model from server, make sure you have network connection
tokenizer = ErnieTokenizer.from_pretrained('ernie-1.0')

ids, _ = tokenizer.encode('hello world')
ids = D.to_variable(np.expand_dims(ids, 0))  # insert extra `batch` dimension
pooled, encoded = model(ids)                 # eager execution
print(pooled.numpy())                        # convert  results to numpy


Don't have GPU? try ERNIE in AIStudio! (please choose the latest version and apply for a GPU environment)

  1. Text classification walkthrough
  2. Sementic analysis
  3. Cloze test
  4. Knowledge Distillation
  5. Ask Ernie
  6. ...


1. install ernie
pip install -i paddle-ernie==0.0.1


git clone -b dygraph --single-branch
pip install -r requirement.txt
pip -e .
3. download pretrained models
Model Description
ERNIE 1.0 Base for Chinese ernie 1.0 base: L12H768A12
ERNIE tiny erine tiny: L3H1024A16
ERNIE 2.0 Base for English ernie 2.0 base: L12H768A12
ERNIE 2.0 Large for English ernie 2.0 large: L24H1024A16
4. download datasets

English Datasets

Download the GLUE datasets by running this script

the --data_dir option in the following section assumes a directory tree like this:

├── dev
│   └── 1
├── test   └── 1
└── train
    └── 1

see demo data for MNLI task.

Chinese Datasets

Datasets Description
XNLI XNLI is a natural language inference dataset in 15 languages. It was jointly built by Facebook and New York University. We use Chinese data of XNLI to evaluate language understanding ability of our model. [url:]
ChnSentiCorp ChnSentiCorp is a sentiment analysis dataset consisting of reviews on online shopping of hotels, notebooks and books.
MSRA-NER MSRA-NER (SIGHAN2006) dataset is released by MSRA for recognizing the names of people, locations and organizations in text.
NLPCC2016-DBQA NLPCC2016-DBQA is a sub-task of NLPCC-ICCPOL 2016 Shared Task which is hosted by NLPCC(Natural Language Processing and Chinese Computing), this task targets on selecting documents from the candidates to answer the questions. [url:]


  • try eager execution with dygraph model :
python3 ./demo/ \
    --from_pretrained ernie_1.0 \
    --data_dir ./data/xnli 
  • Distributed finetune

paddle.distributed.launch is a process manager, we use it to launch python processes on each avalible GPU devices:

when in distributed training, max_steps is used as stopping criteria rather than epoch to prevent dead block. also notice than we shard the train data according to device id to prevent over fitting.


python3 -m paddle.distributed.launch \
./demo/ \
    --data_dir data/mnli \
    --max_steps 10000 \
    --from_pretrained ernie2.0-en

many other demo python scripts:

  1. Sementic Analysis
  2. Pointwise Ranking
  3. Name Entity Recognition(NER)
  4. Machine Reading Comprehension
  5. Text generation...

recomended hyper parameters:

tasks batch size learning rate
CoLA 32 / 64 (base) 3e-5
SST-2 64 / 256 (base) 2e-5
STS-B 128 5e-5
QQP 256 3e-5(base)/5e-5(large)
MNLI 256 / 512 (base) 3e-5
QNLI 256 2e-5
RTE 16 / 4 (base) 2e-5(base)/3e-5(large)
MRPC 16 / 32 (base) 3e-5
WNLI 8 2e-5
XNLI 512 1e-4(base)/4e-5(large)
CMRC2018 64 3e-5
DRCD 64 5e-5(base)/3e-5(large)
MSRA-NER(SIGHAN2006) 16 5e-5(base)/1e-5(large)
ChnSentiCorp 24 5e-5(base)/1e-5(large)
LCQMC 32 2e-5(base)/5e-6(large)
NLPCC2016-DBQA 64 2e-5(base)/1e-5(large)

Distributed pretrain

see here

Online inference

If --inference_model_dir is passed to, a deployable model will be generated at the end of finetuning and your model is ready to serve.

For details about online inferece, see C++ inference API, or you can start a multi-gpu inference server with a few lines of codes:

python -m -m /path/to/saved/inference_model  -p 8881

and call the server just like calling local function (python3 only):

from propeller.service.client import InferenceClient
from ernie.tokenizing_ernie import ErnieTokenizer

client = InferenceClient('tcp://localhost:8881')
tokenizer = ErnieTokenizer.from_pretrained('ernie-1.0')
ids, sids = tokenizer.encode('hello world')
ids = np.expand_dims(ids, 0)
sids = np.expand_dims(sids, 0)
result = client(ids, sids)

A pre-made inference model for ernie-1.0 can be downloaded at here. It can be used for feature-based finetuning or feature extraction.


Knowledge distillation is good way to compress and accelerate ERNIE.

For details about distillation, see here


please cite ERNIE 2.0:

  title={ERNIE 2.0: A Continual Pre-training Framework for Language Understanding},
  author={Sun, Yu and Wang, Shuohuan and Li, Yukun and Feng, Shikun and Tian, Hao and Wu, Hua and Wang, Haifeng},

and ERNIE-gen

  title={ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation},
  author={Xiao, Dongling and Zhang, Han and Li, Yukun and Sun, Yu and Tian, Hao and Wu, Hua and Wang, Haifeng},

Contact us

  • Github Issues: bug reports, feature requests, install issues, usage issues, etc.
  • QQ discussion group: 760439550 (ERNIE discussion group).
  • Forums: discuss implementations, research, etc.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

paddle-ernie-0.0.1.dev1.tar.gz (19.9 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page