Skip to main content
Donate to the Python Software Foundation or Purchase a PyCharm License to Benefit the PSF! Donate Now

PyTorch implementation of BERT score

Project description


Automatic Evaluation Metric described in the paper BERTScore: Evaluating Text Generation with BERT.


*: Equal Contribution


BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity. It has been shown to correlate with human judgment on setence-level and system-level evaluation. Moreover, BERTScore computes precision, recall, and F1 measure, which can be useful for evaluating different language generation tasks.

For an illustration, BERTScore precision can be computed as

If you find this repo useful, please cite:

  title={BERTScore: Evaluating Text Generation with BERT},
  author={Zhang, Tianyi and Kishore, Varsha and Wu, Felix and Weinberger, Kilian Q. and Artzi, Yoav.},
  journal={arXiv preprint arXiv:1904.09675},


Install from pip by

pip install bert-score

Install it from the source by:

git clone
cd bert_score
pip install -r requiremnts.txt
pip install .



We provide a command line interface(CLI) of BERTScore as well as a python module. For the CLI, you can use it as follows:

  1. To evaluate English text files:

We provide example inputs under ./example.

bert-score -r example/refs.txt -c example/hyps.txt --bert bert-base-uncased 
  1. To evaluate Chinese text files:

Please format your input files similar to the ones in ./example.

bert-score -r [references] -c [candidates] --bert bert-base-chinese
  1. To evaluate text files in other languages:

Please format your input files similar to the ones in ./example.

bert-score -r [references] -c [candidates]

See more options by bert-score -h.

For the python module, we provide a demo. Please refer to bert_score/ for more details.


This repo wouldn't be possible without the awesome bert and pytorch-pretrained-BERT.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
bert_score-0.1.2-py3-none-any.whl (9.4 kB) Copy SHA256 hash SHA256 Wheel py3
bert_score-0.1.2.tar.gz (6.9 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page