QAFactEval Summarization Factual Consistency Metric
Project description
QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization
This is the official code repository for the NAACL 2022 paper QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization by Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong.
In our paper, we conduct an extensive comparison of the components of QA-based metrics for factual consistency evaluation in summarization. Our optimized metric builds on QAEval with question consistency filtering and an improved answer overlap metric, leading to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark.
Table of Contents
Updates
5/2/2022 - Initial commit! :)
Using QAFactEval
You can install qafacteval via pip:
pip install qafacteval
You can also install from source:
git clone https://github.com/salesforce/QAFactEval
cd QAFactEval
pip install -e .
For use in scripts
Download the required pretrained models using download_models.sh
.
See run.py
for an example of using the QAFactEval metric:
from qafacteval import QAFactEval
kwargs = {"cuda_device": 0, "use_lerc_quip": True, \
"verbose": True, "generation_batch_size": 32, \
"answering_batch_size": 32, "lerc_batch_size": 8}
model_folder = "" # path to models downloaded with download_models.sh
metric = QAFactEval(
lerc_quip_path=f"{model_folder}/quip-512-mocha",
generation_model_path=f"{model_folder}/generation/model.tar.gz",
answering_model_dir=f"{model_folder}/answering",
lerc_model_path=f"{model_folder}/lerc/model.tar.gz",
lerc_pretrained_model_path=f"{model_folder}/lerc/pretraining.tar.gz",
**kwargs
)
results = metric.score_batch(["This is a source document"], [["This is a summary."]], return_qa_pairs=True)
score = results[0][0]['qa-eval']['lerc_quip']
Citation
When referencing this repository, please cite this paper:
@misc{fabbri-etal-2022-qafacteval,
title = {QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization},
author = {Alexander R. Fabbri and Chien-Sheng Wu and Wenhao Liu and Caiming Xiong},
year={2022},
eprint={2112.08542},
archivePrefix={arXiv},
primaryClass={cs.CL},
url = {https://arxiv.org/abs/2112.08542},
}
License
This repository is released under the BSD-3 License.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for qafacteval-0.10-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 13341e5a75712c90c9bb30d42ff854c99e67684b7b9c2602957c67218f132c40 |
|
MD5 | f2460ee72fff22f1d3f8ea931c22f5c1 |
|
BLAKE2b-256 | 2bfb7e623879338a42f4e983d3fac580aeb25fa1c101d1fbc3c8fea995a770c2 |