Generating Extended and Multilingual Summaries with Pre-trained Transformers

Code for the paper Generating Extended and Multilingual Summaries with Pre-trained Transformers accepted at LREC 2022.

Getting started

Create the environnement, activate and install requirements.

conda create -n mdmls python=3.7
conda activate mdmls
pip install -r requirements.txt

WikinewsSum dataset

Please refer to https://github.com/airklizz/wikinewssum to download the dataset.

Place the train.json, validation.json, and test.json files in the wikinewssum/ folder.

Preprocessing

Prepare the dataset to fine-tune an abstractive model using an extractive pre-abstractive step.

python mdmls/main.py preprocess extractive-bert \
    wikinewssum/train.json \
    wikinewssum/train_pre_abstractive.json \
    --model-checkpoint distilbert-base-multilingual-cased \
    --pre-abstractive \
    --abstractive-model-checkpoint google/mt5-small

Tokenize the dataset.

python mdmls/main.py preprocess tokenize \
    wikinewssum/train_pre_abstractive.json \
    wikinewssum/train_pre_abstractive_tokenized.json \
    --source distilbert-base-multilingual-cased_extractive_summary \
    --model-checkpoint google/mt5-small

The same steps need to be performed for the validation set.

Fine-tuning

Use the command line interface to fine-tune a new model of the WikinewsSum dataset.

python mdmls/main.py train run \
    --train-data-files wikinewssum/train_pre_abstractive_tokenized.json \
    --validation-data-files wikinewssum/validation_pre_abstractive_tokenized.json \
    --training-scenario "new-fine-tuning" \
    --model-checkpoint google/mt5-base

To see all the parameters.

> python mdmls/main.py train run --help
Usage: main.py train run [OPTIONS]

Options:
  --train-data-files TEXT
  --validation-data-files TEXT
  --training-scenario TEXT
  --model-checkpoint TEXT         [default: google/mt5-small]
  --batch-size INTEGER            [default: 8]
  --gradient-accumulation-steps INTEGER
                                  [default: 1]
  --num-train-epochs INTEGER      [default: 8]
  --learning-rate FLOAT           [default: 5.6e-05]
  --weight-decay FLOAT            [default: 0.01]
  --save-total-limit INTEGER      [default: 3]
  --push-to-hub / --no-push-to-hub
                                  [default: push-to-hub]
  --language TEXT
  --max-number-training-sample INTEGER
  --help                          Show this message and exit.

Evaluation

ROUGE scores

ROUGE F-measure results of the three evaluations presented in the paper on WikinewsSum. We compare the extractive models, and mT5 in the three training scenarios and with two different pre-abstractive extractive steps (Oracle and mBERT) for each language of the WikinewsSum dataset in addiction to the all dataset.

BERTScore scores

BERTScore (Zhang et al., 2020b) precision (B-S P), recall (B-S R), and F1 (B-S F) results of the three evaluations presented in the paper on WikinewsSum. We compare the extractive models, and mT5 in the three training scenarios and with two different pre-abstractive extractive steps (Oracle and mBERT) for each language of the WikinewsSum dataset in addiction to the all dataset. Hash code for the BERTScore metric: bert-base-multilingual-cased_L9_no-idf_version=0.3.11(hug_trans=4.13.0)_fast-tokenizer

Usage

The mdmls pip package allows to run the combination of an extractive method combined with an abstractive one.

pip install mdmls

It can be used as follows in Python.

from mdmls import Summarizer

sum = Summarizer()
summary = sum(LONG_TEXT_TO_SUMMARIZE)

Or directly using the CLI.

mdmls "LONG_TEXT_TO_SUMMARIZE"

Models

All the fine-tuned abstractive models are available on the HuggingFace Hub: https://huggingface.co/models?sort=downloads&search=airklizz+mt5+wikinewssum