QGEval-qg 1.0.5

A tool for question generation based on LMs

How to use

Our codes provide the ability for Question Generation, please follow these steps to train your QG models.

Enviroment

run pip install -r requirements.txt to install the required packages.

Train QG Models

1. Prepare data

   You can use the hotpotQA dataset ./data/hotpotqa and squad1.1 dataset ./data/squad1.1 we provided, or use your own dataset which provides passages, references, and answers.

   If you use the datasets we provided, run python process.py to process the data, ready for your training.

   # For example: train T5 based on HotpotQA
   save_dir = 'directory to save processed data'
   hot = T5Hotpot()
   hot.deal_hotpot(save_dir)
   

2. Train your own QG models

   We provide codes to train T5/FLAN-T5/BART-based QG models. You may update training arguments in ./args, and run the code file for specific model to train.

   # model name: code file name
   BART-base/large_finetune: bart.py
   T5-base/large_finetune: T5.py
   Flan-T5-base/large_finetune: Flan-T5.py
   Flan-T5-xl/xxl_LoRA: Flan-T5-lora.py
   

   For example, train T5-base_finetune QG model based on HotpotQA

   • Update the arg_file ./args/t5_base_hotpot.json

     {
         "model_name_or_path": "google-t5/t5-base",
         "tokenizer_name_or_path": "google-t5/t5-base",
         "max_len": 512,
         "target_max_len": 163,
         "output_dir": "./model/t5-base/hotpotqa",
         "overwrite_output_dir": true,
         "per_device_train_batch_size": 16,
         "per_device_eval_batch_size": 16,
         "gradient_accumulation_steps": 8,
         "learning_rate": 0.0001,
         "warmup_steps": 500,
         "weight_decay": 0.01,
         "num_train_epochs": 10,
         "seed": 42,
         "do_train": true,
         "do_eval": true,
         "logging_steps": 100,
         "logging_dir": "./model/t5-base/logs",
         "train_file_path": "./data/data_t5/hotpotqa/pt/train.pt",
         "valid_file_path": "./data/data_t5/hotpotqa/pt/dev.pt",
         "remove_unused_columns": false,
         "prediction_loss_only": true,
         "load_best_model_at_end": true,
         "evaluation_strategy": "epoch",
         "save_strategy": "epoch"
     }
     

   • Run code in T5.py to train QG model

     # train model
     arg_path = './args/t5_base_hotpot.json'
     data_dir = 'directory of your data'    # processed data
     train_path = data_dir + 'train.json'
     dev_path = data_dir + 'dev.json'
     test_path = data_dir + 'test.json'   # optional
     train_qg(arg_path, train_path, dev_path, test_path)
     

Generate questions using QG models

We provide different types of QG models for question generation.

1. Use fine-tuned models in lmqg

   run python apply_lmqg.py to get the questions generated by models in lmqg.

   model_name = 'lmqg/flan-t5-base-squad-qg'
   data_json_path='./data/squad-test.json'
   save_path='./result/squad-test.csv'
   questions = apply_lmqg(
       model_name=model_name,
       data_json_path=data_json_path,
       save_path=save_path,
       batch_size=8
   )
   

2. Use fine-tuned (including LoRA) models we provided for QG

   Run specific code file for each type of model. For example, to use T5-base_finetune model to generate questions on HotpotQA Test Set, run code in T5.py.

   # test
   arg_path = './args/t5_base_hotpot.json'
   test_pt_path = './data/data_t5/hotpotqa/pt/test.pt'    # tokenized test data
   model_dir = './model/t5-base/hotpotqa/'
   tokenizer_dir = model_dir + 'tokenizer/'
   result_save_path = './result/t5-base-finetune/hotpotqa/prediction.csv'
   decoded_texts = predict(model_dir, tokenizer_dir, test_pt_path, result_save_path)
   

3. Use few-shot mode

   Run specific code file for each type of model. For example, to use Flan-T5-xl_fewshot model to generate questions on HotpotQA Test Set, run code in Flan-T5.py.

   # few-shot prediction
   model = 'google/flan-t5-large'
   few_shot_path = './data/squad1.1-few.json' # examples used for few-shot learning
   test_path = './data/squad-dev.json'
   result_save_path = './result/{}/prediction-few.csv'.format(model)
   dataset_name = 'hotpotqa'
   predict_few(model, few_shot_path, test_path, result_save_path, dataset_name)
   

4. Use GPTs

   Run code in generate_gpt.py

   data_path = './test.xlsx'    # your data for QG
   model = 'gpt-3.5-turbo'
   save_dir = './result/'
   few_shot_path = './data/hotpotqa-few.json'
   question_generation(
       data_path=data_path,
       model=model,
       save_dir=save_dir,
       few_shot_path=few_shot_path
   )