VeriScore 2.0.3

Pip package for VeriScore

VeriScore

This is introduction for pip package of VeriScore. It have two type of method to extract claims by

1. Prompting
2. Fine-tuned model

We have a preliminary Colab notebook for demo

VeriScore consists of three parts 1) claim extraction 2) evidence searching and 3) claim verification. We provide an end-to-end pipeline to obtain the VeriScore, along with each of its components individually.

You can choose between a prompting-based approach and a fine-tuned model-based approach using the model_name option. If you specify the path to the checkpoint of a fine-tuned model, it will automatically access the local model to perform inference. If the model name is not in a directory format, it will use an API call instead.

Install

1. Make a new Python 3.9+ environment using virtualenv or conda.
2. Install veriscore pacakge using pip
3. Download en_core_web_sm using spacy library
4. Our code supports inference using fine-tuned models based on the Unsloth library. To use this feature, you need to install the Unsloth library.

pip install --upgrade veriscore
python -m spacy download en_core_web_sm

Setup environment before running code

1. Download prompt folder that have txt file of prompt template (you can see prompt folder VeriScore's repository)
2. Set OpenAI or Claude API key to environment variable of bash for prompting approach

export OPENAI_API_KEY_PERSONAL={your_openai_api_key}
export CLAUDE_API_KEY={your_claude_api_key}

3. Set SERPER API key to environment variable of bash for searching evidence

export SERPER_KEY_PRIVATE={your_serper_api_key}

4. For the prompt-based approach, you need to set data_dir/demos/ with few-shot examples.

Running VeriScore using a command line

This is an end-to-end pipeline for running VeriScore.

 python3 -m veriscore.veriscore --data_dir {data_dir} --input_file {input_file} --model_name_extraction {model_name_extraction} --model_name_verification {model_name_verification}

• data_dir: Directory containing input data. ./data by default.
• input_file: Name of input data file. It should be jsonl format where each line contains
  • question: query to ask
  • response: generated response from of question
  • model: name of model generate response
  • prompt_source: name of dataset provide question like FreshQA
• model_name_extraction: Name of model used for claim extraction. gpt-4-0125-preview by default.
• model_name_verification: Name of model used for claim verification. gpt-4o by default.

Other optional flags:

• output_dir: Directory for saving ouptut data. ./data by default.
• cache_dir: Directory for saving cache data. ./data/cache by default.
• label_n: This is type of label for claim verification. It could be 2 (binary) ro 3 (trinary)
  • 2: query to ask binary supported and unsupported
  • 3: query to ask trinay labels—supported, contradicted, and inconclusive
• search_res_num: A Hyperparameter for number of search result. 5 by default.

Saving output: input_file_name is file name removed jsonl from —-input_file

• extracted claims will be saved to output_dir/claims_{input_file_name}.jsonl.
• searched evidence will be saved to output_dir/evidence_{input_file_name}.jsonl.
• verified claims will be saved to output_dir/model_output/verification_{input_file_name}.jsonl.

Running individual part using a command line

Claim extraction:

 python3 -m veriscore.extract_claims --data_dir {data_dir} --input_file {input_file} --model_name {model_name} 

• input_file: Name of input data file. It should be jsonl format where each line contains
  • question: query to ask
  • response: generated response from of question
  • model: name of model generate response
  • prompt_source: name of dataset provide question like FreshQA
• model_name: Name of model used for claim extraction. gpt-4-0125-preview by default. output:

 {
  "question": question.strip(),
  "prompt_source": prompt_source,
  "response": response.strip(),
  "prompt_tok_cnt": prompt_tok_cnt,
  "response_tok_cnt": response_tok_cnt,
  "model": model,
  "abstained": False,  
  "claim_list": list of claims for each snippet,
  "all_claims": list of all claims
 }

Evidence searching:

 python3 -m veriscore.retrieve_evidence --data_dir {data_dir} --input_file {input_file}

• input_file: Name of input data file. It should be jsonl format where each line contains the keys of the output dictionary from the Claim extraction. output:

 {
  ...
  "claim_snippets_dict": dictionary for claim and list of searched evidence. each evidence have dictionary of {"title": title, "snippet": snippet, "link": link}
 }

Claim verification:

 python3 -m veriscore.verify_claims --data_dir {data_dir} --input_file {input_file} --model_name {model_name}

• input_file: Name of input data file. It should be jsonl format where each line contains the keys of the output dictionary from the Evidence searching. output:

 {
  ...
  "response": result of claim verification
  "clean_output": post-processed label
 }